bitcoind abort unexpectedly #14345

<!-- This issue tracker is only for technical issues related to Bitcoin Core. General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange at https://bitcoin.stackexchange.com. For reporting security issues, please read instructions at https://bitcoincore.org/en/contact/. If the node is "stuck" during sync or giving "block checksum mismatch" errors, please ensure your hardware is stable by running memtest and observe CPU temperature with a load-test tool such as linpack before creating an issue! -->

<!-- Describe the issue -->

<!--- What behavior did you expect? -->

<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

[ec2-user@ip... ~]$ bitcoind
terminate called without an active exception
Aborted

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

[ec2-user@ip... ~]$ bitcoind -version
Bitcoin Core Daemon version v0.16.2
Copyright (C) 2009-2018 The Bitcoin Core developers

Please contribute if you find Bitcoin Core useful. Visit
<https://bitcoincore.org> for further information about the software.
The source code is available from <https://github.com/bitcoin/bitcoin>.

This is experimental software.
Distributed under the MIT software license, see the accompanying file COPYING
or <https://opensource.org/licenses/MIT>

This product includes software developed by the OpenSSL Project for use in the
OpenSSL Toolkit <https://www.openssl.org> and cryptographic software written by
Eric Young and UPnP software written by Thomas Bernard.

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

<!-- Any extra information that might be useful in the debugging process. -->

<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

Please provide a debug.log as well as:

How reliably can you reproduce the issue, what are the steps to do so? What type of machine are you observing the error on (OS/CPU and disk type)?

debug.log

How reliably can you reproduce the issue, what are the steps to do so?

Frequently. I left the daemon running for a while, later screen -r back, I see

What type of machine are you observing the error on (OS/CPU and disk type)?

[ec2-user@ip-172-31-23-227 ~]$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              1
On-line CPU(s) list: 0
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               63
Model name:          Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
Stepping:            2
CPU MHz:             2400.261
BogoMIPS:            4800.08
Hypervisor vendor:   Xen
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            30720K
NUMA node0 CPU(s):   0
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt

Oh. I see this issue.

2018-09-29 06:14:47 UpdateTip: new best=0000000000000011281e473dc5da50b772eb972c689741a99c576eaeaac7895d height=261460 version=0x00000002 log2_work=72.423216 tx=24818113 date='2013-10-03 11:25:13' progress=0.067093 cache=565.3MiB(4355665txo)
2018-09-29 06:14:47 UpdateTip: new best=000000000000001a849f50c9ac539d295071deaf6d6e24b6e8d1c7856c4fdb0f height=261461 version=0x00000002 log2_work=72.423361 tx=24818228 date='2013-10-03 11:26:34' progress=0.067093 cache=565.3MiB(4355673txo)
2018-09-29 06:14:47 Error: Out of memory. Terminating.

So btcd will increase mem usage as time goes on?

[ec2-user@ip-172-31-23-227 ~]$ free -m
              total        used        free      shared  buff/cache   available
Mem:            987          46         823           0         117         805
Swap:             0           0           0

After I start bitcoind for a while

[ec2-user@ip-172-31-23-227 ~]$ free -m
              total        used        free      shared  buff/cache   available
Mem:            987         375          63           0         548         463
Swap:             0           0           0

Later

[ec2-user@ip-172-31-23-227 ~]$ free -m
              total        used        free      shared  buff/cache   available
Mem:            987         860          60           0          66          16
Swap:             0           0           0

Should I simply change to a VM with bigger memory? Is there a cap for the mem usage?

Oh. I see this issue.

2018-09-29 06:14:47 UpdateTip: new best=0000000000000011281e473dc5da50b772eb972c689741a99c576eaeaac7895d height=261460 version=0x00000002 log2_work=72.423216 tx=24818113 date='2013-10-03 11:25:13' progress=0.067093 cache=565.3MiB(4355665txo)
2018-09-29 06:14:47 UpdateTip: new best=000000000000001a849f50c9ac539d295071deaf6d6e24b6e8d1c7856c4fdb0f height=261461 version=0x00000002 log2_work=72.423361 tx=24818228 date='2013-10-03 11:26:34' progress=0.067093 cache=565.3MiB(4355673txo)
2018-09-29 06:14:47 Error: Out of memory. Terminating.

have you tried reducing the dbcache option? try 128 or 256

bitcoind -dbcache=128

or add dbcache=128 into bitcoin.conf

Sorry I updated my EC2 instance and didn't try that parameter.