Assertion chainman().ActiveChain()[height] fails on startup on memory-constrained system #29422

issue vostrnad openend this issue on February 11, 2024

vostrnad commented at 6:07 pm on February 11, 2024: none

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

bitcoind crashes on startup with this error: (not logged in debug.log)

0node/interfaces.cpp:525 getBlockHash: Assertion `chainman().ActiveChain()[height]' failed.
1Aborted (core dumped)

This is after trying to bring the node up after it was killed by the system during IBD (likely due to OOM).

Expected behaviour

bitcoind should not crash, or at least it should exit with a more helpful error log.

Steps to reproduce

Run bitcoind with these configuration options:

0disablewallet=1
1prune=550
2listen=0
3coinstatsindex=1

Relevant log output

Last few lines before being killed (last line not recorded in debug.log):

02024-02-11T19:09:40Z UpdateTip: new best=00000000000005ab0d71214ffa5b38cd51e13525892a221bbf748fee8de806ad height=140916 version=0x00000001 log2_work=66.122154 tx=1282449 date='2011-08-14T11:18:30Z' progress=0.001328 cache=104.2MiB(896112txo)
12024-02-11T19:09:40Z UpdateTip: new best=0000000000000390717df9a1baaf8e65f9b96c87ee994d20dd897a95e8adc3e9 height=140917 version=0x00000001 log2_work=66.122300 tx=1282533 date='2011-08-14T11:27:11Z' progress=0.001328 cache=104.2MiB(896392txo)
22024-02-11T19:09:41Z UpdateTip: new best=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a height=140918 version=0x00000001 log2_work=66.122445 tx=1282552 date='2011-08-14T11:32:11Z' progress=0.001328 cache=104.2MiB(896387txo)
32024-02-11T19:09:41Z UpdateTip: new best=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c height=140919 version=0x00000001 log2_work=66.122591 tx=1282659 date='2011-08-14T12:12:06Z' progress=0.001328 cache=104.2MiB(896508txo)
42024-02-11T19:09:41Z UpdateTip: new best=000000000000017fab19329089a1ab54e1f4ab110b1ef0c5d4f9990c2815d624 height=140920 version=0x00000001 log2_work=66.122737 tx=1282673 date='2011-08-14T12:16:33Z' progress=0.001328 cache=104.2MiB(896564txo)
52024-02-11T19:09:41Z UpdateTip: new best=00000000000006a0e9cbe57335835c382bc09f11045b68dbb834495f4ae22faf height=140921 version=0x00000001 log2_work=66.122882 tx=1282777 date='2011-08-14T12:41:24Z' progress=0.001328 cache=104.2MiB(896561txo)
62024-02-11T19:09:41Z UpdateTip: new best=000000000000052e30dd0b273b500fde21e4bee67d1f5cd05fef91a663dadfcd height=140922 version=0x00000001 log2_work=66.123028 tx=1282926 date='2011-08-14T12:47:38Z' progress=0.001329 cache=104.2MiB(896633txo)
7Killed

After trying to start the node again (last two lines not recorded in debug.log):

 02024-02-11T19:46:07Z Bitcoin Core version v26.0.0 (release build)
 12024-02-11T19:46:07Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -upnp=0
 22024-02-11T19:46:07Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -natpmp=0
 32024-02-11T19:46:07Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -discover=0
 42024-02-11T19:46:07Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -listenonion=0
 52024-02-11T19:46:07Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -i2pacceptincoming=0
 62024-02-11T19:46:07Z Using the 'x86_shani(1way,2way)' SHA256 implementation
 72024-02-11T19:46:07Z Using RdSeed as an additional entropy source
 82024-02-11T19:46:07Z Using RdRand as an additional entropy source
 92024-02-11T19:46:07Z Default data directory /home/***/.bitcoin
102024-02-11T19:46:07Z Using data directory /home/***/.bitcoin
112024-02-11T19:46:07Z Config file: /home/***/.bitcoin/bitcoin.conf
122024-02-11T19:46:07Z Config file arg: coinstatsindex="1"
132024-02-11T19:46:07Z Config file arg: disablewallet="1"
142024-02-11T19:46:07Z Config file arg: listen="0"
152024-02-11T19:46:07Z Config file arg: prune="550"
162024-02-11T19:46:07Z Using at most 125 automatic connections (1024 file descriptors available)
172024-02-11T19:46:07Z Using 16 MiB out of 16 MiB requested for signature cache, able to store 524288 elements
182024-02-11T19:46:07Z Using 16 MiB out of 16 MiB requested for script execution cache, able to store 524288 elements
192024-02-11T19:46:07Z Script verification uses 0 additional threads
202024-02-11T19:46:07Z Wallet disabled!
212024-02-11T19:46:07Z scheduler thread start
222024-02-11T19:46:07Z Binding RPC on address ::1 port 8332
232024-02-11T19:46:07Z Binding RPC on address 127.0.0.1 port 8332
242024-02-11T19:46:07Z [http] creating work queue of depth 16
252024-02-11T19:46:07Z Using random cookie authentication.
262024-02-11T19:46:07Z Generated RPC authentication cookie /home/***/.bitcoin/.cookie
272024-02-11T19:46:07Z [http] starting 4 worker threads
282024-02-11T19:46:07Z Using /16 prefix for IP bucketing
292024-02-11T19:46:07Z init message: Loading P2P addresses…
302024-02-11T19:46:07Z Loaded 8837 addresses from peers.dat  20ms
312024-02-11T19:46:07Z init message: Loading banlist…
322024-02-11T19:46:07Z SetNetworkActive: true
332024-02-11T19:46:07Z /home/***/.bitcoin/fee_estimates.dat is not found. Continue anyway.
342024-02-11T19:46:07Z Cache configuration:
352024-02-11T19:46:07Z * Using 2.0 MiB for block index database
362024-02-11T19:46:07Z * Using 8.0 MiB for chain state database
372024-02-11T19:46:07Z * Using 440.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
382024-02-11T19:46:07Z init message: Loading block index…
392024-02-11T19:46:07Z Assuming ancestors of block 00000000000000000001a0a448d6cf2546b06801389cc030b2b18c6491266815 have valid signatures.
402024-02-11T19:46:07Z Setting nMinimumChainWork=000000000000000000000000000000000000000052b2559353df4117b7348b64
412024-02-11T19:46:07Z Prune configured to target 550 MiB on disk for block and undo files.
422024-02-11T19:46:07Z Opening LevelDB in /home/***/.bitcoin/blocks/index
432024-02-11T19:46:13Z Opened LevelDB successfully
442024-02-11T19:46:13Z Using obfuscation key for /home/***/.bitcoin/blocks/index: 0000000000000000
452024-02-11T19:46:17Z LoadBlockIndexDB: last block file = 3
462024-02-11T19:46:17Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=4007, size=94150965, heights=136926...140942, time=2011-07-18...2011-08-14)
472024-02-11T19:46:17Z Checking all blk files are present...
482024-02-11T19:46:17Z LoadBlockIndexDB(): Block files have previously been pruned
492024-02-11T19:46:17Z Initializing chainstate Chainstate [ibd] @ height -1 (null)
502024-02-11T19:46:17Z Opening LevelDB in /home/***/.bitcoin/chainstate
512024-02-11T19:46:17Z Opened LevelDB successfully
522024-02-11T19:46:17Z Using obfuscation key for /home/***/.bitcoin/chainstate: c31b181c8ff3ce9b
532024-02-11T19:46:17Z Opening LevelDB in /home/***/.bitcoin/chainstate
542024-02-11T19:46:17Z Opened LevelDB successfully
552024-02-11T19:46:17Z Using obfuscation key for /home/***/.bitcoin/chainstate: c31b181c8ff3ce9b
562024-02-11T19:46:17Z [Chainstate [ibd] @ height -1 (null)] resized coinsdb cache to 8.0 MiB
572024-02-11T19:46:17Z [Chainstate [ibd] @ height -1 (null)] resized coinstip cache to 440.0 MiB
582024-02-11T19:46:17Z init message: Verifying blocks…
592024-02-11T19:46:17Z  block index            9824ms
602024-02-11T19:46:17Z Opening LevelDB in /home/***/.bitcoin/indexes/coinstats/db
612024-02-11T19:46:17Z Opened LevelDB successfully
622024-02-11T19:46:17Z Using obfuscation key for /home/***/.bitcoin/indexes/coinstats/db: 0000000000000000
632024-02-11T19:46:17Z init message: Pruning blockstore…
642024-02-11T19:46:17Z initload thread start
65node/interfaces.cpp:525 getBlockHash: Assertion `chainman().ActiveChain()[height]' failed.
66Aborted (core dumped)

How did you obtain Bitcoin Core

Pre-built binaries

What version of Bitcoin Core are you using?

v26.0

Operating system and version

Ubuntu 20.04.6 LTS

Machine specifications

The system only has 1 GB of RAM.

maflcko added the label Bug on Feb 12, 2024
maflcko added the label UTXO Db and Indexes on Feb 12, 2024
maflcko added the label Data corruption on Feb 12, 2024

willcl-ark commented at 1:37 pm on February 12, 2024: member

I have not been able to reproduce this error using v26.0 binaries in a docker container limited to 1GB ram. At the same reported moment of your first crash I was only using ~550MB of ram:

  02024-02-12T12:14:50Z Bitcoin Core version v26.0.0 (release build)
  12024-02-12T12:14:50Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -upnp=0
  22024-02-12T12:14:50Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -natpmp=0
  32024-02-12T12:14:50Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -discover=0
  42024-02-12T12:14:50Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -listenonion=0
  52024-02-12T12:14:50Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -i2pacceptincoming=0
  62024-02-12T12:14:50Z Using the 'x86_shani(1way,2way)' SHA256 implementation
  72024-02-12T12:14:50Z Using RdSeed as an additional entropy source
  82024-02-12T12:14:50Z Using RdRand as an additional entropy source
  92024-02-12T12:14:50Z Default data directory /home/bitcoin/.bitcoin
 102024-02-12T12:14:50Z Using data directory /home/bitcoin/.bitcoin
 112024-02-12T12:14:50Z Config file: /home/bitcoin/.bitcoin/bitcoin.conf (not found, skipping)
 122024-02-12T12:14:50Z Command-line arg: coinstatsindex="1"
 132024-02-12T12:14:50Z Command-line arg: datadir="/home/bitcoin/.bitcoin"
 142024-02-12T12:14:50Z Command-line arg: disablewallet="1"
 152024-02-12T12:14:50Z Command-line arg: listen="0"
 162024-02-12T12:14:50Z Command-line arg: prune="550"
 172024-02-12T12:14:50Z Using at most 125 automatic connections (1048576 file descriptors available)
 182024-02-12T12:14:50Z Using 16 MiB out of 16 MiB requested for signature cache, able to store 524288 elements
 192024-02-12T12:14:50Z Using 16 MiB out of 16 MiB requested for script execution cache, able to store 524288 elements
 202024-02-12T12:14:50Z Script verification uses 15 additional threads
 212024-02-12T12:14:50Z Wallet disabled!
 222024-02-12T12:14:50Z scheduler thread start
 232024-02-12T12:14:50Z Binding RPC on address ::1 port 8332
 242024-02-12T12:14:50Z [libevent:warning] getaddrinfo: address family for nodename not supported
 252024-02-12T12:14:50Z Binding RPC on address ::1 port 8332 failed.
 262024-02-12T12:14:50Z Binding RPC on address 127.0.0.1 port 8332
 272024-02-12T12:14:50Z [http] creating work queue of depth 16
 282024-02-12T12:14:50Z Using random cookie authentication.
 292024-02-12T12:14:50Z Generated RPC authentication cookie /home/bitcoin/.bitcoin/.cookie
 302024-02-12T12:14:50Z [http] starting 4 worker threads
 312024-02-12T12:14:50Z Using /16 prefix for IP bucketing
 322024-02-12T12:14:50Z init message: Loading P2P addresses…
 332024-02-12T12:14:50Z Creating peers.dat because the file was not found ("/home/bitcoin/.bitcoin/peers.dat")
 342024-02-12T12:14:50Z init message: Loading banlist…
 352024-02-12T12:14:50Z Recreating the banlist database
 362024-02-12T12:14:50Z SetNetworkActive: true
 372024-02-12T12:14:50Z /home/bitcoin/.bitcoin/fee_estimates.dat is not found. Continue anyway.
 382024-02-12T12:14:50Z Cache configuration:
 392024-02-12T12:14:50Z * Using 2.0 MiB for block index database
 402024-02-12T12:14:50Z * Using 8.0 MiB for chain state database
 412024-02-12T12:14:50Z * Using 440.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
 422024-02-12T12:14:50Z init message: Loading block index…
 432024-02-12T12:14:50Z Assuming ancestors of block 00000000000000000001a0a448d6cf2546b06801389cc030b2b18c6491266815 have valid signatures.
 442024-02-12T12:14:50Z Setting nMinimumChainWork=000000000000000000000000000000000000000052b2559353df4117b7348b64
 452024-02-12T12:14:50Z Prune configured to target 550 MiB on disk for block and undo files.
 462024-02-12T12:14:50Z Opening LevelDB in /home/bitcoin/.bitcoin/blocks/index
 472024-02-12T12:14:51Z Opened LevelDB successfully
 482024-02-12T12:14:51Z Using obfuscation key for /home/bitcoin/.bitcoin/blocks/index: 0000000000000000
 492024-02-12T12:14:51Z LoadBlockIndexDB: last block file = 0
 502024-02-12T12:14:51Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=0, size=0, heights=0...0, time=1970-01-01...1970-01-01)
 512024-02-12T12:14:51Z Checking all blk files are present...
 522024-02-12T12:14:51Z Initializing databases...
 532024-02-12T12:14:51Z Initializing chainstate Chainstate [ibd] @ height -1 (null)
 542024-02-12T12:14:51Z Opening LevelDB in /home/bitcoin/.bitcoin/chainstate
 552024-02-12T12:14:51Z Opened LevelDB successfully
 562024-02-12T12:14:51Z Wrote new obfuscate key for /home/bitcoin/.bitcoin/chainstate: c14c06302791e0c7
 572024-02-12T12:14:51Z Using obfuscation key for /home/bitcoin/.bitcoin/chainstate: c14c06302791e0c7
 582024-02-12T12:14:51Z Opening LevelDB in /home/bitcoin/.bitcoin/chainstate
 592024-02-12T12:14:51Z Opened LevelDB successfully
 602024-02-12T12:14:51Z Using obfuscation key for /home/bitcoin/.bitcoin/chainstate: c14c06302791e0c7
 612024-02-12T12:14:51Z [Chainstate [ibd] @ height -1 (null)] resized coinsdb cache to 8.0 MiB
 622024-02-12T12:14:51Z [Chainstate [ibd] @ height -1 (null)] resized coinstip cache to 440.0 MiB
 632024-02-12T12:14:51Z init message: Verifying blocks…
 642024-02-12T12:14:51Z  block index              81ms
 652024-02-12T12:14:51Z Opening LevelDB in /home/bitcoin/.bitcoin/indexes/coinstats/db
 662024-02-12T12:14:51Z Opened LevelDB successfully
 672024-02-12T12:14:51Z Using obfuscation key for /home/bitcoin/.bitcoin/indexes/coinstats/db: 0000000000000000
 682024-02-12T12:14:51Z init message: Pruning blockstore…
 692024-02-12T12:14:51Z initload thread start
 702024-02-12T12:14:51Z UpdateTip: new best=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f height=0 version=0x00000001 log2_work=32.000022 tx=1 date='2009-01-03T18:15:05Z' progress=0.000000 cache=0.3MiB(0txo)
 712024-02-12T12:14:51Z block tree size = 1
 722024-02-12T12:14:51Z nBestHeight = 0
 732024-02-12T12:14:51Z Failed to open mempool file from disk. Continuing anyway.
 742024-02-12T12:14:51Z initload thread exit
 752024-02-12T12:14:51Z 0 block-relay-only anchors will be tried for connections.
 762024-02-12T12:14:51Z init message: Starting network threads…
 772024-02-12T12:14:51Z coinstatsindex thread start
 782024-02-12T12:14:51Z coinstatsindex is enabled
 792024-02-12T12:14:51Z coinstatsindex thread exit
 802024-02-12T12:14:51Z net thread start
 812024-02-12T12:14:51Z opencon thread start
 822024-02-12T12:14:51Z init message: Done loading
 832024-02-12T12:14:51Z msghand thread start
 842024-02-12T12:14:51Z addcon thread start
 852024-02-12T12:14:51Z dnsseed thread start
 862024-02-12T12:14:51Z Loading addresses from DNS seed seed.bitcoin.sprovoost.nl.
 872024-02-12T12:14:51Z Loading addresses from DNS seed seed.bitcoin.sipa.be.
 882024-02-12T12:14:51Z New outbound-full-relay v1 peer connected: version: 70016, blocks=830127, peer=0
 892024-02-12T12:14:54Z New outbound-full-relay v1 peer connected: version: 70016, blocks=830127, peer=1
 902024-02-12T12:14:56Z Loading addresses from DNS seed seed.btc.petertodd.org.
 912024-02-12T12:14:56Z Loading addresses from DNS seed seed.bitcoin.jonasschnelli.ch.
 922024-02-12T12:14:56Z Loading addresses from DNS seed dnsseed.bluematt.me.
 932024-02-12T12:14:56Z Loading addresses from DNS seed seed.bitcoin.wiz.biz.
 942024-02-12T12:14:56Z Loading addresses from DNS seed dnsseed.emzy.de.
 952024-02-12T12:14:56Z Loading addresses from DNS seed seed.bitcoinstats.com.
 962024-02-12T12:14:56Z Loading addresses from DNS seed dnsseed.bitcoin.dashjr.org.
 972024-02-12T12:14:56Z 184 addresses found from DNS seeds
 982024-02-12T12:14:56Z dnsseed thread exit
 992024-02-12T12:15:00Z New outbound-full-relay v1 peer connected: version: 70016, blocks=830127, peer=2
1002024-02-12T12:15:01Z New outbound-full-relay v1 peer connected: version: 70016, blocks=830127, peer=3
1012024-02-12T12:15:08Z New outbound-full-relay v1 peer connected: version: 70016, blocks=830127, peer=4
1022024-02-12T12:15:09Z New outbound-full-relay v1 peer connected: version: 70016, blocks=830127, peer=5
1032024-02-12T12:15:21Z New outbound-full-relay v1 peer connected: version: 70016, blocks=830127, peer=6
1042024-02-12T12:15:40Z New outbound-full-relay v1 peer connected: version: 70016, blocks=830127, peer=7
1052024-02-12T12:16:01Z Pre-synchronizing blockheaders, height: 2000 (~0.25%)
106
107# snip
108
1092024-02-12T12:33:29Z UpdateTip: new best=000000000000012347df20e12052bc71771c843897f1be4989912eb3301340bf height=140861 version=0x00000001 log2_work=66.114116 tx=1279613 date='2011-08-14T00:38:48Z' progress=0.001325 cache=103.7MiB(891721txo)
1102024-02-12T12:33:29Z UpdateTip: new best=00000000000005741a4a0135154fc02c9f72680918c524ba141987fdf01c8073 height=140862 version=0x00000001 log2_work=66.114263 tx=1279670 date='2011-08-14T01:07:38Z' progress=0.001325 cache=103.7MiB(891692txo)
1112024-02-12T12:33:29Z UpdateTip: new best=00000000000002b3093b70f6816fc369e9beb00f1dfeef1861d68a0a085c2e13 height=140863 version=0x00000001 log2_work=66.114409 tx=1279763 date='2011-08-14T01:51:26Z' progress=0.001325 cache=103.7MiB(891623txo)
1122024-02-12T12:33:29Z UpdateTip: new best=00000000000006f3e58dbacd07603134fead8df2023392f40a9da329732955d7 height=140864 version=0x00000001 log2_work=66.114556 tx=1279862 date='2011-08-14T02:04:52Z' progress=0.001325 cache=103.7MiB(891586txo)
1132024-02-12T12:33:29Z UpdateTip: new best=00000000000001bef6775f0abb1c7fec28539196bf5794ad47bcd72b40562259 height=140865 version=0x00000001 log2_work=66.114702 tx=1280064 date='2011-08-14T02:20:21Z' progress=0.001325 cache=103.7MiB(891745txo)
1142024-02-12T12:33:29Z UpdateTip: new best=0000000000000013efe7e2f7b9c87e64ae07f430cd852ac78f6185c86f60e59f height=140866 version=0x00000001 log2_work=66.114849 tx=1280232 date='2011-08-14T02:24:10Z' progress=0.001325 cache=103.7MiB(891881txo)
1152024-02-12T12:33:29Z UpdateTip: new best=00000000000007bd3f1b41385934579c100bbeb760c1ccc48b177e915eea699e height=140867 version=0x00000001 log2_work=66.114995 tx=1280242 date='2011-08-14T02:27:27Z' progress=0.001325 cache=103.7MiB(891874txo)
1162024-02-12T12:33:29Z UpdateTip: new best=00000000000001db1d1734deb6aa6bcfff41e4e2b0c201d0e1d2891a298e948f height=140868 version=0x00000001 log2_work=66.115141 tx=1280317 date='2011-08-14T02:48:02Z' progress=0.001325 cache=103.7MiB(891862txo)
1172024-02-12T12:33:29Z UpdateTip: new best=0000000000000835280c5ed61c5d2654a4f63f18e6245495e0c788a52662cce3 height=140869 version=0x00000001 log2_work=66.115288 tx=1280335 date='2011-08-14T02:57:35Z' progress=0.001325 cache=103.7MiB(891861txo)
1182024-02-12T12:33:29Z UpdateTip: new best=00000000000003c2f938d35d4c2bbf70909e0bf59f7000220fb1134f931b0c18 height=140870 version=0x00000001 log2_work=66.115434 tx=1280459 date='2011-08-14T03:18:23Z' progress=0.001326 cache=103.7MiB(891894txo)
1192024-02-12T12:33:29Z UpdateTip: new best=00000000000004fb8ea29d53ce1ccd049048ed74dfb2162e82b8dd48a92657cd height=140871 version=0x00000001 log2_work=66.115581 tx=1280477 date='2011-08-14T03:21:27Z' progress=0.001326 cache=103.7MiB(891902txo)
1202024-02-12T12:33:29Z UpdateTip: new best=000000000000048117412797f7115b2662e4b2dacc4b5528bd974aa91fd717a0 height=140872 version=0x00000001 log2_work=66.115727 tx=1280502 date='2011-08-14T03:25:07Z' progress=0.001326 cache=103.7MiB(892027txo)
1212024-02-12T12:33:29Z UpdateTip: new best=00000000000006b04676b35f33a6ca6e7d76c2cb0546b01b5e2029b915057d9b height=140873 version=0x00000001 log2_work=66.115874 tx=1280530 date='2011-08-14T03:34:28Z' progress=0.001326 cache=103.7MiB(892031txo)
1222024-02-12T12:33:29Z UpdateTip: new best=00000000000005c4c4e6c72fa5be197faba59461cd7f8cdb0433b754aed263cc height=140874 version=0x00000001 log2_work=66.116020 tx=1280540 date='2011-08-14T03:44:13Z' progress=0.001326 cache=104.0MiB(892129txo)
1232024-02-12T12:33:29Z UpdateTip: new best=0000000000000537fa486009b38bd6528f66cc8c0cad8c1f2a45602ba86e416e height=140875 version=0x00000001 log2_work=66.116166 tx=1280558 date='2011-08-14T03:45:50Z' progress=0.001326 cache=104.0MiB(892258txo)
1242024-02-12T12:33:29Z UpdateTip: new best=0000000000000795ee1e0c8944db976c8fe974f6428532c1344bb2935b96378f height=140876 version=0x00000001 log2_work=66.116313 tx=1280603 date='2011-08-14T03:48:39Z' progress=0.001326 cache=104.0MiB(892269txo)
1252024-02-12T12:33:29Z UpdateTip: new best=00000000000005b39075a8d1c0c8a452ce2ef6cc294ce57c0ad82e22a8f38952 height=140877 version=0x00000001 log2_work=66.116459 tx=1280611 date='2011-08-14T03:52:04Z' progress=0.001326 cache=104.0MiB(892274txo)
1262024-02-12T12:33:29Z UpdateTip: new best=0000000000000117019fc53aa4307393732bf9463301fef7d68633486f8ad9ae height=140878 version=0x00000001 log2_work=66.116605 tx=1280628 date='2011-08-14T03:55:44Z' progress=0.001326 cache=104.0MiB(892265txo)
1272024-02-12T12:33:29Z UpdateTip: new best=000000000000029e8a9dd7b78d9466db21e42c4b0b2d165c4d5bd1d029375a88 height=140879 version=0x00000001 log2_work=66.116751 tx=1280738 date='2011-08-14T04:04:48Z' progress=0.001326 cache=104.0MiB(892361txo)
1282024-02-12T12:33:29Z UpdateTip: new best=000000000000049f1dd2d67a116940ef7cb0d38fbabbae8d90b56be7085a7cef height=140880 version=0x00000001 log2_work=66.116898 tx=1280751 date='2011-08-14T04:05:55Z' progress=0.001326 cache=104.0MiB(892616txo)
1292024-02-12T12:33:29Z UpdateTip: new best=00000000000008c8aeb45f8c445b6b807c08f4631430f8ef1619d227c803b0d1 height=140881 version=0x00000001 log2_work=66.117044 tx=1280762 date='2011-08-14T04:08:22Z' progress=0.001326 cache=104.0MiB(892616txo)
1302024-02-12T12:33:29Z UpdateTip: new best=000000000000082c362d823868906428151c3137174e52f63084cc3f1145e343 height=140882 version=0x00000001 log2_work=66.117190 tx=1280780 date='2011-08-14T04:11:50Z' progress=0.001326 cache=104.0MiB(892609txo)
1312024-02-12T12:33:29Z UpdateTip: new best=00000000000008d904a971a53424ecbc298bf8ac7b0f795ccaed7270c505d971 height=140883 version=0x00000001 log2_work=66.117337 tx=1280801 date='2011-08-14T04:18:08Z' progress=0.001326 cache=104.0MiB(892710txo)
1322024-02-12T12:33:29Z UpdateTip: new best=00000000000004377be62d34aab0ca14cf9f4c8ed4755c2fa9714d610d96c7c2 height=140884 version=0x00000001 log2_work=66.117483 tx=1280896 date='2011-08-14T04:52:19Z' progress=0.001326 cache=104.0MiB(892851txo)
1332024-02-12T12:33:29Z UpdateTip: new best=00000000000001b2356cadc5fc3e91eae953b50de6c7a48a74122442b75620ae height=140885 version=0x00000001 log2_work=66.117629 tx=1280971 date='2011-08-14T05:24:06Z' progress=0.001326 cache=104.0MiB(893014txo)
1342024-02-12T12:33:29Z UpdateTip: new best=00000000000008cfa9b2a237a94d7f61d9c428e99b391e132f78ff97fb1133b1 height=140886 version=0x00000001 log2_work=66.117775 tx=1281051 date='2011-08-14T05:29:22Z' progress=0.001326 cache=104.0MiB(893080txo)
1352024-02-12T12:33:29Z UpdateTip: new best=000000000000056df910e7e8d7a9cf1ecda41ff99b4b533e734348bcf0a3c1e9 height=140887 version=0x00000001 log2_work=66.117921 tx=1281061 date='2011-08-14T05:33:36Z' progress=0.001326 cache=104.0MiB(893089txo)
1362024-02-12T12:33:29Z UpdateTip: new best=000000000000043368b2c47b36f245d2ba62a16fc2866156da4142701d30b555 height=140888 version=0x00000001 log2_work=66.118067 tx=1281100 date='2011-08-14T05:41:51Z' progress=0.001326 cache=104.0MiB(893046txo)
1372024-02-12T12:33:29Z UpdateTip: new best=000000000000055c91e49b5548bfe867b1d7bcc3b1040c5d6e7139569fadb5da height=140889 version=0x00000001 log2_work=66.118214 tx=1281101 date='2011-08-14T05:50:56Z' progress=0.001326 cache=104.0MiB(893047txo)
1382024-02-12T12:33:29Z UpdateTip: new best=0000000000000386f5fda5383fa95533b0858205b859cac2be1a143c83262bc4 height=140890 version=0x00000001 log2_work=66.118360 tx=1281151 date='2011-08-14T05:54:14Z' progress=0.001326 cache=104.0MiB(893000txo)
1392024-02-12T12:33:29Z UpdateTip: new best=000000000000001ccf9f4e71e484fd4ac2dcf4c7179aec17dac081978522e084 height=140891 version=0x00000001 log2_work=66.118506 tx=1281152 date='2011-08-14T06:12:11Z' progress=0.001326 cache=104.0MiB(893048txo)
1402024-02-12T12:33:29Z UpdateTip: new best=0000000000000521e7f446a2929d03d8fd23c012d98ac53f2f8c9cd2d4f820a6 height=140892 version=0x00000001 log2_work=66.118652 tx=1281223 date='2011-08-14T06:05:07Z' progress=0.001326 cache=104.0MiB(893218txo)
1412024-02-12T12:33:29Z UpdateTip: new best=00000000000005ba76c956fa46732181a5d49dd7052ad8a5761587e2eaf29dcb height=140893 version=0x00000001 log2_work=66.118798 tx=1281229 date='2011-08-14T06:09:15Z' progress=0.001326 cache=104.0MiB(893215txo)
1422024-02-12T12:33:29Z UpdateTip: new best=000000000000080b6e8d9331df86458779461a5d2e335f796e0e803929af3662 height=140894 version=0x00000001 log2_work=66.118944 tx=1281234 date='2011-08-14T06:09:27Z' progress=0.001326 cache=104.0MiB(893218txo)
1432024-02-12T12:33:29Z UpdateTip: new best=0000000000000246de84d61c9e8600057d390e1ab5bda26b334f3d537af5f8a5 height=140895 version=0x00000001 log2_work=66.119090 tx=1281299 date='2011-08-14T06:39:08Z' progress=0.001326 cache=104.0MiB(893339txo)
1442024-02-12T12:33:29Z UpdateTip: new best=00000000000002a502ccdbe643c93035fbf735f9f3772fa40f6c948ef8b09f05 height=140896 version=0x00000001 log2_work=66.119236 tx=1281314 date='2011-08-14T06:42:40Z' progress=0.001326 cache=104.0MiB(893337txo)
1452024-02-12T12:33:29Z UpdateTip: new best=00000000000006d015a42e460f0a08cc0feaafd93730e7d5644d725e80fa3357 height=140897 version=0x00000001 log2_work=66.119382 tx=1281424 date='2011-08-14T07:16:41Z' progress=0.001327 cache=104.0MiB(893294txo)
1462024-02-12T12:33:29Z UpdateTip: new best=00000000000003fb8061a4685c2f2595bdfcd0e4595f5194ec779a3df7ba124a height=140898 version=0x00000001 log2_work=66.119528 tx=1281452 date='2011-08-14T07:19:58Z' progress=0.001327 cache=104.0MiB(893285txo)
1472024-02-12T12:33:29Z UpdateTip: new best=000000000000017c60fba3b5a5a3bce9be24252f22148d9a7078246c8b4b46c5 height=140899 version=0x00000001 log2_work=66.119674 tx=1281485 date='2011-08-14T07:29:15Z' progress=0.001327 cache=104.0MiB(893975txo)
1482024-02-12T12:33:29Z UpdateTip: new best=00000000000006b52211755a6550b7195f1783909ac296a4c932d779f837aea4 height=140900 version=0x00000001 log2_work=66.119820 tx=1281536 date='2011-08-14T07:58:18Z' progress=0.001327 cache=104.2MiB(895013txo)
1492024-02-12T12:33:29Z UpdateTip: new best=00000000000000f6bdfeefc9f2a005da2d5038bea0ea36f1b1e2a6cb82c863a9 height=140901 version=0x00000001 log2_work=66.119966 tx=1281537 date='2011-08-14T07:58:34Z' progress=0.001327 cache=104.2MiB(895014txo)
1502024-02-12T12:33:29Z UpdateTip: new best=00000000000005991004f70bfc3fd2a391b1cfe9fb182b755140841eb3d57ecd height=140902 version=0x00000001 log2_work=66.120112 tx=1281643 date='2011-08-14T08:16:49Z' progress=0.001327 cache=104.2MiB(895282txo)
1512024-02-12T12:33:29Z UpdateTip: new best=0000000000000690ecfa8e241961bc0f337864374f998a02b9a10a251e519987 height=140903 version=0x00000001 log2_work=66.120258 tx=1281797 date='2011-08-14T08:32:37Z' progress=0.001327 cache=104.2MiB(895377txo)
1522024-02-12T12:33:29Z UpdateTip: new best=00000000000008560799379b2d1a51f166e36f76a893f28e829271fa3758d094 height=140904 version=0x00000001 log2_work=66.120404 tx=1281804 date='2011-08-14T08:42:19Z' progress=0.001327 cache=104.2MiB(895375txo)
1532024-02-12T12:33:29Z UpdateTip: new best=00000000000001a74e6fa019ede32722173708afc84d2e0118576acb6d1146be height=140905 version=0x00000001 log2_work=66.120550 tx=1281820 date='2011-08-14T08:45:48Z' progress=0.001327 cache=104.2MiB(895366txo)
1542024-02-12T12:33:29Z UpdateTip: new best=00000000000007b64474d6436a3c32344a09c8fdd476d51af22374307649690f height=140906 version=0x00000001 log2_work=66.120696 tx=1281833 date='2011-08-14T08:57:17Z' progress=0.001327 cache=104.2MiB(895378txo)
1552024-02-12T12:33:29Z UpdateTip: new best=00000000000000f403070c91e8b6d600c4a6f275d3cb100a61e78cb123ab8614 height=140907 version=0x00000001 log2_work=66.120842 tx=1281890 date='2011-08-14T08:58:16Z' progress=0.001327 cache=104.2MiB(895508txo)
1562024-02-12T12:33:29Z UpdateTip: new best=00000000000002444daaac90a79da204f38a31d02f14503b48b92f06f7e3465e height=140908 version=0x00000001 log2_work=66.120988 tx=1281898 date='2011-08-14T08:59:08Z' progress=0.001327 cache=104.2MiB(895508txo)
1572024-02-12T12:33:29Z UpdateTip: new best=00000000000003d961404286ce3d29d45e402725225ffab55f77db911bbf475a height=140909 version=0x00000001 log2_work=66.121133 tx=1281925 date='2011-08-14T09:01:40Z' progress=0.001327 cache=104.2MiB(895533txo)
1582024-02-12T12:33:29Z UpdateTip: new best=00000000000003b2f362306df3232af83f397ff58bb8562c296bff290d28d857 height=140910 version=0x00000001 log2_work=66.121279 tx=1281994 date='2011-08-14T09:15:56Z' progress=0.001327 cache=104.2MiB(895565txo)
1592024-02-12T12:33:29Z UpdateTip: new best=00000000000002d686e56746e94d8587ce9fe79793c0b28b750317677a830b8d height=140911 version=0x00000001 log2_work=66.121425 tx=1282155 date='2011-08-14T09:33:55Z' progress=0.001327 cache=104.2MiB(895878txo)
1602024-02-12T12:33:29Z UpdateTip: new best=00000000000007f15e2e295a697bba76415eac8149d2cb9bd4167ced0fdde4db height=140912 version=0x00000001 log2_work=66.121571 tx=1282167 date='2011-08-14T09:36:15Z' progress=0.001327 cache=104.2MiB(895890txo)
1612024-02-12T12:33:29Z UpdateTip: new best=000000000000036430dda4c2e0e2cb0f3b98893fc3e3cadd4db99e6ce6cd5bc7 height=140913 version=0x00000001 log2_work=66.121717 tx=1282174 date='2011-08-14T09:36:59Z' progress=0.001327 cache=104.2MiB(895890txo)
1622024-02-12T12:33:29Z UpdateTip: new best=0000000000000452918ccdebd47fd6e808838204d3311badf7d20eee3cacd884 height=140914 version=0x00000001 log2_work=66.121862 tx=1282175 date='2011-08-14T09:39:11Z' progress=0.001327 cache=104.2MiB(895891txo)
1632024-02-12T12:33:29Z UpdateTip: new best=000000000000086c4deffb797bbd084cebeff98b9967c9a6450e9118813b5f79 height=140915 version=0x00000001 log2_work=66.122008 tx=1282342 date='2011-08-14T10:54:11Z' progress=0.001328 cache=104.2MiB(895861txo)
1642024-02-12T12:33:29Z UpdateTip: new best=00000000000005ab0d71214ffa5b38cd51e13525892a221bbf748fee8de806ad height=140916 version=0x00000001 log2_work=66.122154 tx=1282449 date='2011-08-14T11:18:30Z' progress=0.001328 cache=104.2MiB(896112txo)
1652024-02-12T12:33:29Z UpdateTip: new best=0000000000000390717df9a1baaf8e65f9b96c87ee994d20dd897a95e8adc3e9 height=140917 version=0x00000001 log2_work=66.122300 tx=1282533 date='2011-08-14T11:27:11Z' progress=0.001328 cache=104.2MiB(896392txo)
1662024-02-12T12:33:29Z UpdateTip: new best=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a height=140918 version=0x00000001 log2_work=66.122445 tx=1282552 date='2011-08-14T11:32:11Z' progress=0.001328 cache=104.2MiB(896387txo)
1672024-02-12T12:33:29Z UpdateTip: new best=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c height=140919 version=0x00000001 log2_work=66.122591 tx=1282659 date='2011-08-14T12:12:06Z' progress=0.001328 cache=104.2MiB(896508txo)
1682024-02-12T12:33:29Z UpdateTip: new best=000000000000017fab19329089a1ab54e1f4ab110b1ef0c5d4f9990c2815d624 height=140920 version=0x00000001 log2_work=66.122737 tx=1282673 date='2011-08-14T12:16:33Z' progress=0.001328 cache=104.2MiB(896564txo)
1692024-02-12T12:33:29Z UpdateTip: new best=00000000000006a0e9cbe57335835c382bc09f11045b68dbb834495f4ae22faf height=140921 version=0x00000001 log2_work=66.122882 tx=1282777 date='2011-08-14T12:41:24Z' progress=0.001328 cache=104.2MiB(896561txo)
1702024-02-12T12:33:29Z UpdateTip: new best=000000000000052e30dd0b273b500fde21e4bee67d1f5cd05fef91a663dadfcd height=140922 version=0x00000001 log2_work=66.123028 tx=1282926 date='2011-08-14T12:47:38Z' progress=0.001328 cache=104.2MiB(896633txo)
1712024-02-12T12:33:29Z UpdateTip: new best=0000000000000505d10f019c9b3f65adfb5f3108b768ab1ee20e11de5ebb6a00 height=140923 version=0x00000001 log2_work=66.123174 tx=1282950 date='2011-08-14T12:57:11Z' progress=0.001328 cache=104.2MiB(896593txo)
1722024-02-12T12:33:29Z UpdateTip: new best=00000000000001848c99ecb30ba01ef8845a4f3a919d12d0bdf100de8fed0e23 height=140924 version=0x00000001 log2_work=66.123319 tx=1282980 date='2011-08-14T13:02:39Z' progress=0.001328 cache=104.2MiB(896603txo)
1732024-02-12T12:33:32Z UpdateTip: new best=00000000000000aca192e11511b57f2315c05ad35b74ee791a3c583e70dd37df height=140925 version=0x00000001 log2_work=66.123465 tx=1283013 date='2011-08-14T13:11:24Z' progress=0.001328 cache=0.3MiB(0txo)
1742024-02-12T12:33:32Z UpdateTip: new best=000000000000045d8ee15145b337a68a8fcac0ec3383a6dc66fe8231e975a86d height=140926 version=0x00000001 log2_work=66.123611 tx=1283102 date='2011-08-14T13:16:23Z' progress=0.001328 cache=0.3MiB(167txo)
1752024-02-12T12:33:32Z UpdateTip: new best=00000000000000ac816bbafff668eef4db476dd798668af107f251cac8df757a height=140927 version=0x00000001 log2_work=66.123756 tx=1283183 date='2011-08-14T13:40:30Z' progress=0.001328 cache=0.3MiB(620txo)
1762024-02-12T12:33:32Z UpdateTip: new best=0000000000000301efaaad8762bd28e6aeb9062267bcc2f04adceb1b3cf063de height=140928 version=0x00000001 log2_work=66.123902 tx=1283336 date='2011-08-14T14:13:16Z' progress=0.001329 cache=0.3MiB(1033txo)
1772024-02-12T12:33:32Z UpdateTip: new best=00000000000007059f7a8a49f40d8cac4a886bdcb0da9d5628c612755e19749f height=140929 version=0x00000001 log2_work=66.124047 tx=1283441 date='2011-08-14T14:13:21Z' progress=0.001329 cache=0.3MiB(1311txo)
1782024-02-12T12:33:32Z UpdateTip: new best=0000000000000218609989b97674ece3ea8a893600e09ff62054f16c3eb9b7fa height=140930 version=0x00000001 log2_work=66.124193 tx=1283492 date='2011-08-14T14:21:56Z' progress=0.001329 cache=0.3MiB(1555txo)
1792024-02-12T12:33:32Z UpdateTip: new best=0000000000000294b1b3a69b435ab1834d7ec9e3e25360503ed35e343cd64fe1 height=140931 version=0x00000001 log2_work=66.124338 tx=1283578 date='2011-08-14T14:35:25Z' progress=0.001329 cache=0.3MiB(1739txo)
1802024-02-12T12:33:32Z UpdateTip: new best=000000000000062fc6063597ed32f5171d9063a652ca8942550791c7ebe3459d height=140932 version=0x00000001 log2_work=66.124484 tx=1283663 date='2011-08-14T14:46:38Z' progress=0.001329 cache=0.3MiB(1964txo)
1812024-02-12T12:33:32Z UpdateTip: new best=00000000000005860d8f5623a5d03b5283b4d6132e50277ee9134eeb14483369 height=140933 version=0x00000001 log2_work=66.124629 tx=1283701 date='2011-08-14T14:56:51Z' progress=0.001329 cache=0.3MiB(2039txo)
1822024-02-12T12:33:32Z UpdateTip: new best=0000000000000862242536576cd7e538a24dcabeb69a1d85a422652f90be0c26 height=140934 version=0x00000001 log2_work=66.124775 tx=1283777 date='2011-08-14T15:02:13Z' progress=0.001329 cache=0.3MiB(2130txo)
1832024-02-12T12:33:32Z UpdateTip: new best=000000000000041c36d18be75e7d8a140ea529e1311fe7aaf3db02305a83f1d8 height=140935 version=0x00000001 log2_work=66.124920 tx=1283790 date='2011-08-14T15:04:07Z' progress=0.001329 cache=0.3MiB(2156txo)
1842024-02-12T12:33:32Z UpdateTip: new best=0000000000000266c8a15a5bfd7c54d851ed72679738e0f1aaeaadae5ba05a74 height=140936 version=0x00000001 log2_work=66.125066 tx=1283845 date='2011-08-14T15:09:59Z' progress=0.001329 cache=0.5MiB(2824txo)
1852024-02-12T12:33:32Z UpdateTip: new best=000000000000088eab42674fb4ae951639a352c4f775eb893d06d612887e3095 height=140937 version=0x00000001 log2_work=66.125211 tx=1283854 date='2011-08-14T15:10:37Z' progress=0.001329 cache=0.5MiB(2837txo)
1862024-02-12T12:33:32Z UpdateTip: new best=0000000000000423ef50e704cb5ffe149886ab845f778190df792b1a0b2170d6 height=140938 version=0x00000001 log2_work=66.125357 tx=1283922 date='2011-08-14T15:23:42Z' progress=0.001329 cache=0.5MiB(3052txo)
1872024-02-12T12:33:32Z UpdateTip: new best=00000000000005de3eab6d895b427781e9d67b61b539e41e131b07acc260621a height=140939 version=0x00000001 log2_work=66.125502 tx=1283961 date='2011-08-14T15:28:31Z' progress=0.001329 cache=0.5MiB(3157txo)
1882024-02-12T12:33:32Z UpdateTip: new best=00000000000000508fcd68007b0f2a61bf50fc01c2b5ae389c33c40a0f8b5d62 height=140940 version=0x00000001 log2_work=66.125647 tx=1284046 date='2011-08-14T15:39:33Z' progress=0.001329 cache=0.5MiB(3418txo)
1892024-02-12T12:33:32Z UpdateTip: new best=0000000000000876a0a8e270a36e6e099a86469f6ac49a2cae1a59b16f66395c height=140941 version=0x00000001 log2_work=66.125793 tx=1284105 date='2011-08-14T15:47:05Z' progress=0.001329 cache=0.5MiB(3678txo)
1902024-02-12T12:33:32Z UpdateTip: new best=00000000000007ec7c11f7e93eed3773d33162100528dc943de2e8d1f709452c height=140942 version=0x00000001 log2_work=66.125938 tx=1284186 date='2011-08-14T15:57:36Z' progress=0.001329 cache=0.5MiB(3787txo)
1912024-02-12T12:33:32Z UpdateTip: new best=00000000000005abbec6d40ccdac6edb69d47ebb1da7be2a1a473eb6e727037d height=140943 version=0x00000001 log2_work=66.126083 tx=1284192 date='2011-08-14T15:59:54Z' progress=0.001329 cache=0.5MiB(3815txo)
1922024-02-12T12:33:32Z UpdateTip: new best=0000000000000272901ca119d34b260360e09ca7e882c688809063f1194f9882 height=140944 version=0x00000001 log2_work=66.126229 tx=1284337 date='2011-08-14T16:03:05Z' progress=0.001330 cache=0.5MiB(4003txo)
1932024-02-12T12:33:32Z UpdateTip: new best=000000000000015f51e585bcf3a6ecd629c40983e736c5b4b1e6850a393d8823 height=140945 version=0x00000001 log2_work=66.126374 tx=1284428 date='2011-08-14T16:30:55Z' progress=0.001330 cache=0.5MiB(4248txo)
1942024-02-12T12:33:32Z UpdateTip: new best=000000000000065bc012e8943231fcfb8334eeeaae6c850c4876d54b692bd724 height=140946 version=0x00000001 log2_work=66.126519 tx=1284552 date='2011-08-14T16:43:23Z' progress=0.001330 cache=0.5MiB(4613txo)
1952024-02-12T12:33:32Z UpdateTip: new best=00000000000006114513c1f5e2f100fc9733c68d4fc65dd51aff1d166953890a height=140947 version=0x00000001 log2_work=66.126665 tx=1284641 date='2011-08-14T16:54:39Z' progress=0.001330 cache=0.5MiB(4827txo)
1962024-02-12T12:33:32Z UpdateTip: new best=0000000000000106456526f7b87a1e94145341d34b1d16b7a860f73c54399f82 height=140948 version=0x00000001 log2_work=66.126810 tx=1284760 date='2011-08-14T17:05:46Z' progress=0.001330 cache=0.8MiB(5058txo)
1972024-02-12T12:33:32Z UpdateTip: new best=000000000000071d77d4c4e2f152de9b8062e8e348280aaf74d77696ea929eb6 height=140949 version=0x00000001 log2_work=66.126955 tx=1284865 date='2011-08-14T17:34:46Z' progress=0.001330 cache=0.8MiB(5304txo)
1982024-02-12T12:33:32Z UpdateTip: new best=00000000000007eb70c6ce97ca7770f6eeee8dd589f5ab18925e72f20e4d44af height=140950 version=0x00000001 log2_work=66.127100 tx=1284956 date='2011-08-14T17:58:02Z' progress=0.001330 cache=0.8MiB(5440txo)
1992024-02-12T12:33:32Z UpdateTip: new best=0000000000000508583487d3d945f92621f74e1cacc0e869e766b603d0717807 height=140951 version=0x00000001 log2_work=66.127246 tx=1285036 date='2011-08-14T18:03:16Z' progress=0.001330 cache=0.8MiB(5512txo)
2002024-02-12T12:33:32Z UpdateTip: new best=00000000000003d7be2a7aa098ddcc5628bee828c6a075e0c12db89bd8af0e26 height=140952 version=0x00000001 log2_work=66.127391 tx=1285099 date='2011-08-14T18:10:52Z' progress=0.001330 cache=0.8MiB(5842txo)
2012024-02-12T12:33:32Z UpdateTip: new best=00000000000000db3704afe8284a8dc27d9ca51642eb0971dc0f6ab598d5037c height=140953 version=0x00000001 log2_work=66.127536 tx=1285154 date='2011-08-14T18:21:53Z' progress=0.001330 cache=0.8MiB(6063txo)
2022024-02-12T12:33:32Z UpdateTip: new best=000000000000053eca0681e9aca454e32e9977192bdedee276e00410eb35cb36 height=140954 version=0x00000001 log2_work=66.127681 tx=1285202 date='2011-08-14T18:29:08Z' progress=0.001330 cache=0.8MiB(6120txo)
2032024-02-12T12:33:32Z UpdateTip: new best=00000000000000ff4bed5f36753f51b0ea1d08945f10bff61fc5b6ea0cc0f1d8 height=140955 version=0x00000001 log2_work=66.127826 tx=1285221 date='2011-08-14T18:31:45Z' progress=0.001331 cache=0.8MiB(6145txo)
2042024-02-12T12:33:32Z UpdateTip: new best=0000000000000237057110d7eec1eaab32b3c9f42e7faf571b2fd283da26807b height=140956 version=0x00000001 log2_work=66.127972 tx=1285227 date='2011-08-14T18:33:21Z' progress=0.001331 cache=0.8MiB(6150txo)
2052024-02-12T12:33:32Z UpdateTip: new best=00000000000001b858a0827f6a73cb29df3b9d3d593009c054e5ba67937b04d4 height=140957 version=0x00000001 log2_work=66.128117 tx=1285237 date='2011-08-14T18:49:20Z' progress=0.001331 cache=0.8MiB(6219txo)
2062024-02-12T12:33:32Z UpdateTip: new best=000000000000032d6ad003850d71b7f05292ccc9f64868314ec462b70434e8fb height=140958 version=0x00000001 log2_work=66.128262 tx=1285305 date='2011-08-14T18:53:01Z' progress=0.001331 cache=0.8MiB(6442txo)
2072024-02-12T12:33:32Z UpdateTip: new best=0000000000000167be8a3289c0c7e2889dd0987bd1f887e1ba2a5dc64388922f height=140959 version=0x00000001 log2_work=66.128407 tx=1285377 date='2011-08-14T19:01:22Z' progress=0.001331 cache=0.8MiB(6546txo)
2082024-02-12T12:33:32Z UpdateTip: new best=00000000000008558154af33b562c997bd76d4b0077203ea289c3d374aece23d height=140960 version=0x00000001 log2_work=66.128552 tx=1285389 date='2011-08-14T19:07:00Z' progress=0.001331 cache=0.8MiB(6712txo)
2092024-02-12T12:33:32Z UpdateTip: new best=000000000000027e5622cb0fb9beaccbe3e57f3e544a077fe1fd8793c1873f14 height=140961 version=0x00000001 log2_work=66.128697 tx=1285499 date='2011-08-14T19:36:11Z' progress=0.001331 cache=0.8MiB(6979txo)
2102024-02-12T12:33:32Z UpdateTip: new best=00000000000002e9e8e54f36b30a40bd5c39e7e2dc27df704dd10ca5721f9c75 height=140962 version=0x00000001 log2_work=66.128842 tx=1285505 date='2011-08-14T19:37:39Z' progress=0.001331 cache=0.8MiB(6990txo)
2112024-02-12T12:33:32Z UpdateTip: new best=000000000000012a6c8e1a13235fb4818b05e211d18c01163862f77e91a65f1f height=140963 version=0x00000001 log2_work=66.128987 tx=1285516 date='2011-08-14T19:39:19Z' progress=0.001331 cache=0.8MiB(7008txo)
2122024-02-12T12:33:32Z UpdateTip: new best=00000000000006dc443b8f3d634e3a4329b9d4b80506ac0ebc8b303762c311bf height=140964 version=0x00000001 log2_work=66.129132 tx=1285579 date='2011-08-14T19:47:29Z' progress=0.001331 cache=0.8MiB(7314txo)
2132024-02-12T12:33:32Z UpdateTip: new best=00000000000005edfe2025e11a5a7567cc08341a10525d7948557e260ad0cb4c height=140965 version=0x00000001 log2_work=66.129277 tx=1285598 date='2011-08-14T19:59:26Z' progress=0.001331 cache=0.8MiB(7473txo)
2142024-02-12T12:33:32Z UpdateTip: new best=000000000000019ea2fe645af8ec6e4c2afcf18438fec38cc83b1a26ad1f7630 height=140966 version=0x00000001 log2_work=66.129422 tx=1285740 date='2011-08-14T20:02:14Z' progress=0.001331 cache=1.1MiB(7650txo)
2152024-02-12T12:33:32Z UpdateTip: new best=00000000000007986f8206ae0b5c8087d96a9c688f3d2e366387d22a35f5138b height=140967 version=0x00000001 log2_work=66.129567 tx=1285752 date='2011-08-14T20:05:53Z' progress=0.001331 cache=1.1MiB(7691txo)
2162024-02-12T12:33:32Z UpdateTip: new best=00000000000001d8f5859ad595ba84306d24786f562a098cc1baae70f77b821b height=140968 version=0x00000001 log2_work=66.129712 tx=1285795 date='2011-08-14T20:10:27Z' progress=0.001331 cache=1.1MiB(7744txo)
2172024-02-12T12:33:32Z UpdateTip: new best=000000000000005ea6ebae5c0fd069109a2d64087946b56a78accca4614ff2a9 height=140969 version=0x00000001 log2_work=66.129857 tx=1285803 date='2011-08-14T20:32:33Z' progress=0.001331 cache=1.1MiB(7843txo)
2182024-02-12T12:33:32Z UpdateTip: new best=0000000000000131d20cd3cecfa702e1cec9a1ec59d6c260fcfd288605d1bed7 height=140970 version=0x00000001 log2_work=66.130002 tx=1285913 date='2011-08-14T20:33:05Z' progress=0.001331 cache=1.1MiB(8243txo)
2192024-02-12T12:33:32Z UpdateTip: new best=00000000000007307ae4690c715a7d0b65cc83c3dd2d8a9e01b5fc63cbaef825 height=140971 version=0x00000001 log2_work=66.130147 tx=1285995 date='2011-08-14T20:48:21Z' progress=0.001331 cache=1.1MiB(8567txo)
2202024-02-12T12:33:32Z UpdateTip: new best=000000000000008a571f4c9bbd0576958608ff439bc0c2dd1d1be1eba23f8987 height=140972 version=0x00000001 log2_work=66.130292 tx=1286016 date='2011-08-14T20:53:17Z' progress=0.001331 cache=1.1MiB(8599txo)
2212024-02-12T12:33:32Z UpdateTip: new best=0000000000000154324ed7172f6855c1ea1c3c2aaff0b18f32f0b31184d15996 height=140973 version=0x00000001 log2_work=66.130437 tx=1286019 date='2011-08-14T20:54:16Z' progress=0.001331 cache=1.1MiB(8602txo)
2222024-02-12T12:33:32Z UpdateTip: new best=000000000000038384a63cafdadd1d318c7e2480553409fcd4add240e39c4057 height=140974 version=0x00000001 log2_work=66.130582 tx=1286135 date='2011-08-14T21:04:12Z' progress=0.001331 cache=1.1MiB(8754txo)
2232024-02-12T12:33:32Z UpdateTip: new best=00000000000006aec6265b6d31e7687bc5d1bca4cda2e019cbb9f1044d8d228b height=140975 version=0x00000001 log2_work=66.130727 tx=1286160 date='2011-08-14T21:10:25Z' progress=0.001331 cache=1.1MiB(8962txo)
2242024-02-12T12:33:32Z UpdateTip: new best=0000000000000696aecdff296091db165de18901b1d4b5aa449b2c53bac6a9ba height=140976 version=0x00000001 log2_work=66.130872 tx=1286242 date='2011-08-14T21:25:50Z' progress=0.001332 cache=1.1MiB(9096txo)
2252024-02-12T12:33:32Z UpdateTip: new best=00000000000008d9e540380394d13fc452ab0fed66025bdd31cab7e382ae7642 height=140977 version=0x00000001 log2_work=66.131017 tx=1286297 date='2011-08-14T22:00:53Z' progress=0.001332 cache=1.1MiB(9351txo)
2262024-02-12T12:33:32Z UpdateTip: new best=00000000000004305e40fcf07b556128e0e7171e758c3f68897102c00968fa9f height=140978 version=0x00000001 log2_work=66.131161 tx=1286356 date='2011-08-14T22:27:32Z' progress=0.001332 cache=1.1MiB(9496txo)
2272024-02-12T12:33:32Z UpdateTip: new best=000000000000075c5022aaaff34ab12c3a2b279438bc16b480d5f851141d6a31 height=140979 version=0x00000001 log2_work=66.131306 tx=1286468 date='2011-08-14T22:30:54Z' progress=0.001332 cache=1.1MiB(9719txo)
2282024-02-12T12:33:32Z UpdateTip: new best=000000000000064cdf32a18b3d56d8a2056e9f6756246a469f502d5d3a18a28a height=140980 version=0x00000001 log2_work=66.131451 tx=1286602 date='2011-08-14T22:33:55Z' progress=0.001332 cache=1.1MiB(9853txo)
2292024-02-12T12:33:32Z UpdateTip: new best=00000000000007f737bffac49b36478a00a2c3b24adcbcd1c1483121f19fe5ac height=140981 version=0x00000001 log2_work=66.131596 tx=1286642 date='2011-08-14T22:37:48Z' progress=0.001332 cache=1.1MiB(9942txo)
2302024-02-12T12:33:32Z UpdateTip: new best=00000000000006885b464eb98a18d7bec9cf2970c7243a3aac4b90352f4f51f6 height=140982 version=0x00000001 log2_work=66.131741 tx=1286656 date='2011-08-14T22:40:34Z' progress=0.001332 cache=1.1MiB(9970txo)
2312024-02-12T12:33:32Z UpdateTip: new best=00000000000005e7407253dc4a294434a5ac442725423d94d089009fde203efc height=140983 version=0x00000001 log2_work=66.131885 tx=1286665 date='2011-08-14T22:44:07Z' progress=0.001332 cache=1.1MiB(9986txo)
2322024-02-12T12:33:32Z UpdateTip: new best=000000000000052fd45a0c511c976bd24c4c0739ea85e6daffe6a1e0a6cd5bf4 height=140984 version=0x00000001 log2_work=66.132030 tx=1286684 date='2011-08-14T22:53:43Z' progress=0.001332 cache=1.1MiB(10040txo)
2332024-02-12T12:33:32Z UpdateTip: new best=000000000000014be3a92bb42395fffb2e13992d3e9d7a77ce32f7982c9dd62e height=140985 version=0x00000001 log2_work=66.132175 tx=1286685 date='2011-08-14T22:53:41Z' progress=0.001332 cache=1.1MiB(10041txo)
2342024-02-12T12:33:32Z UpdateTip: new best=0000000000000239020b988c9be65fc3627be0afe3bfaa9a06f204fa845d67fa height=140986 version=0x00000001 log2_work=66.132320 tx=1286771 date='2011-08-14T23:03:06Z' progress=0.001332 cache=1.3MiB(10201txo)
2352024-02-12T12:33:32Z UpdateTip: new best=00000000000005a936d150d27640ceb9abb890e1ed0bca3a88a76a0c675dd726 height=140987 version=0x00000001 log2_work=66.132464 tx=1286810 date='2011-08-14T23:03:52Z' progress=0.001332 cache=1.3MiB(10232txo)
2362024-02-12T12:33:32Z UpdateTip: new best=000000000000066b139c485ba6a957b006c33d00dcbb48a52c4f452acf022494 height=140988 version=0x00000001 log2_work=66.132609 tx=1286855 date='2011-08-14T23:15:20Z' progress=0.001332 cache=1.4MiB(10311txo)
2372024-02-12T12:33:32Z UpdateTip: new best=0000000000000890fc205764b6a4044ab33115bc89f74d507e94e1f8ed7fb85e height=140989 version=0x00000001 log2_work=66.132754 tx=1286926 date='2011-08-14T23:33:22Z' progress=0.001332 cache=1.4MiB(10507txo)
2382024-02-12T12:33:32Z UpdateTip: new best=00000000000007ea60af6c685642587a202d9d0b463e5560a030b0e9b8ec6c82 height=140990 version=0x00000001 log2_work=66.132898 tx=1286948 date='2011-08-14T23:34:37Z' progress=0.001332 cache=1.4MiB(10548txo)
2392024-02-12T12:33:32Z UpdateTip: new best=00000000000000878428f32fcc4965c584f684289e21edc34fc18844dc945c1b height=140991 version=0x00000001 log2_work=66.133043 tx=1287013 date='2011-08-14T23:47:22Z' progress=0.001332 cache=1.4MiB(10645txo)
2402024-02-12T12:33:32Z UpdateTip: new best=00000000000000860e6b30b1eca07419ea1359ad930efb510d01ed8a5656450c height=140992 version=0x00000001 log2_work=66.133188 tx=1287029 date='2011-08-14T23:48:50Z' progress=0.001332 cache=1.4MiB(10674txo)

I have now synced to block 340,000 without any errors. I have seen RAM usage go as high as 950MB though and wonder if you were running other services on the machine? If so, they could have triggered the OOM…

Overall though, bitcoind seems to run stably in a 1GB container so far as I can tell.

It’s more difficult to address the second part of your report, as I cannot recreate the state of your bitcoin data directory. If bitcoin core repeatedly fails startup in this way we might be able to get closer to the issue if you run using bitcoind -debug=1. However I’m not convinced that we’ll learn a lot more from this. I assume that nullptr is being returned from the index into ActiveChain(), which when passed to getBlockHash() raises the assert. This could indicate that the current sequence of updating the chainstate manager can become inconsistent during a crash but it’s not immediately clear to me where that might be happening.

It’s likely that the short-term fix for you will be to -reindex and try to ensure that the machine isn’t running any other processes which trigger OOM. If you go for a reindex and are running other processes which you can’t disable, you might want to try running with -blocksonly enabled during the sync, as well as reducing dbcache to give the system some more headroom (e.g. (-dbcache=300).

vostrnad commented at 10:19 pm on February 12, 2024: none

I can reproduce the issue every time on my machine. Here it is with -debug=1:

  02024-02-12T20:01:01Z [net] Requesting block 000000000000073bc48e2c26389c0034f0c3fc448cfc0571fc0382bce8f3aa0d (141081) peer=25
  12024-02-12T20:01:01Z [net] sending getdata (37 bytes) peer=25
  22024-02-12T20:01:01Z [net] received: block (25657 bytes) peer=18
  32024-02-12T20:01:01Z [net] received block 000000000000045d8ee15145b337a68a8fcac0ec3383a6dc66fe8231e975a86d peer=18
  42024-02-12T20:01:01Z [net] Requesting block 00000000000005e2fe30aa3a0fc497fe44403b824fec3b254f8cdbb4664df8f1 (141082) peer=18
  52024-02-12T20:01:01Z [net] sending getdata (37 bytes) peer=18
  62024-02-12T20:01:01Z [net] received: block (49068 bytes) peer=17
  72024-02-12T20:01:01Z [net] received block 000000000000052e30dd0b273b500fde21e4bee67d1f5cd05fef91a663dadfcd peer=17
  82024-02-12T20:01:01Z [net] Requesting block 00000000000000e400a3d3ada8f4d4f39276084bb58d6223b6e5682db3b56583 (141083) peer=17
  92024-02-12T20:01:01Z [net] sending getdata (37 bytes) peer=17
 102024-02-12T20:01:01Z [net] received: block (58205 bytes) peer=7
 112024-02-12T20:01:01Z [net] received block 0000000000000301efaaad8762bd28e6aeb9062267bcc2f04adceb1b3cf063de peer=7
 122024-02-12T20:01:01Z [net] Requesting block 0000000000000205dbf966de130752f8f93c66ccf8210be03a577d0cfaf2cd7a (141084) peer=7
 132024-02-12T20:01:01Z [net] sending getdata (37 bytes) peer=7
 142024-02-12T20:01:01Z [net] received: block (49168 bytes) peer=0
 152024-02-12T20:01:01Z [net] received block 00000000000000ac816bbafff668eef4db476dd798668af107f251cac8df757a peer=0
 162024-02-12T20:01:01Z [net] Requesting block 00000000000008debbce6b72a32ba3c60c83665637cde91312f3d29fede0e29b (141085) peer=0
 172024-02-12T20:01:01Z [net] sending getdata (37 bytes) peer=0
 182024-02-12T20:01:01Z [net] received: block (10552 bytes) peer=9
 192024-02-12T20:01:01Z [net] received block 000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a peer=9
 202024-02-12T20:01:01Z [bench]   - Load block from disk: 0.08ms
 212024-02-12T20:01:01Z [bench]     - Sanity checks: 0.02ms [25.91s (0.18ms/blk)]
 222024-02-12T20:01:01Z [bench]     - Fork checks: 0.04ms [73.81s (0.52ms/blk)]
 232024-02-12T20:01:01Z [bench]       - Connect 19 transactions: 0.08ms (0.004ms/tx, 0.002ms/txin) [95.12s (0.67ms/blk)]
 242024-02-12T20:01:01Z [bench]     - Verify 49 txins: 0.09ms (0.002ms/txin) [131.28s (0.93ms/blk)]
 252024-02-12T20:01:01Z [bench]     - Write undo data: 0.06ms [75.28s (0.53ms/blk)]
 262024-02-12T20:01:01Z [bench]     - Index writing: 0.01ms [30.07s (0.21ms/blk)]
 272024-02-12T20:01:01Z [validation] BlockChecked: block hash=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a state=Valid
 282024-02-12T20:01:01Z [bench]   - Connect total: 0.28ms [435.70s (3.09ms/blk)]
 292024-02-12T20:01:01Z [bench]   - Flush: 0.04ms [66.27s (0.47ms/blk)]
 302024-02-12T20:01:01Z [bench]   - Writing chainstate: 0.01ms [26.22s (0.19ms/blk)]
 312024-02-12T20:01:01Z [estimatefee] Blockpolicy estimates updated by 0 of 0 block txs, since last block 0 of 0 tracked, mempool map size 0, max target 0 from current
 322024-02-12T20:01:01Z UpdateTip: new best=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a height=140918 version=0x00000001 log2_work=66.122445 tx=1282552 date='2011-08-14T11:32:11Z' progress=0.001328 cache=104.2MiB(896387txo)
 332024-02-12T20:01:01Z [bench]   - Connect postprocess: 0.10ms [150.69s (1.07ms/blk)]
 342024-02-12T20:01:01Z [bench] - Connect block: 0.51ms [768.21s (5.45ms/blk)]
 352024-02-12T20:01:01Z [validation] Enqueuing BlockConnected: block hash=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a block height=140918
 362024-02-12T20:01:01Z [validation] Enqueuing UpdatedBlockTip: new block hash=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a fork block hash=0000000000000390717df9a1baaf8e65f9b96c87ee994d20dd897a95e8adc3e9 (in IBD=true)
 372024-02-12T20:01:01Z [bench]   - Load block from disk: 19.41ms
 382024-02-12T20:01:01Z [bench]     - Sanity checks: 0.09ms [25.91s (0.18ms/blk)]
 392024-02-12T20:01:01Z [bench]     - Fork checks: 0.24ms [73.81s (0.52ms/blk)]
 402024-02-12T20:01:01Z [bench]       - Connect 107 transactions: 0.39ms (0.004ms/tx, 0.002ms/txin) [95.12s (0.67ms/blk)]
 412024-02-12T20:01:01Z [bench]     - Verify 227 txins: 0.43ms (0.002ms/txin) [131.28s (0.93ms/blk)]
 422024-02-12T20:01:01Z [bench]     - Write undo data: 19.32ms [75.30s (0.53ms/blk)]
 432024-02-12T20:01:01Z [bench]     - Index writing: 0.05ms [30.07s (0.21ms/blk)]
 442024-02-12T20:01:01Z [validation] BlockChecked: block hash=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c state=Valid
 452024-02-12T20:01:01Z [bench]   - Connect total: 20.25ms [435.72s (3.09ms/blk)]
 462024-02-12T20:01:01Z [bench]   - Flush: 0.33ms [66.27s (0.47ms/blk)]
 472024-02-12T20:01:01Z [bench]   - Writing chainstate: 0.04ms [26.22s (0.19ms/blk)]
 482024-02-12T20:01:01Z [estimatefee] Blockpolicy estimates updated by 0 of 0 block txs, since last block 0 of 0 tracked, mempool map size 0, max target 0 from current
 492024-02-12T20:01:01Z UpdateTip: new best=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c height=140919 version=0x00000001 log2_work=66.122591 tx=1282659 date='2011-08-14T12:12:06Z' progress=0.001328 cache=104.2MiB(896508txo)
 502024-02-12T20:01:01Z [bench]   - Connect postprocess: 9.31ms [150.70s (1.07ms/blk)]
 512024-02-12T20:01:01Z [bench] - Connect block: 49.35ms [768.26s (5.45ms/blk)]
 522024-02-12T20:01:01Z [validation] Enqueuing BlockConnected: block hash=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c block height=140919
 532024-02-12T20:01:01Z [validation] Enqueuing UpdatedBlockTip: new block hash=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c fork block hash=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a (in IBD=true)
 542024-02-12T20:01:01Z [bench]   - Load block from disk: 0.06ms
 552024-02-12T20:01:01Z [bench]     - Sanity checks: 0.02ms [25.91s (0.18ms/blk)]
 562024-02-12T20:01:01Z [bench]     - Fork checks: 0.06ms [73.81s (0.52ms/blk)]
 572024-02-12T20:01:01Z [bench]       - Connect 14 transactions: 0.05ms (0.004ms/tx, 0.003ms/txin) [95.12s (0.67ms/blk)]
 582024-02-12T20:01:01Z [bench]     - Verify 17 txins: 0.06ms (0.004ms/txin) [131.28s (0.93ms/blk)]
 592024-02-12T20:01:01Z [bench]     - Write undo data: 0.04ms [75.30s (0.53ms/blk)]
 602024-02-12T20:01:01Z [bench]     - Index writing: 0.01ms [30.07s (0.21ms/blk)]
 612024-02-12T20:01:01Z [validation] BlockChecked: block hash=000000000000017fab19329089a1ab54e1f4ab110b1ef0c5d4f9990c2815d624 state=Valid
 622024-02-12T20:01:01Z [bench]   - Connect total: 0.24ms [435.72s (3.09ms/blk)]
 632024-02-12T20:01:01Z [bench]   - Flush: 0.04ms [66.27s (0.47ms/blk)]
 642024-02-12T20:01:01Z [bench]   - Writing chainstate: 0.01ms [26.22s (0.19ms/blk)]
 652024-02-12T20:01:01Z [estimatefee] Blockpolicy estimates updated by 0 of 0 block txs, since last block 0 of 0 tracked, mempool map size 0, max target 0 from current
 662024-02-12T20:01:01Z UpdateTip: new best=000000000000017fab19329089a1ab54e1f4ab110b1ef0c5d4f9990c2815d624 height=140920 version=0x00000001 log2_work=66.122737 tx=1282673 date='2011-08-14T12:16:33Z' progress=0.001328 cache=104.2MiB(896564txo)
 672024-02-12T20:01:01Z [bench]   - Connect postprocess: 0.09ms [150.70s (1.07ms/blk)]
 682024-02-12T20:01:01Z [bench] - Connect block: 0.44ms [768.26s (5.45ms/blk)]
 692024-02-12T20:01:01Z [validation] Enqueuing BlockConnected: block hash=000000000000017fab19329089a1ab54e1f4ab110b1ef0c5d4f9990c2815d624 block height=140920
 702024-02-12T20:01:01Z [validation] Enqueuing UpdatedBlockTip: new block hash=000000000000017fab19329089a1ab54e1f4ab110b1ef0c5d4f9990c2815d624 fork block hash=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c (in IBD=true)
 712024-02-12T20:01:01Z [leveldb] WriteBatch memory usage: db=db, before=0.0MiB, after=0.0MiB
 722024-02-12T20:01:01Z [validation] UpdatedBlockTip: new block hash=000000000000086c4deffb797bbd084cebeff98b9967c9a6450e9118813b5f79 fork block hash=0000000000000452918ccdebd47fd6e808838204d3311badf7d20eee3cacd884 (in IBD=true)
 732024-02-12T20:01:01Z [validation] BlockConnected: block hash=00000000000005ab0d71214ffa5b38cd51e13525892a221bbf748fee8de806ad block height=140916
 742024-02-12T20:01:01Z [leveldb] WriteBatch memory usage: db=db, before=0.0MiB, after=0.0MiB
 752024-02-12T20:01:01Z [validation] UpdatedBlockTip: new block hash=00000000000005ab0d71214ffa5b38cd51e13525892a221bbf748fee8de806ad fork block hash=000000000000086c4deffb797bbd084cebeff98b9967c9a6450e9118813b5f79 (in IBD=true)
 762024-02-12T20:01:01Z [validation] BlockConnected: block hash=0000000000000390717df9a1baaf8e65f9b96c87ee994d20dd897a95e8adc3e9 block height=140917
 772024-02-12T20:01:01Z [leveldb] WriteBatch memory usage: db=db, before=0.0MiB, after=0.0MiB
 782024-02-12T20:01:01Z [validation] UpdatedBlockTip: new block hash=0000000000000390717df9a1baaf8e65f9b96c87ee994d20dd897a95e8adc3e9 fork block hash=00000000000005ab0d71214ffa5b38cd51e13525892a221bbf748fee8de806ad (in IBD=true)
 792024-02-12T20:01:01Z [validation] BlockConnected: block hash=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a block height=140918
 802024-02-12T20:01:02Z [leveldb] WriteBatch memory usage: db=db, before=0.0MiB, after=0.0MiB
 812024-02-12T20:01:02Z [validation] UpdatedBlockTip: new block hash=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a fork block hash=0000000000000390717df9a1baaf8e65f9b96c87ee994d20dd897a95e8adc3e9 (in IBD=true)
 822024-02-12T20:01:02Z [validation] BlockConnected: block hash=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c block height=140919
 832024-02-12T20:01:02Z [leveldb] WriteBatch memory usage: db=db, before=0.0MiB, after=0.0MiB
 842024-02-12T20:01:02Z [validation] UpdatedBlockTip: new block hash=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c fork block hash=000000000000038559aa65f99789894199f2f4f08bfba66454864b5e514ba58a (in IBD=true)
 852024-02-12T20:01:02Z [validation] BlockConnected: block hash=000000000000017fab19329089a1ab54e1f4ab110b1ef0c5d4f9990c2815d624 block height=140920
 862024-02-12T20:01:02Z [leveldb] WriteBatch memory usage: db=db, before=0.0MiB, after=0.0MiB
 872024-02-12T20:01:02Z [validation] UpdatedBlockTip: new block hash=000000000000017fab19329089a1ab54e1f4ab110b1ef0c5d4f9990c2815d624 fork block hash=000000000000000121734a048a6dcd9f5184929ef6b4d60fab74612de0e5e26c (in IBD=true)
 882024-02-12T20:01:02Z [bench]   - Load block from disk: 9.61ms
 892024-02-12T20:01:02Z [bench]     - Sanity checks: 0.10ms [25.91s (0.18ms/blk)]
 902024-02-12T20:01:02Z [bench]     - Fork checks: 0.17ms [73.81s (0.52ms/blk)]
 912024-02-12T20:01:02Z [bench]       - Connect 104 transactions: 9.35ms (0.090ms/tx, 0.046ms/txin) [95.13s (0.68ms/blk)]
 922024-02-12T20:01:02Z [bench]     - Verify 202 txins: 9.39ms (0.047ms/txin) [131.29s (0.93ms/blk)]
 932024-02-12T20:01:02Z [bench]     - Write undo data: 0.19ms [75.30s (0.53ms/blk)]
 942024-02-12T20:01:02Z [bench]     - Index writing: 0.04ms [30.07s (0.21ms/blk)]
 952024-02-12T20:01:02Z [validation] BlockChecked: block hash=00000000000006a0e9cbe57335835c382bc09f11045b68dbb834495f4ae22faf state=Valid
 962024-02-12T20:01:02Z [bench]   - Connect total: 10.04ms [435.73s (3.09ms/blk)]
 972024-02-12T20:01:02Z [bench]   - Flush: 0.19ms [66.27s (0.47ms/blk)]
 982024-02-12T20:01:02Z [bench]   - Writing chainstate: 0.04ms [26.22s (0.19ms/blk)]
 992024-02-12T20:01:02Z [estimatefee] Blockpolicy estimates updated by 0 of 0 block txs, since last block 0 of 0 tracked, mempool map size 0, max target 0 from current
1002024-02-12T20:01:02Z UpdateTip: new best=00000000000006a0e9cbe57335835c382bc09f11045b68dbb834495f4ae22faf height=140921 version=0x00000001 log2_work=66.122882 tx=1282777 date='2011-08-14T12:41:24Z' progress=0.001328 cache=104.2MiB(896561txo)
1012024-02-12T20:01:02Z [bench]   - Connect postprocess: 9.22ms [150.71s (1.07ms/blk)]
1022024-02-12T20:01:02Z [bench] - Connect block: 29.10ms [768.29s (5.45ms/blk)]
1032024-02-12T20:01:02Z [validation] Enqueuing BlockConnected: block hash=00000000000006a0e9cbe57335835c382bc09f11045b68dbb834495f4ae22faf block height=140921
1042024-02-12T20:01:02Z [validation] Enqueuing UpdatedBlockTip: new block hash=00000000000006a0e9cbe57335835c382bc09f11045b68dbb834495f4ae22faf fork block hash=000000000000017fab19329089a1ab54e1f4ab110b1ef0c5d4f9990c2815d624 (in IBD=true)
1052024-02-12T20:01:02Z [validation] BlockConnected: block hash=00000000000006a0e9cbe57335835c382bc09f11045b68dbb834495f4ae22faf block height=140921
1062024-02-12T20:01:02Z [bench]   - Load block from disk: 0.34ms
1072024-02-12T20:01:02Z [bench]     - Sanity checks: 0.07ms [25.91s (0.18ms/blk)]
1082024-02-12T20:01:02Z [bench]     - Fork checks: 0.18ms [73.81s (0.52ms/blk)]
1092024-02-12T20:01:02Z [bench]       - Connect 149 transactions: 0.29ms (0.002ms/tx, 0.001ms/txin) [95.13s (0.68ms/blk)]
1102024-02-12T20:01:02Z [bench]     - Verify 210 txins: 0.30ms (0.001ms/txin) [131.29s (0.93ms/blk)]
1112024-02-12T20:01:02Z [validation] Pre-allocating up to position 0xc00000 in rev00003.dat
1122024-02-12T20:01:02Z [bench]     - Write undo data: 19.25ms [75.32s (0.53ms/blk)]
1132024-02-12T20:01:02Z [bench]     - Index writing: 0.06ms [30.07s (0.21ms/blk)]
1142024-02-12T20:01:02Z [validation] BlockChecked: block hash=000000000000052e30dd0b273b500fde21e4bee67d1f5cd05fef91a663dadfcd state=Valid
1152024-02-12T20:01:02Z [bench]   - Connect total: 19.97ms [435.75s (3.09ms/blk)]
1162024-02-12T20:01:02Z [bench]   - Flush: 0.22ms [66.27s (0.47ms/blk)]
1172024-02-12T20:01:02Z [prune] coinstatsindex limited pruning to height 140909
1182024-02-12T20:01:02Z [bench] FlushStateToDisk: find files to prune started
1192024-02-12T20:01:02Z [leveldb] WriteBatch memory usage: db=db, before=0.0MiB, after=0.0MiB
1202024-02-12T20:01:05Z [addrman] Selected [2001:8a0:c6da:9200:999f:517e:4e5a:f105]:8333 from new
1212024-02-12T20:01:05Z [net] Making network specific connection to [2001:8a0:c6da:9200:999f:517e:4e5a:f105]:8333 on ipv6.
1222024-02-12T20:01:05Z [net:debug] trying v1 connection [2001:8a0:c6da:9200:999f:517e:4e5a:f105]:8333 lastseen=99.9hrs
1232024-02-12T20:01:05Z [net] connect() to [2001:8a0:c6da:9200:999f:517e:4e5a:f105]:8333 failed: Network is unreachable (101)
1242024-02-12T20:01:07Z [prune] [normal] target=550MiB actual=388MiB diff=161MiB min_height=0 max_prune_height=140633 removed 1 blk/rev pairs
1252024-02-12T20:01:07Z [bench] FlushStateToDisk: find files to prune completed (5340.60ms)
1262024-02-12T20:01:07Z [leveldb:debug] Level-0 table [#7](/bitcoin-bitcoin/7/): started
1272024-02-12T20:01:07Z [leveldb] WriteBatch memory usage: db=index, before=5.6MiB, after=5.6MiB
1282024-02-12T20:01:07Z [bench] FlushStateToDisk: write block and undo data to disk started
1292024-02-12T20:01:07Z [bench] FlushStateToDisk: write block and undo data to disk completed (89.89ms)
1302024-02-12T20:01:07Z [bench] FlushStateToDisk: write block index to disk started
1312024-02-12T20:01:08Z [leveldb:debug] Level-0 table [#7](/bitcoin-bitcoin/7/): 5044540 bytes OK
1322024-02-12T20:01:09Z [leveldb:debug] Delete type=0 [#4](/bitcoin-bitcoin/4/)
133Killed

  02024-02-12T20:07:12Z Bitcoin Core version v26.0.0 (release build)
  12024-02-12T20:07:12Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -upnp=0
  22024-02-12T20:07:12Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -natpmp=0
  32024-02-12T20:07:12Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -discover=0
  42024-02-12T20:07:12Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -listenonion=0
  52024-02-12T20:07:12Z InitParameterInteraction: parameter interaction: -listen=0 -> setting -i2pacceptincoming=0
  62024-02-12T20:07:12Z Using the 'x86_shani(1way,2way)' SHA256 implementation
  72024-02-12T20:07:12Z Using RdSeed as an additional entropy source
  82024-02-12T20:07:12Z Using RdRand as an additional entropy source
  92024-02-12T20:07:12Z Default data directory /home/***/.bitcoin
 102024-02-12T20:07:12Z Using data directory /home/***/.bitcoin
 112024-02-12T20:07:12Z Config file: /home/***/.bitcoin/bitcoin.conf
 122024-02-12T20:07:12Z Config file arg: coinstatsindex="1"
 132024-02-12T20:07:12Z Config file arg: disablewallet="1"
 142024-02-12T20:07:12Z Config file arg: listen="0"
 152024-02-12T20:07:12Z Config file arg: prune="550"
 162024-02-12T20:07:12Z Command-line arg: debug="1"
 172024-02-12T20:07:12Z Using at most 125 automatic connections (1024 file descriptors available)
 182024-02-12T20:07:12Z Using 16 MiB out of 16 MiB requested for signature cache, able to store 524288 elements
 192024-02-12T20:07:12Z Using 16 MiB out of 16 MiB requested for script execution cache, able to store 524288 elements
 202024-02-12T20:07:12Z Script verification uses 0 additional threads
 212024-02-12T20:07:12Z Wallet disabled!
 222024-02-12T20:07:12Z scheduler thread start
 232024-02-12T20:07:12Z [http] Allowing HTTP connections from: 127.0.0.0/8 ::1/128
 242024-02-12T20:07:12Z Binding RPC on address ::1 port 8332
 252024-02-12T20:07:12Z Binding RPC on address 127.0.0.1 port 8332
 262024-02-12T20:07:12Z [http] Initialized HTTP server
 272024-02-12T20:07:12Z [http] creating work queue of depth 16
 282024-02-12T20:07:12Z [rpc] Starting RPC
 292024-02-12T20:07:12Z [rpc] Starting HTTP RPC server
 302024-02-12T20:07:12Z Using random cookie authentication.
 312024-02-12T20:07:12Z Generated RPC authentication cookie /home/***/.bitcoin/.cookie
 322024-02-12T20:07:12Z [http] Registering HTTP handler for / (exactmatch 1)
 332024-02-12T20:07:12Z [http] Registering HTTP handler for /wallet/ (exactmatch 0)
 342024-02-12T20:07:12Z [http] Starting HTTP server
 352024-02-12T20:07:12Z [http] starting 4 worker threads
 362024-02-12T20:07:12Z Using /16 prefix for IP bucketing
 372024-02-12T20:07:12Z init message: Loading P2P addresses…
 382024-02-12T20:07:12Z [http] Entering http event loop
 392024-02-12T20:07:12Z [addrman] CheckAddrman: new 10628, tried 27, total 10655 started
 402024-02-12T20:07:12Z [addrman] CheckAddrman: completed (109.33ms)
 412024-02-12T20:07:12Z Loaded 10655 addresses from peers.dat  208ms
 422024-02-12T20:07:12Z init message: Loading banlist…
 432024-02-12T20:07:12Z [net] Loaded 0 banned node addresses/subnets  9ms
 442024-02-12T20:07:12Z [net] setting try another outbound peer=false
 452024-02-12T20:07:12Z SetNetworkActive: true
 462024-02-12T20:07:12Z [estimatefee] Reading estimates: 190 buckets counting confirms up to 48 blocks
 472024-02-12T20:07:12Z [estimatefee] Reading estimates: 190 buckets counting confirms up to 12 blocks
 482024-02-12T20:07:12Z [estimatefee] Reading estimates: 190 buckets counting confirms up to 1008 blocks
 492024-02-12T20:07:12Z Cache configuration:
 502024-02-12T20:07:12Z * Using 2.0 MiB for block index database
 512024-02-12T20:07:12Z * Using 8.0 MiB for chain state database
 522024-02-12T20:07:12Z * Using 440.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
 532024-02-12T20:07:12Z init message: Loading block index…
 542024-02-12T20:07:12Z Assuming ancestors of block 00000000000000000001a0a448d6cf2546b06801389cc030b2b18c6491266815 have valid signatures.
 552024-02-12T20:07:12Z Setting nMinimumChainWork=000000000000000000000000000000000000000052b2559353df4117b7348b64
 562024-02-12T20:07:12Z Prune configured to target 550 MiB on disk for block and undo files.
 572024-02-12T20:07:12Z [leveldb] LevelDB using max_open_files=1000 (default=1000)
 582024-02-12T20:07:12Z Opening LevelDB in /home/***/.bitcoin/blocks/index
 592024-02-12T20:07:12Z [leveldb:debug] Recovering log [#6](/bitcoin-bitcoin/6/)
 602024-02-12T20:07:35Z [leveldb:debug] Level-0 table [#9](/bitcoin-bitcoin/9/): started
 612024-02-12T20:07:42Z [leveldb:debug] Level-0 table [#9](/bitcoin-bitcoin/9/): 109196585 bytes OK
 622024-02-12T20:07:42Z [leveldb:debug] Delete type=3 [#2](/bitcoin-bitcoin/2/)
 632024-02-12T20:07:42Z [leveldb:debug] Delete type=0 [#6](/bitcoin-bitcoin/6/)
 642024-02-12T20:07:42Z Opened LevelDB successfully
 652024-02-12T20:07:42Z Using obfuscation key for /home/***/.bitcoin/blocks/index: 0000000000000000
 662024-02-12T20:08:12Z [rand] Feeding 13595 bytes of dynamic environment data into RNG
 672024-02-12T20:08:20Z LoadBlockIndexDB: last block file = 3
 682024-02-12T20:08:20Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=3998, size=93962849, heights=136925...140929, time=2011-07-18...2011-08-14)
 692024-02-12T20:08:20Z Checking all blk files are present...
 702024-02-12T20:08:22Z LoadBlockIndexDB(): Block files have previously been pruned
 712024-02-12T20:08:22Z [blockstorage] Prune: UnlinkPrunedFiles deleted blk/rev (00000)
 722024-02-12T20:08:26Z Initializing chainstate Chainstate [ibd] @ height -1 (null)
 732024-02-12T20:08:26Z [leveldb] LevelDB using max_open_files=1000 (default=1000)
 742024-02-12T20:08:26Z Opening LevelDB in /home/***/.bitcoin/chainstate
 752024-02-12T20:08:26Z [leveldb:debug] Recovering log [#6](/bitcoin-bitcoin/6/)
 762024-02-12T20:08:26Z [leveldb:debug] Delete type=0 [#6](/bitcoin-bitcoin/6/)
 772024-02-12T20:08:26Z [leveldb:debug] Delete type=3 [#4](/bitcoin-bitcoin/4/)
 782024-02-12T20:08:26Z Opened LevelDB successfully
 792024-02-12T20:08:26Z Using obfuscation key for /home/***/.bitcoin/chainstate: ebcd358dfcc3ace2
 802024-02-12T20:08:26Z [leveldb] LevelDB using max_open_files=1000 (default=1000)
 812024-02-12T20:08:26Z Opening LevelDB in /home/***/.bitcoin/chainstate
 822024-02-12T20:08:26Z [leveldb:debug] Recovering log [#8](/bitcoin-bitcoin/8/)
 832024-02-12T20:08:26Z [leveldb:debug] Delete type=0 [#8](/bitcoin-bitcoin/8/)
 842024-02-12T20:08:26Z [leveldb:debug] Delete type=3 [#7](/bitcoin-bitcoin/7/)
 852024-02-12T20:08:26Z Opened LevelDB successfully
 862024-02-12T20:08:26Z Using obfuscation key for /home/***/.bitcoin/chainstate: ebcd358dfcc3ace2
 872024-02-12T20:08:26Z [Chainstate [ibd] @ height -1 (null)] resized coinsdb cache to 8.0 MiB
 882024-02-12T20:08:26Z [Chainstate [ibd] @ height -1 (null)] resized coinstip cache to 440.0 MiB
 892024-02-12T20:08:26Z init message: Verifying blocks…
 902024-02-12T20:08:26Z  block index           73860ms
 912024-02-12T20:08:26Z [leveldb] LevelDB using max_open_files=1000 (default=1000)
 922024-02-12T20:08:26Z Opening LevelDB in /home/***/.bitcoin/indexes/coinstats/db
 932024-02-12T20:08:26Z [leveldb:debug] Recovering log [#898](/bitcoin-bitcoin/898/)
 942024-02-12T20:08:26Z [leveldb:debug] Level-0 table [#901](/bitcoin-bitcoin/901/): started
 952024-02-12T20:08:26Z [leveldb:debug] Level-0 table [#901](/bitcoin-bitcoin/901/): 44314 bytes OK
 962024-02-12T20:08:26Z [leveldb:debug] Delete type=0 [#898](/bitcoin-bitcoin/898/)
 972024-02-12T20:08:26Z [leveldb:debug] Delete type=3 [#2](/bitcoin-bitcoin/2/)
 982024-02-12T20:08:26Z Opened LevelDB successfully
 992024-02-12T20:08:26Z Using obfuscation key for /home/***/.bitcoin/indexes/coinstats/db: 0000000000000000
1002024-02-12T20:08:26Z init message: Pruning blockstore…
1012024-02-12T20:08:26Z [bench] FlushStateToDisk: find files to prune started
1022024-02-12T20:08:26Z [bench] FlushStateToDisk: find files to prune completed (0.00ms)
1032024-02-12T20:08:26Z initload thread start
104node/interfaces.cpp:525 getBlockHash: Assertion `chainman().ActiveChain()[height]' failed.
105Aborted (core dumped)

Note that the machine does not ever actually have 1 whole GB of available memory, even after a reboot:

0MiB Mem :    844.0 total,    124.2 free,    230.7 used,    489.2 buff/cache

maflcko commented at 12:57 pm on February 13, 2024: member

This happens because the program aborts after writing the coinstats index, but before writing the block index.

I tried to reproduce this by adding an abort, but it didn’t work, so far:

 0diff --git a/src/validation.cpp b/src/validation.cpp
 1index 0552bde665..6493e3cb05 100644
 2--- a/src/validation.cpp
 3+++ b/src/validation.cpp
 4@@ -2652,6 +2652,7 @@ bool Chainstate::FlushStateToDisk(
 5             // Then update all block file information (which may refer to block and undo files).
 6             {
 7                 LOG_TIME_MILLIS_WITH_CATEGORY("write block index to disk", BCLog::BENCH);
 8+                assert(false);
 9 
10                 if (!m_blockman.WriteBlockIndexDB()) {
11                     return FatalError(m_chainman.GetNotifications(), state, "Failed to write to block index database");

magks commented at 11:43 am on February 19, 2024: none

Just want to add that on a Raspberry PI 3B with 1 GB memory I also was running into a similiar issue during IBD however it was on testnet rather than mainnet that I got around by creating a swapfile. The PI was also running Unbound and had about 500MB free memory upon starting the IBD. The kernel would panic and upon rebooting dmesg revealed the OOM errors. So I created a swapfile and monitored with free. Indeed the memory filled and would occasionally dip into swap. I can’t recall now what the maximum usage was (can’t imagine it being more than 100-200MB), but I had created a large swapfile of 8GB to ensure there would be no issue. I had all indexes turned on, listen off, connected manually to just another testnet node in the LAN.
mzumsande commented at 10:28 pm on July 3, 2024: contributor
I could reproduce this, both on 26.0 and master, by simulating the oom by adding the assert after the block index was flushed to disk (but before the utxo set was).
0diff --git a/src/validation.cpp b/src/validation.cpp 1index fe466fe5aa..352c802576 100644 2--- a/src/validation.cpp 3+++ b/src/validation.cpp 4@@ -2891,6 +2891,7 @@ bool Chainstate::FlushStateToDisk( 5 if (!m_blockman.WriteBlockIndexDB()) { 6 return FatalError(m_chainman.GetNotifications(), state, _("Failed to write to block index database.")); 7 } 8+ assert(false); 9 } 10 // Finally remove any pruned files 11 if (fFlushForPrune) {
With this, I’d sync signet from genesis with an index (blockfilter or coinstats) until it crashes after the first prune event / flush. During the next startup, I get the chainman().ActiveChain()[height] failed assertion.

The reason for this is that we don’t connect the genesis block in this scenario. Since we’ve loaded the block tree db we know about the header chain, but with the genesis block being pruned, and no utxo set available, FindMostWorkChain() won’t find a block that can be connected.

As a consequence, getBlockHash which is called with height 0 in GetSummary(), asserts. However, even if we avoid that by restarting without an index, the outcome wouldn’t be better: We’d wait forever here: https://github.com/bitcoin/bitcoin/blob/3714692644f45808a6480525abc36870aeee1de4/src/init.cpp#L1783-L1785 for the genesis block to be connected, which will never happen.

There is nothing we can do to recover in this situation because we have no utxo set and the initial blocks have been pruned. My suggestion would be to make the “initload” thread return early if, after ImportBlocks() finished, we don’t have at least the genesis block connected, and shutdown with an InitError telling the user to reindex.

Contributors
vostrnad willcl-ark maflcko magks mzumsande

Labels
Bug UTXO Db and Indexes Data corruption