resync everything when out of memory

PolarGo commented at 5:46 pm on November 22, 2017: none

SITUATION: After starting a fresh installed bitcoind it starts to sync blocks. After several hours bitcoind abort with error “out of memory”. The machine I’m using has 1 GB RAM. I had no swap partition active.

The process runs under runit; when it aborts it gets automatically restarted.

Now bitcoind starts re-syncing from the beginning, instead of starting from where it left off.

PROBLEM: That creates a lot of unnecessary traffic (many GB) and unnecessary time delay (many hours).

EXPECTED: After an error (in this case out of memory), bitcoind should start from where it left of without wasting time and IP bandwidth.

After I realized this, I added swap space, but still, everything has to be resynced over the network.

bitcoin 0.15 OS: void linux RAM: 1GB CPU: via-c7-D i686

TheBlueMatt commented at 7:30 pm on November 22, 2017: member

Note that in this case things aren’t “resynced over the network” they are loaded back from disk. Sadly, not a ton do be done about this - trying to flush as we go would limit sync speed. Ideally we’d be looking at the memory pressure in systems to flush when necessary, or at least partially flush, but that’s a rather large change.

On November 22, 2017 9:46:39 AM PST, PolarGo notifications@github.com wrote:

SITUATION: After starting a fresh installed bitcoind it starts to sync blocks. After several hours bitcoind abort with error “out of memory”. The machine I’m using has 1 GB RAM. I had no swap partition active.

The process runs under runit; when it aborts it gets automatically restarted.

Now bitcoind starts re-syncing from the beginning, instead of starting from where it left off.

PROBLEM: That creates a lot of unnecessary traffic (many GB) and unnecessary time delay (many hours).

EXPECTED: After an error (in this case out of memory), bitcoind should start from where it left of without wasting time and IP bandwidth.

After I realized this, I added swap space, but still, everything has to be resynced over the network.

bitcoin 0.15 OS: void linux RAM: 1GB CPU: via-c7-D i686

– You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/issues/11752

jonasschnelli commented at 7:30 pm on November 22, 2017: contributor

Starting again from the scratch has probably to do with corruptions or un-flushed data. Check or pastebin your debug.log for more details.

If you have only 1GB of memory, try reduce some parameters. -dbcache=50 -maxmempool=10

PolarGo commented at 7:55 pm on November 22, 2017: none

Ok, maybe it didn’t start from scratch; but it looks like it went back in time for more than a year (2013-05-22 => 2012-03-11).

I wonder what the best course of action is: Just restart the process when it runs out of memory and don’t add swap space. Or add swap space, but that will probably make things much slower?

Once all blocks are synced (and assumed that I run the node continuously) will it still run out of memory or is this just a special thing during initial syncing?

2017-11-22 15:39:52 Error: Out of memory. Terminating. […] 2017-11-22 15:39:55 Bitcoin version v0.15.1.0-g7b57bc9 2017-11-22 15:39:55 Assuming ancestors of block 0000000000000000003b9ce759c2a087d52abc4266f8f4ebd6d768b89defa50a have valid signatures. 2017-11-22 15:39:55 Setting nMinimumChainWork=000000000000000000000000000000000000000000723d3581fe1bd55373540a […] 2017-11-22 15:39:58 Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements 2017-11-22 15:39:58 Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements 2017-11-22 15:39:58 Using 0 threads for script verification 2017-11-22 15:39:58 scheduler thread start […] 2017-11-22 15:39:58 * Using 440.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space) 2017-11-22 15:39:58 init message: Loading block index… 2017-11-22 15:39:58 Opening LevelDB in /data/bitcoin/blocks/index 2017-11-22 15:40:08 Opened LevelDB successfully 2017-11-22 15:40:08 Using obfuscation key for /data/bitcoin/blocks/index: 0000000000000000 2017-11-22 15:41:17 LoadBlockIndexDB: last block file = 62 2017-11-22 15:41:17 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=257, size=39918830, heights=237412…237699, time=2013-05-22…2013-05-24) 2017-11-22 15:41:17 Checking all blk files are present… 2017-11-22 15:41:17 LoadBlockIndexDB: transaction index disabled 2017-11-22 15:41:17 Opening LevelDB in /data/bitcoin/chainstate 2017-11-22 15:41:30 Opened LevelDB successfully 2017-11-22 15:41:32 Using obfuscation key for /data/bitcoin/chainstate: d337b51226b6028e 2017-11-22 15:41:32 Loaded best chain: hashBestChain=000000000000057da116560f22fbfbe62f365c7b92282a88894065566141a474 height=170672 date=2012-03-11 14:54:17 progress=0.009494 2017-11-22 15:41:32 init message: Rewinding blocks… 2017-11-22 15:41:43 init message: Verifying blocks… 2017-11-22 15:41:43 Verifying last 6 blocks at level 3 2017-11-22 15:41:43 [0%]…[16%]…[33%]…[50%]…[66%]…[83%]…[99%]…[DONE]. 2017-11-22 15:41:52 No coin database inconsistencies in last 7 blocks (329 transactions) 2017-11-22 15:41:52 block index 113584ms 2017-11-22 15:41:52 init message: Loading wallet… 2017-11-22 15:41:52 nFileVersion = 150100 2017-11-22 15:41:52 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total 2017-11-22 15:41:53 wallet 1432ms 2017-11-22 15:41:53 setKeyPool.size() = 2000 2017-11-22 15:41:53 mapWallet.size() = 0 2017-11-22 15:41:53 mapAddressBook.size() = 1 2017-11-22 15:41:53 mapBlockIndex.size() = 495578 2017-11-22 15:41:53 nBestHeight = 170672 […] 2017-11-22 15:41:53 torcontrol thread start 2017-11-22 15:41:56 Bound to [::]:8333 2017-11-22 15:41:56 Bound to 0.0.0.0:8333 2017-11-22 15:41:56 init message: Loading P2P addresses… 2017-11-22 15:41:57 UpdateTip: new best=00000000000000059e8f079529be7ef2b0478b64f23bc35f4ff869adfc22e7b1 height=170673 version=0x00000001 log2_work=67.818878 tx=2595833 date=’2012-03-11 15:08:00’ progress=0.009495 cache=0.1MiB(518txo)

PolarGo commented at 7:57 pm on November 22, 2017: none

PS: and that is the last line before “out of memory”:

2017-11-22 15:39:52 UpdateTip: new best=00000000000000156d9f3470de536cf7163ff34a698831ebfd8de211ca192eb5 height=246038 version=0x00000002 log2_work=70.679583 tx=20547163 date=‘2013-07-11 13:41:25’ progress=0.075154 cache=624.4MiB(6466675txo)

jonasschnelli commented at 7:58 pm on November 22, 2017: contributor

Out of memory can cause corruptions. Just restating with runit is not a solution. What you need to do is reduce memory usage on your system. Remove all unnecessary applications and optimise bitcoin core.

Create a bitcoin.conf and add something like that into you config

0dbcache=20
1maxconnections=3
2par=2
3maxmempool=50

This should allow you to no longer run out of memory.

PolarGo commented at 8:01 pm on November 22, 2017: none

ok, thanks, will try that.

however, it’s not hard for bitcoind to determine the amount of RAM in the system and adjust these values to some workable default values (like ones you just mentioned).

TheBlueMatt commented at 8:01 pm on November 22, 2017: member

Out of memory should absolutely never cause corruption! In case you run out of memory you will see it “resync” after restarting because of unflushed data which needs to be reconnected. As @jonasschnelli points out, try things to reduce memory usage. For further questions, please see the Bitcoin Stack Exchange.

On November 22, 2017 11:58:57 AM PST, Jonas Schnelli notifications@github.com wrote:

Out of memory can cause corruptions. Just restating with runit is not a solution. What you need to do is reduce memory usage on your system. Remove all unnecessary applications and optimise bitcoin core.

Create a bitcoin.conf and add something like that into you config
0dbcache=20
1maxconnections=3
2par=2
3maxmempool=50
This should allow you to no longer run out of memory.

– You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/issues/11752#issuecomment-346458242

laanwj added the label Questions and Help on Nov 23, 2017

laanwj closed this on Nov 23, 2017

MarcoFalke locked this on Sep 8, 2021

resync everything when out of memory #11752