bitcoin-qt out-of-virtual-address-space problem #2353

EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ProcessMessages()

EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ThreadMessageHandler()

terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted

The only thing I did was to wipe everything clean and start version 0.8.0. It started to download the block chain for a while and when I got back home it was dead. Is there anything I can do to prepare myself so I can gather more information if it happens again? Thinking of crash dumps, logs etc.

bad_alloc means it is running out of memory. How much memory do you have? What OS?

magnus@desktop|21:59|~/src/bitcoin ((v0.8.0)) $ cat /proc/meminfo MemTotal: 16445204 kB MemFree: 4247964 kB Buffers: 341888 kB Cached: 8300268 kB SwapCached: 0 kB Active: 10045068 kB Inactive: 1818152 kB Active(anon): 2951020 kB Inactive(anon): 303216 kB Active(file): 7094048 kB Inactive(file): 1514936 kB Unevictable: 80 kB Mlocked: 80 kB HighTotal: 15751096 kB HighFree: 4179560 kB LowTotal: 694108 kB LowFree: 68404 kB SwapTotal: 16659452 kB SwapFree: 16659452 kB Dirty: 3440 kB Writeback: 0 kB AnonPages: 3221360 kB Mapped: 233656 kB Shmem: 33172 kB Slab: 236588 kB SReclaimable: 200160 kB SUnreclaim: 36428 kB KernelStack: 5144 kB PageTables: 31576 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 24882052 kB Committed_AS: 7121976 kB VmallocTotal: 122880 kB VmallocUsed: 63688 kB VmallocChunk: 52276 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 98296 kB DirectMap2M: 815104 kB

I'm running Lubuntu 12.04, 32 bit.

This is how bitcoin runs out of memory (virtual address space) while downloading blocks:

Whenever a node receives a block that it cannot link to the chain, it stores it in memory in mapOrphanBlocks. There is no limit to the size of that data. When a node is downloading blocks, it may download and keep in memory many blocks that it can't currently link to the chain. In particular, if your node connects to another node which is also downloading blocks and which is further along in the download, the other node will spam your node with an "inv" message for every new block that it downloads (no, the logic in CBlock::AcceptBlock does not prevent it), and your node will download and store those blocks in mapOrphanBlocks. This can occur with multiple connected nodes simultaneously which are at different points in the download.

I encountered this problem several times with 0.8.0rc1. With the current size of the blockchain, it is not hard to exhaust the virtual address space (at most 4GiB) in 32-bit builds, which includes all Windows builds.

The fix could proceed in one of two directions: (1) store orphan blocks on disk instead of in memory, or (2) don't keep an unbounded set of orphan blocks. For (2), the ideal would be to prevent downloading of too many orphans rather than discarding orphans that have already been downloaded, which is wasteful. The goal would be to download blocks roughly in order. One idea is to make use of "getheaders" during block download. Headers would be downloaded and linked together, and blocks would only be downloaded if we already know based on headers that they are not too far away from the chain of blocks that we already have.

Headers-first synchronization (being worked on) removes the concept of orphan blocks entirely, as a block is only downloaded when its ancestry is already known, so it can always be saved on disk.

Since 0.9.0 the number of orphan blocks that is stored has been restricted, so this immediate issue should be fixed.