Core dump on bitcoind 0.15.1 with -mempoolexpiry=24

randyrossi commented at 8:33 pm on January 19, 2018: none

I upgraded my full node to bitcoin 0.15.1 last night from 0.14.1 and saw that less than 24 hours later I got a core dump:

/usr/local/bin/bitcoind -mempoolexpiry=24 bitcoind: net_processing.cpp:1096: void ProcessGetData(CNode*, const Consensus::Params&, CConnman*, const std::atomic&): Assertion `!“cannot load block from disk”’ failed. Aborted (core dumped)

The previous version (0.14.1) would run indefinitely without ever crashing. However, I just started using -mempoolexpiry=24 option to cut down on memory usage and I’m wondering if this change causes the issue. (So 2 things changed that resulted in core dump, version and additional argument).

Should not crash.

Only saw this once.

Binary distribution of 0.15.1 for linux 64 bit

x86_64 - AMD A4-4000 APU with Radeon(tm) HD Graphics

TheBlueMatt commented at 10:15 pm on January 19, 2018: member

Assertion `!“cannot load block from disk”’ failed.

Historically, this has almost always implied hardware instability - please do a memtest, prime95/linpack/similar CPU test and some kind of HDD SMART test to verify your hardware is reasonably stable. It seems likely performance changes in 0.15 resulted in increased CPU/memory stress exposing more unstable hardware than previous releases.

randyrossi commented at 3:17 am on January 21, 2018: none

I don’t think this is hardware. I moved my hard drive into a different machine. Same problem. Crashed twice today 30 minutes apart from each other. Is it safe to move back to 14.1 after having run 15.1? I noticed some database upgrade message in the logs.

bolekC commented at 9:21 pm on January 21, 2018: none

What if you don’t use -mempoolexpiry=24 option? Are you still getting crashes?

TheBlueMatt commented at 9:24 pm on January 21, 2018: member

@randyrossi the datadir is corrupted, so moving it to a new computer will not un-corrupt it. It seems likely the first computer corrupted it, and it could be a bug, but seems more likely to be a hardware instability on that first machine. @bolekC that should have absolutely nothing to do with a “cannot load block from disk” error.

MarcoFalke added the label Questions and Help on Jan 21, 2018

MarcoFalke added the label Data corruption on Jan 21, 2018

bolekC commented at 9:36 pm on January 21, 2018: none

@TheBlueMatt Sure it “should have nothing to do with” that. Anyway I think it should be easy to test if that is not causing a problem. Some tricky interactions potentially possible.

TheBlueMatt commented at 9:41 pm on January 21, 2018: member

@bolekC given the issue is data corruption, the error is incredibly likely to be nondeterministic, much better to first do a hardware test (and confirm if the error is reproducible) before chaing your tail blaming random options for nondeterministic errors. Note further that I’m much more interested in hardware test results from @randyrossi than other errors for a reason - we’ve seen increased data corruption errors on 0.15.1, and while it seems much more likely that this is due to performance improvements putting more strain on hardware (which we’ve seen in the past on several releases), its also possible its due to some database changes which have resulted in actual instability.

randyrossi commented at 10:55 pm on January 21, 2018: none

@bolekC I removed the -mempoolexpiry=24 option and it still crashed (twice) on the new hardware

It’s likely data corruption that 14.1 doesn’t care about but 15.1 does then (14.1 remains stable). I didn’t have time to run a HW test on previous HW. Will do that a.s.a.p.

fanquake closed this on Mar 6, 2018

MarcoFalke locked this on Sep 8, 2021

Core dump on bitcoind 0.15.1 with -mempoolexpiry=24 #12230