Core dump on bitcoind 0.15.1 with -mempoolexpiry=24 #12230

<!-- Describe the issue -->

I upgraded my full node to bitcoin 0.15.1 last night from 0.14.1 and saw that less than 24 hours later I got a core dump:

/usr/local/bin/bitcoind -mempoolexpiry=24 bitcoind: net_processing.cpp:1096: void ProcessGetData(CNode, const Consensus::Params&, CConnman, const std::atomic<bool>&): Assertion `!"cannot load block from disk"' failed. Aborted (core dumped)

The previous version (0.14.1) would run indefinitely without ever crashing. However, I just started using -mempoolexpiry=24 option to cut down on memory usage and I'm wondering if this change causes the issue. (So 2 things changed that resulted in core dump, version and additional argument).

<!--- What behavior did you expect? -->

Should not crash.

<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

Only saw this once.

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

Binary distribution of 0.15.1 for linux 64 bit

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

x86_64 - AMD A4-4000 APU with Radeon(tm) HD Graphics

Assertion `!"cannot load block from disk"' failed.

Historically, this has almost always implied hardware instability - please do a memtest, prime95/linpack/similar CPU test and some kind of HDD SMART test to verify your hardware is reasonably stable. It seems likely performance changes in 0.15 resulted in increased CPU/memory stress exposing more unstable hardware than previous releases.

I don't think this is hardware. I moved my hard drive into a different machine. Same problem. Crashed twice today 30 minutes apart from each other. Is it safe to move back to 14.1 after having run 15.1? I noticed some database upgrade message in the logs.

What if you don't use -mempoolexpiry=24 option? Are you still getting crashes?

@randyrossi the datadir is corrupted, so moving it to a new computer will not un-corrupt it. It seems likely the first computer corrupted it, and it could be a bug, but seems more likely to be a hardware instability on that first machine. @bolekC that should have absolutely nothing to do with a "cannot load block from disk" error.

@TheBlueMatt Sure it "should have nothing to do with" that. Anyway I think it should be easy to test if that is not causing a problem. Some tricky interactions potentially possible.

@bolekC given the issue is data corruption, the error is incredibly likely to be nondeterministic, much better to first do a hardware test (and confirm if the error is reproducible) before chaing your tail blaming random options for nondeterministic errors. Note further that I'm much more interested in hardware test results from @randyrossi than other errors for a reason - we've seen increased data corruption errors on 0.15.1, and while it seems much more likely that this is due to performance improvements putting more strain on hardware (which we've seen in the past on several releases), its also possible its due to some database changes which have resulted in actual instability.

@bolekC I removed the -mempoolexpiry=24 option and it still crashed (twice) on the new hardware

It's likely data corruption that 14.1 doesn't care about but 15.1 does then (14.1 remains stable). I didn't have time to run a HW test on previous HW. Will do that a.s.a.p.