LevelDB corruption on macOS Big Sur beta

Sjors commented at 1:25 pm on June 29, 2020: member

For more problems with macOS Big Sur (beta) see #19406.

I first ran the binary release v0.20.0 and it synced at least a few dozen blocks without issue (around height 604,000).

Depends built with: make NO_ZMQ=1 DEBUG=1 NO_QT=1 NO_WALLET=1 Configured with: ./configure --enable-debug --enable-werror --prefix=/Users/sjors/dev/bitcoin/depends/x86_64-apple-darwin20.0.0

On an existing node, bitcoind crashes at different heights, and often for different ldb files. Example log:

 0src/bitcoind -debug=leveldb
 1
 22020-06-29T12:12:52Z UpdateTip: new best=000000000000000000106cbbe535af482a10094e546425de969cc83d42c1ab54 height=604383 version=0x3fffe000 log2_work=91.348713 tx=475833455 date='2019-11-18T16:43:29Z' progress=0.877056 cache=29.6MiB(216045txo)
 32020-06-29T12:12:52Z leveldb: Generated table [#707286](/bitcoin-bitcoin/707286/)@3: 36594 keys, 2171204 bytes
 42020-06-29T12:12:52Z Pre-allocating up to position 0x4000000 in blk01868.dat
 52020-06-29T12:12:52Z leveldb: Generated table [#707287](/bitcoin-bitcoin/707287/)@3: 36898 keys, 2171437 bytes
 62020-06-29T12:12:52Z leveldb: Generated table [#707288](/bitcoin-bitcoin/707288/)@3: 35879 keys, 2170291 bytes
 72020-06-29T12:12:52Z leveldb: Generated table [#707289](/bitcoin-bitcoin/707289/)@3: 36289 keys, 2170993 bytes
 82020-06-29T12:12:52Z leveldb: Generated table [#707290](/bitcoin-bitcoin/707290/)@3: 36425 keys, 2171307 bytes
 92020-06-29T12:12:52Z leveldb: Generated table [#707291](/bitcoin-bitcoin/707291/)@3: 36176 keys, 2170300 bytes
102020-06-29T12:12:52Z leveldb: Generated table [#707292](/bitcoin-bitcoin/707292/)@3: 37432 keys, 2172172 bytes
112020-06-29T12:12:52Z leveldb: Generated table [#707293](/bitcoin-bitcoin/707293/)@3: 13346 keys, 818093 bytes
122020-06-29T12:12:52Z leveldb: Compacted 1@3 + 12@4 files => 22528033 bytes
132020-06-29T12:12:53Z leveldb: compacted to: files[ 0 0 3 167 1703 0 0 ]
142020-06-29T12:12:53Z leveldb: Delete type=2 [#704114](/bitcoin-bitcoin/704114/)
152020-06-29T12:12:53Z leveldb: Delete type=2 [#704115](/bitcoin-bitcoin/704115/)
162020-06-29T12:12:53Z leveldb: Delete type=2 [#705541](/bitcoin-bitcoin/705541/)
172020-06-29T12:12:53Z leveldb: Delete type=2 [#705542](/bitcoin-bitcoin/705542/)
182020-06-29T12:12:53Z leveldb: Delete type=2 [#704112](/bitcoin-bitcoin/704112/)
192020-06-29T12:12:53Z leveldb: Delete type=2 [#704113](/bitcoin-bitcoin/704113/)
202020-06-29T12:12:53Z leveldb: Delete type=2 [#705543](/bitcoin-bitcoin/705543/)
212020-06-29T12:12:53Z leveldb: Delete type=2 [#703169](/bitcoin-bitcoin/703169/)
222020-06-29T12:12:53Z leveldb: Delete type=2 [#703172](/bitcoin-bitcoin/703172/)
232020-06-29T12:12:53Z leveldb: Delete type=2 [#703173](/bitcoin-bitcoin/703173/)
242020-06-29T12:12:53Z leveldb: Delete type=2 [#703171](/bitcoin-bitcoin/703171/)
252020-06-29T12:12:53Z leveldb: Delete type=2 [#703170](/bitcoin-bitcoin/703170/)
262020-06-29T12:12:53Z leveldb: Delete type=2 [#705749](/bitcoin-bitcoin/705749/)
272020-06-29T12:12:53Z leveldb: Compacting 1@3 + 10@4 files
282020-06-29T12:12:53Z LevelDB read failure: Corruption: block checksum mismatch: /Users/sjors/Library/Application Support/Bitcoin/chainstate/705763.ldb
292020-06-29T12:12:53Z Fatal LevelDB error: Corruption: block checksum mismatch: /Users/sjors/Library/Application Support/Bitcoin/chainstate/705763.ldb
302020-06-29T12:12:53Z You can use -debug=leveldb to get more complete diagnostic messages
312020-06-29T12:12:53Z Error: Error reading from database, shutting down.

When building without depends or using the 0.20 release, afterwards, results in the same errors, so most likely those files are actually corrupt.

I’m syncing a node (built with depends) from scratch. So far no crash (height=225,000).

Unless someone else sees this (which could suggest an upgrade issue), I’ll close this in a while.

Sjors added the label Bug on Jun 29, 2020

MarcoFalke commented at 1:41 pm on June 29, 2020: member

Your hardware is probably broken. See #19406 (comment)

Sjors commented at 1:48 pm on June 29, 2020: member

That seems a bit too coincidental. A compiler crash and the first IBD failure in ages, exactly when I upgrade to Big Sur. It’s an old MacBook, but not unused; it been serving as third “monitor” for more than a year without anything like this happening.

MarcoFalke commented at 2:45 pm on June 29, 2020: member

It would be hard to detect hardware issues on a “third monitor”. My suggestion would be to test the hardware. See my default reply template:

Bitcoin Core makes heavy use of CPU, RAM and disk IO. Hardware defects might only become visible when running Bitcoin Core. You might want to check your hardware for defects.

memtest86 to check your RAM
to check the CPU behaviour under load, use linpack or Prime95
to test your storage device use smartctl or CrystalDiskInfo

Source: https://bitcoin.stackexchange.com/a/12206

Sjors commented at 3:04 pm on June 29, 2020: member

I put “monitor” in quotes, because it’s actually a shared desktop (via Synergy). It’s true that it doesn’t often run a max CPU capacity, so I’ll run a bunch of those checks.

The most important thing to rule out is a broken compiler that produces bugs that our test suite doesn’t detect.

Sjors commented at 7:58 pm on June 29, 2020: member

I did another recompile and this time it’s happily syncing, on the existing “corrupt” data directory. Weird. Let’s see if it can sync 6 months of blocks until the tip.

Update: it broke at height 620,783

Sjors commented at 5:30 pm on June 30, 2020: member

Fun fact: it’s actually not that easy to run memtest86 on a Mac anymore. Single user mode is now so restricted, you can’t see /usr/local/bin from there, and in normal mode /bin or /sbin are read-only. I couldn’t figure out where to even put the binary so that it can still be found.

Update: I continued IBD using the release binary and it finished the remaining 25K blocks without issue.

Sjors commented at 7:15 pm on July 1, 2020: member

I’m going to blame the machine or the entire OS for now. Even syncing a node from scratch using a release binary sometimes results in this error.

Sjors closed this on Jul 1, 2020

Sjors commented at 8:42 pm on July 1, 2020: member

Finally the mac diagnotic tool complained about a potential memory error (PPM002). It didn’t the last couple of times, so perhaps it needs to be hot. Anyway, I’m decommissioning this machine and will try a future beta on my second oldest Mac instead…

MarcoFalke locked this on Feb 15, 2022

LevelDB corruption on macOS Big Sur beta #19410