Fatal LevelDB error: Corruption: block checksum mismatch on Linux ext4 SATA SSDs

cryptoquick commented at 8:27 pm on August 21, 2024: none

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

I’ve been struggling with the errors mentioned here for a little while now: #30159 (comment)

I’m running this on a desktop machine, not in the cloud, and on external SATA ext4 disks because my main system drive is formatted with Btrfs, which doesn’t work very well with key-value databases, I’ve found.

Expected behaviour

I get this error during IBD and also during reindexing.

Steps to reproduce

This the command I’m running:

bitcoind -server -txindex=1 -datadir=/mnt/ThreeEight/bitcoind-mainnet -rpccookiefile=/mnt/ThreeEight/bitcoind-mainnet/.cookie

Relevant log output

02024-08-21T19:18:30Z UpdateTip: new best=0000000000000000000187d6a944e562d55881b738db06f0580d138c6af52754 height=856674 version=0x2192a000 log2_work=95.095637 tx=1058965890 date='2024-08-14T01:14:02Z' progress=0.995862 cache=253.5MiB(2160229txo)
12024-08-21T19:18:30Z UpdateTip: new best=0000000000000000000197cc7a6d13c63cd086ca46372cc94577ff62fa56d568 height=856675 version=0x3e000000 log2_work=95.095650 tx=1058971585 date='2024-08-14T01:18:33Z' progress=0.995863 cache=253.8MiB(2162814txo)
22024-08-21T19:18:30Z LevelDB read failure: Corruption: block checksum mismatch: /mnt/ThreeEight/bitcoind-mainnet/chainstate/873236.ldb
32024-08-21T19:18:30Z Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/ThreeEight/bitcoind-mainnet/chainstate/873236.ldb
42024-08-21T19:18:30Z You can use -debug=leveldb to get more complete diagnostic messages
52024-08-21T19:18:30Z Error: Error reading from database, shutting down.
6Error: Error reading from database, shutting down.
72024-08-21T19:18:30Z Error reading from database: Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/ThreeEight/bitcoind-mainnet/chainstate/873236.ldb
8fish: Job 1, 'bitcoind -server -txindex=1 -da…' terminated by signal SIGABRT (Abort)

How did you obtain Bitcoin Core

Package manager

What version of Bitcoin Core are you using?

v27.1.0

Operating system and version

Arch Linux

Machine specifications

AMD 5950 CPU 128GB DDR4 4TB NVMe for OS 2TB & 3.8 TB 2.5" SATA SSDs

sipa commented at 9:16 pm on August 21, 2024: member

This looks like a LevelDB corruption inside the txindex index. Does -reindex wipe those? If not, that would explain why a reindex doesn’t fix the situation.

maflcko commented at 6:40 am on August 22, 2024: member

This looks like a LevelDB corruption inside the txindex index.

Are you sure, to me the error message reads Error reading from database: Fatal LevelDB error: Corruption: block checksum mismatch: $DATADIR/chainstate/873236.ldb, whereas a index corruption should happen in $DATADIR/indexes/, according to https://github.com/bitcoin/bitcoin/blob/master/doc/files.md#data-directory-layout, no?

Btrfs, which doesn’t work very well with key-value databases, I’ve found.

Are there any observable downsides?

on external SATA ext4 disks

What is the exact setup and connection, given that you are using several storage units?

Without further information, my guess would be that the external USB cable (or external connection) is flaky, but this is just a blind guess.

maflcko commented at 6:47 am on August 22, 2024: member

Bitcoin Core makes heavy use of CPU, RAM and storage IO. Hardware defects might only become visible when running Bitcoin Core. You might want to check your hardware for defects.

Use software such as memtest86 to check your RAM.
Use software such as linpack, or Prime95 to check the CPU behaviour under load.
Use software such as smartctl, fsck, badblocks, or CrystalDiskInfo to test your storage device use.

Source: https://bitcoin.stackexchange.com/a/12206

maflcko added the label Block storage on Aug 22, 2024

maflcko added the label Data corruption on Aug 22, 2024

maflcko added the label Questions and Help on Aug 22, 2024

sipa commented at 9:34 am on August 22, 2024: member

@maflcko Ah yes, it is the version here that had a corruption in the txindex: https://bitcoin.stackexchange.com/q/123999/208

cryptoquick commented at 11:00 am on August 22, 2024: none

Apologies, I meant to say internal SATA drives. There’s no USB enclosure, they’re inside the case and plugged into SATA 3.

I tried upgrading the BIOS. I’ll see if that helps.

cryptoquick commented at 8:40 pm on August 22, 2024: none

I ran into the error earlier on in the IBD this time. It seems like there’s no rhyme or reason as to why this is occurring.

 02024-08-22T12:58:13Z UpdateTip: new best=0000000000000000011a06e4b1a3c497d247f785a4899e44ae800a9236438e16 height=424472 version=0x20000000 log2_work=85.108754 tx=148148770 date='2016-08-10T00:15:52Z' progress=0.137300 cache=124.8MiB(1146194txo)
 12024-08-22T12:58:13Z UpdateTip: new best=0000000000000000003a58cdf5401248a1330480a1c9b99440a5f974fb61ce17 height=424473 version=0x30000000 log2_work=85.108784 tx=148149386 date='2016-08-10T00:19:04Z' progress=0.137301 cache=124.8MiB(1144515txo)
 22024-08-22T12:58:13Z Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/ThreeEight/bitcoind-mainnet/indexes/txindex/052774.ldb
 32024-08-22T12:58:13Z You can use -debug=leveldb to get more complete diagnostic messages
 42024-08-22T12:58:13Z
 5
 6************************
 7EXCEPTION: 15dbwrapper_error
 8Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/ThreeEight/bitcoind-mainnet/indexes/txindex/052774.ldb
 9bitcoin in scheduler
10
11
12
13************************
14EXCEPTION: 15dbwrapper_error
15Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/ThreeEight/bitcoind-mainnet/indexes/txindex/052774.ldb
16bitcoin in scheduler
17
18terminate called after throwing an instance of 'dbwrapper_error'
19  what():  Fatal LevelDB error: Corruption: block checksum mismatch: /mnt/ThreeEight/bitcoind-mainnet/indexes/txindex/052774.ldb
20fish: Job 1, 'bitcoind -server -txindex=1 -da…' terminated by signal SIGABRT (Abort)

I’m not quite sure what to do.

maflcko commented at 6:13 am on August 23, 2024: member

I’m not quite sure what to do.

Given that the corruption happens for any kind of index (txindex, chainstate), a hardware or software issue on your side is the most likely.

I’d try to check if the internal cable is properly attached and the connector isn’t dusty or dirty. Then I’d try some stuff from #30692 (comment) with some caution (Backups are generally recommended, especially if data corruption is likely).

github12101 commented at 11:47 am on August 24, 2024: none

I am also inclined to say this is hardware error. I had these and it turned out to be unreliable RAM. Put that database on Btrfs, it should never have checksum corruption error there. If it does, on Btrfs, and you have kernel errors about checksum, then this is 100% hardware problem. I run full node on Btrfs and I don’t have any problems.

cryptoquick commented at 1:43 pm on August 27, 2024: none

I’m trying to check my RAM using memtest86, but I’m having trouble booting into that too. Give me a bit to troubleshoot.

Tajuras commented at 3:00 pm on August 29, 2024: none

I have the same problem, with windows 10 and windows 11. SSD Sandisk 2T. Computer is Ryzen 3900x absolutely stable. Error occurs in version 27.1 Once i rollback to version 25 i am completely stable now.

maflcko commented at 3:04 pm on August 29, 2024: member

I have the same problem, with windows 10 and windows 11. SSD Sandisk 2T. Computer is Ryzen 3900x absolutely stable. Error occurs in version 27.1 Once i rollback to version 25 i am completely stable now.

Are you saying this is reproducible in 27.1, or did the error happen only once with 27.1?

Tajuras commented at 3:10 pm on August 29, 2024: none

It happen to me only in 27.1. I tested in 2 machines. In standard HD it takes more time to occur. In the one with sandisk SSD 2T, it occurs fast (in the same day after a few hours running). I get the conclusion that the problem might be in version 27.1. I gave up from it because every time error occurs i have to reindex chainstate and it takes 1-2 days. Making a copy from working node is a pain in the ass too. Only chainstate seems to corrupt. I tested fat32 and ntfs file systems. I am a computer programmer, have 3 different nodes in 3 computers. Currently i am only in version 25

Tajuras commented at 3:15 pm on August 29, 2024: none

In my main machine, is ryzen with 32gb ddr4 with 2 sticks of memory. The other machine is Xeon, with 32gb memory, in quad channel ddr3. So i dont think its memory, because the 2 computers are stable. I can run prime95 in both for 2 hours with no problem.

maflcko commented at 3:19 pm on August 29, 2024: member

Which type of Sandisk SSD is it? The “Extreme” ones are known to eat the data, according to https://duckduckgo.com/?q=SSD+Sandisk+Extreme+data+loss

Tajuras commented at 3:22 pm on August 29, 2024: none

SanDisk SSD PLUS 2000GB : 2000,3 GB Serial: 23281C800047 Padrao: ACS-3 | ACS-2 Revision 3 S.M.A.R.T., APM, NCQ, TRIM, DevSleep, GPL SATA/600 | SATA/600

From CrystalDiskinfo

Tajuras commented at 3:26 pm on August 29, 2024: none

I will wait version 28 to come to test it again. For now i am using it with this Sandisk 2T, stable in v25 for about 1 month now.
I let bitcoin core running, with electrum personal server all day while i am working.

cryptoquick commented at 10:01 pm on August 30, 2024: none

@Tajuras Which version of 25 are you running? I tried both 26.2 and 25.2 and I get the same problem with both.

github12101 commented at 11:01 pm on August 30, 2024: none

@Tajuras Which version of 25 are you running? I tried both 26.2 and 25.2 and I get the same problem with both.

Have you memtested your machine yet?

cryptoquick commented at 11:03 pm on August 30, 2024: none

Have you memtested your machine yet?

I’ve been having trouble with getting the grub configuration correct to actually boot into it. I’ll need to use a USB drive.

How long do you recommend I run the test?

github12101 commented at 11:35 pm on August 30, 2024: none

Have you memtested your machine yet?

I’ve been having trouble with getting the grub configuration correct to actually boot into it. I’ll need to use a USB drive.

How long do you recommend I run the test?

Running from USB will definitely simplify the procedure. I recommend running for several hours, maybe overnight. Also, get your storage media scanned for errors. SMART long scan + ext4 filesystem check. Also, you were saying bitcoind doesn’t work very well with Btrfs, can you check again and report back? Let’s see if you will be getting any errors. I am running bitcoind on Btrfs no problem, like previously mentioned.

maflcko commented at 10:31 am on October 8, 2024: member

Did you have any progress testing the hardware?

Also, now that 28.0 is released, you may want to test and try it.

maflcko commented at 10:02 am on October 15, 2024: member

Closing for now due to inactivity, but please do provide a comment with more details, if you find anything.

maflcko closed this on Oct 15, 2024

cryptoquick commented at 12:20 pm on October 15, 2024: none

bitcoind hasn’t been updated on Arch yet. I’m waiting for that to come out. https://archlinux.org/packages/?sort=&q=bitcoin&maintainer=&flagged=

jb55 commented at 3:41 pm on October 30, 2024: contributor

In my main machine, is ryzen with 32gb ddr4 with 2 sticks of memory. The other machine is Xeon, with 32gb memory, in quad channel ddr3. So i dont think its memory, because the 2 computers are stable. I can run prime95 in both for 2 hours with no problem.

it’s true, I just ran disk diagnostics, no issues. this is a new 4tb ssd. going to run a memory diagnostic… but my system has been stable so I would be very surprised if it was that. I’m also running on ZFS. I keep hitting this, even after a full redownload…

version is 28.0

github12101 commented at 10:28 pm on October 30, 2024: none

In my main machine, is ryzen with 32gb ddr4 with 2 sticks of memory. The other machine is Xeon, with 32gb memory, in quad channel ddr3. So i dont think its memory, because the 2 computers are stable. I can run prime95 in both for 2 hours with no problem.

it’s true, I just ran disk diagnostics, no issues. this is a new 4tb ssd. going to run a memory diagnostic… but my system has been stable so I would be very surprised if it was that. I’m also running on ZFS. I keep hitting this, even after a full redownload…

version is 28.0

You have “block checksum mismatch” errorswhile using ZFS? Anything in the ZFS logs to confirm that? Or only bitcoind is complaining? Can you send the logs?

willcl-ark commented at 9:14 am on October 31, 2024: member

In my main machine, is ryzen with 32gb ddr4 with 2 sticks of memory. The other machine is Xeon, with 32gb memory, in quad channel ddr3. So i dont think its memory, because the 2 computers are stable. I can run prime95 in both for 2 hours with no problem.

it’s true, I just ran disk diagnostics, no issues. this is a new 4tb ssd. going to run a memory diagnostic… but my system has been stable so I would be very surprised if it was that. I’m also running on ZFS. I keep hitting this, even after a full redownload…

version is 28.0

I can test this out on a Ryzen CPU with ZFS on SSD (but with only dual channel DDR4).

Is there anything else about the config that you can provide so I can try to reproduce?
What is your ZFS pool setup like?
Is your SSD also SATA-connected like OP?
Are you running any Bitcoin Core options that may be relevant?

jb55 commented at 3:55 pm on October 31, 2024: contributor

zfs-2.2.6-1 sata connected zfs pool is not complicated, just a single drive (4tb ssd) ryzen cpu AMD Ryzen 7 1800X Eight-Core Processor

I ran smartctl no issues, this is a brand new SSD because I thought it was my HDD.

 0=== START OF INFORMATION SECTION ===
 1Device Model:     SPCC Solid State Disk
 2Serial Number:    230707575120019
 3LU WWN Device Id: 0 000000 000000000
 4Firmware Version: VE1R9007
 5User Capacity:    4,000,787,030,016 bytes [4.00 TB]
 6Sector Size:      512 bytes logical/physical
 7Rotation Rate:    Solid State Device
 8Form Factor:      2.5 inches
 9TRIM Command:     Available, deterministic
10Device is:        Not in smartctl database 7.3/5387
11ATA Version is:   ACS-3, ATA8-ACS T13/1699-D revision 6
12SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
13Local Time is:    Thu Oct 31 08:45:04 2024 PDT
14SMART support is: Available - device has SMART capability.
15SMART support is: Enabled
16
17
18SMART Self-test log structure revision number 1
19Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
20# 1  Extended offline    Completed without error       00%      2382         -
21# 2  Short offline       Completed without error       00%      2382         -

I ran badblocks, no output

zfs status -v shows no checksum errors:

0  pool: titan
1 state: ONLINE
2config:
3
4	NAME                                         STATE     READ WRITE CKSUM
5	titan                                        ONLINE       0     0     0
6	  ata-SPCC_Solid_State_Disk_230707575120019  ONLINE       0     0     0
7
8errors: No known data errors

haven’t run a full memtest86 yet but my system has been stable for like 5+ years so I would be very surprised if there were memory issues.

couldn’t recover from a reindex-chainstate or reindex so I deleted everything and started over. it hit another leveldb corruption issue during new IBD at around 2015, but manually restarting with reindex-chainstate seemed to fix it this time.

going to try zfs snapshotting once its stable just to protect against this… but I’ve never had this bad of bitcoin instability until recent versions… 0.27+

my settings are:

0txindex=1
1dbcache=32
2blockfilterindex=1
3peerbloomfilters=1
4peerblockfilters=1

memory:

  0# dmidecode 3.6
  1Getting SMBIOS data from sysfs.
  2SMBIOS 3.0 present.
  3
  4Handle 0x0028, DMI type 16, 23 bytes
  5Physical Memory Array
  6	Location: System Board Or Motherboard
  7	Use: System Memory
  8	Error Correction Type: None
  9	Maximum Capacity: 16 GB
 10	Error Information Handle: 0x0027
 11	Number Of Devices: 4
 12
 13Handle 0x002F, DMI type 17, 40 bytes
 14Memory Device
 15	Array Handle: 0x0028
 16	Error Information Handle: 0x002E
 17	Total Width: 64 bits
 18	Data Width: 64 bits
 19	Size: 8 GB
 20	Form Factor: DIMM
 21	Set: None
 22	Locator: DIMM 0
 23	Bank Locator: CHANNEL A
 24	Type: DDR4
 25	Type Detail: Synchronous Unbuffered (Unregistered)
 26	Speed: 1067 MT/s
 27	Manufacturer: Unknown
 28	Serial Number: 00000000
 29	Asset Tag: Not Specified
 30	Part Number: CMK16GX4
 31	Rank: 1
 32	Configured Memory Speed: 1067 MT/s
 33	Minimum Voltage: 1.2 V
 34	Maximum Voltage: 1.2 V
 35	Configured Voltage: 1.2 V
 36
 37Handle 0x0032, DMI type 17, 40 bytes
 38Memory Device
 39	Array Handle: 0x0028
 40	Error Information Handle: 0x0031
 41	Total Width: 64 bits
 42	Data Width: 64 bits
 43	Size: 8 GB
 44	Form Factor: DIMM
 45	Set: None
 46	Locator: DIMM 1
 47	Bank Locator: CHANNEL A
 48	Type: DDR4
 49	Type Detail: Synchronous Unbuffered (Unregistered)
 50	Speed: 1067 MT/s
 51	Manufacturer: Unknown
 52	Serial Number: 00000000
 53	Asset Tag: Not Specified
 54	Part Number: CMK16GX4
 55	Rank: 1
 56	Configured Memory Speed: 1067 MT/s
 57	Minimum Voltage: 1.2 V
 58	Maximum Voltage: 1.2 V
 59	Configured Voltage: 1.2 V
 60
 61Handle 0x0035, DMI type 17, 40 bytes
 62Memory Device
 63	Array Handle: 0x0028
 64	Error Information Handle: 0x0034
 65	Total Width: 64 bits
 66	Data Width: 64 bits
 67	Size: 8 GB
 68	Form Factor: DIMM
 69	Set: None
 70	Locator: DIMM 0
 71	Bank Locator: CHANNEL B
 72	Type: DDR4
 73	Type Detail: Synchronous Unbuffered (Unregistered)
 74	Speed: 1067 MT/s
 75	Manufacturer: Unknown
 76	Serial Number: 00000000
 77	Asset Tag: Not Specified
 78	Part Number: CMK16GX4
 79	Rank: 1
 80	Configured Memory Speed: 1067 MT/s
 81	Minimum Voltage: 1.2 V
 82	Maximum Voltage: 1.2 V
 83	Configured Voltage: 1.2 V
 84
 85Handle 0x0038, DMI type 17, 40 bytes
 86Memory Device
 87	Array Handle: 0x0028
 88	Error Information Handle: 0x0037
 89	Total Width: 64 bits
 90	Data Width: 64 bits
 91	Size: 8 GB
 92	Form Factor: DIMM
 93	Set: None
 94	Locator: DIMM 1
 95	Bank Locator: CHANNEL B
 96	Type: DDR4
 97	Type Detail: Synchronous Unbuffered (Unregistered)
 98	Speed: 1067 MT/s
 99	Manufacturer: Unknown
100	Serial Number: 00000000
101	Asset Tag: Not Specified
102	Part Number: CMK16GX4
103	Rank: 1
104	Configured Memory Speed: 1067 MT/s
105	Minimum Voltage: 1.2 V
106	Maximum Voltage: 1.2 V
107	Configured Voltage: 1.2 V

jb55 commented at 4:09 pm on October 31, 2024: contributor

hmm I have a small dbcache size I just noticed… can’t remember why I set it to that. maybe the increased fsyncs is increasing the chance of a bug somehow? can’t think of anything else if its not a hardware issue.

github12101 commented at 4:12 pm on October 31, 2024: none

hmm I have a small dbcache size I just noticed… can’t remember why I set it to that. maybe the increased fsyncs is increasing the chance of a bug somehow? can’t think of anything else if its not a hardware issue.

Can/did you run memtest86 or memtest86+, for several hours? No errors?

jb55 commented at 4:14 pm on October 31, 2024: contributor

hmm I have a small dbcache size I just noticed… can’t remember why I set it to that. maybe the increased fsyncs is increasing the chance of a bug somehow? can’t think of anything else if its not a hardware issue.

Can/did you run memtest86, for several hours? No errors?

I haven’t done that yet because I run my entire business (https://damus.io) off this node. my system has been stable for years.

willcl-ark commented at 4:41 pm on October 31, 2024: member

32MB does seem pretty small…

github12101 commented at 4:45 pm on October 31, 2024: none

hmm I have a small dbcache size I just noticed… can’t remember why I set it to that. maybe the increased fsyncs is increasing the chance of a bug somehow? can’t think of anything else if its not a hardware issue.

Can/did you run memtest86, for several hours? No errors?

I haven’t done that yet because I run my entire business (https://damus.io) off this node. my system has been stable for years.

I am afraid, until you do, you can’t rule out memory problems. Until this is checked, blaming bitcoind for the instability is an invalid claim.

github12101 commented at 4:47 pm on October 31, 2024: none

32MB does seem pretty small…

Good point, @jb55 any only 32MiB for dbcache? I gave it the following: dbcache=16384

Is it recommended giving it as much as you can spare. If you have free RAM, give it a couple of gigabytes.

Please note that despite setting max 16GiB for dbcache, currently entire process is only using 2GiB (2 days since last restart). So it’s not a big of a burden.

jb55 commented at 5:21 pm on October 31, 2024: contributor

I set it to 1024, will see if that helps

davidgumberg commented at 5:30 pm on October 31, 2024: contributor

Even if a bigger dbcache mitigates the issue because of fewer flushes, using a low dbcache setting should not result in leveldb block checksum mismatches, I will experiment with syncing a node with -dbache=32.

With such a low dbcache you are definitely doing way more writing than a node with default dbcache, so I imagine the rate of wear on the disk is somewhat high, I think 32MB might be low enough that a flush gets triggered every block?

jb55 commented at 5:32 pm on October 31, 2024: contributor

I think 32MB might be low enough that a flush gets triggered every block?

I didn’t see that in the logs

davidgumberg commented at 5:44 pm on October 31, 2024: contributor

I didn’t see that in the logs

You can see flushes with -debug=coindb

Every block was ambitious, sorry, maybe every ~15 blocks or so.

jb55 commented at 6:02 pm on October 31, 2024: contributor

maybe ECC ram would be smarter too… could just be getting hit by cosmic rays or something

github12101 commented at 3:49 am on November 1, 2024: none

maybe ECC ram would be smarter too… could just be getting hit by cosmic rays or something

Cosmic rays are quite unlikely. But simple error because of failing RAM stick, very likely. You may not even seen this in day to day work. But because bitcoind is writing a lot of data (and everything what it writes go through RAM first), then it may be helping you to expose otherwise unknown hardware problem.

willcl-ark commented at 9:43 am on November 5, 2024: member

I can test this out on a Ryzen CPU with ZFS on SSD (but with only dual channel DDR4).

* Is there anything else about the config that you can provide so I can try to reproduce?

* What is your ZFS pool setup like?

* Is your SSD also SATA-connected like OP?

* Are you running any Bitcoin Core options that may be relevant?

OK I could not reproduce this on either a native ZFS zpool nor an “ext4 on ZFS” zpool (configurations found here: https://github.com/nix-community/disko/blob/master/example/zfs.nix).

Both times bitcoin core v28.0 synced to tip using the exact same configuration you have been using:

0txindex=1
1dbcache=32
2blockfilterindex=1
3peerbloomfilters=1
4peerblockfilters=1

The only difference is that my SSDs are connected via NVMe, not SATA. I used a mirror vdev, but again, I don’t see how this would make any difference, and no errors on either disk were detected over the duration of both runs:

At this point, in my mind, this is strongly points to either a hardware or OS-level error, and not any regression in Bitcoin Core.

If you have any more ideas on how I may be able to repro this, then please let me know here, and I can try to test them when I find time.

jb55 commented at 6:04 pm on November 12, 2024: contributor

fwiw I ran a full memtest for 4 hours and had no memory issues, so at this point I have no idea. it seems to be running stable for now.