core stops to run with `Failed to read block` error

vincenzopalazzo commented at 10:47 am on February 22, 2023: none

Expected behavior

While my node was running it get an internal failure that force it to stop the running

02023-02-21T23:29:22Z Syncing basic block filter index with block chain from height 624447
12023-02-21T23:29:34Z ERROR: SerializeFileDB: Rename-into-place failed
22023-02-21T23:29:52Z Syncing basic block filter index with block chain from height 624889
32023-02-21T23:30:22Z ERROR: ReadBlockFromDisk: Deserialize or I/O error - ReadCompactSize(): size too large: iostream error at FlatFilePos(nFile=2030, nPos=82255446)
42023-02-21T23:30:22Z *** ThreadSync: Failed to read block 0000000000000000000def83407e184fbc5f53b47effdaef7b15e2d6f6b8579d from disk
52023-02-21T23:30:22Z Error: A fatal internal error occurred, see debug.log for details

The code is based on the commit 94070029fb6b783833973f9fe08a3a871994492f

To reproduce

No idea how I can reproduce it, now I switch to the tagged version to see if this happens too

System information

 0➜  bitcoin git:(master) neofetch                                        
 1       _,met$$$$$gg.          vincent@x86 
 2    ,g$$$$$$$$$$$$$$$P.       ----------- 
 3  ,g$$P"     """Y$$.".        OS: Debian GNU/Linux 11 (bullseye) x86_64 
 4 ,$$P'              `$$$.     Host: OptiPlex 3050 
 5',$$P       ,ggs.     `$$b:   Kernel: 5.10.0-21-amd64 
 6`d$$'     ,$P"'   .    $$$    Uptime: 5 days, 16 hours, 33 mins 
 7 $$P      d$'     ,    $$P    Packages: 502 (dpkg) 
 8 $$:      $$.   -    ,d$$'    Shell: zsh 5.8 
 9 $$;      Y$b._   _,d$P'      Terminal: /dev/pts/1 
10 Y$$.    `.`"Y$$$$P"'         CPU: Intel i5-7500T (4) @ 3.300GHz 
11 `$$b      "-.__              GPU: Intel HD Graphics 630 
12  `Y$$                        Memory: 2214MiB / 7840MiB 
13   `Y$$.
14     `$$b.                                            
15       `Y$$b.                                         
16          `"Y$b._
17              `"""

vincenzopalazzo added the label Bug on Feb 22, 2023

maflcko added the label Data corruption on Feb 22, 2023

willcl-ark commented at 11:36 am on February 22, 2023: member

We do not currently propagate the exact error code raised by fs::rename so it might be difficult to diagnose precisely… The most usual cases are the filesystem being full or corrupt. Are you sure your filesystem has enough free space?

Seeing as it later hit the ReadCompactSize deserialize error afterwards, it does feel more likely to be a fs corruption issue…

vincenzopalazzo commented at 12:18 pm on February 22, 2023: none

The most usual cases are the filesystem being full or corrupt. Are you sure your filesystem has enough free space?

Yes 100%

0/dev/sdb1      4883759996 2603409780 2280350216  54% /media/Matrox

Seeing as it later hit the ReadCompactSize deserialize error afterward, it does feel more likely to be an fs corruption issue…

The strage thing is that this is running for days, so I did not get any downside that can end up in something half written or just put the trash inside the .dat file.

I open the issue because this happens while I was running the master, I ran this node for years now with a tagged version, and I had no problem with it. So it may be a regression?

willcl-ark commented at 1:59 pm on March 23, 2023: member

Hey @vincenzopalazzo just wanted to follow up here…

Was this something that you managed to fix, has it gone away on it’s own, or are you still experiencing problems with it?

vincenzopalazzo commented at 4:24 pm on March 23, 2023: none

@willcl-ark after the downgrade to the tagged version I have not try it again, sorry!

willcl-ark commented at 1:16 pm on March 24, 2023: member

Do you think it’s something you will try again in the future? It would be good to get reproduction steps if possible…

vincenzopalazzo commented at 2:33 pm on March 25, 2023: none

yeah I will try asap

pinheadmz commented at 1:51 pm on April 27, 2023: member

Checking in here, is this still an issue? We’ll close for now, please comment if you can reproduce.

pinheadmz closed this on Apr 27, 2023

bitcoin locked this on Apr 26, 2024

core stops to run with Failed to read block error #27142

core stops to run with `Failed to read block` error #27142