Reindex initiated by Bitcoin-Qt leaves 293 byte zero-padded gap in blk00000.dat #5028

1. 
   trevelocity commented at 3:16 am on October 2, 2014: none

   After setting up a bitcoind server I found that the file “blocks/blk00000.dat” had a 293 byte zero-padded gap immediately flowing the genesis block. This is not a problem for bitcoind since the LoadExternalBlockFile() routine in main.cpp has a mechanism to scan for the message start sequence if blocks are not contiguous. It is a problem for other programs (e.g. blockparser) which assume that “nSize” points to the next block. I apologize in advance if this is already a known issue.

   The issue can be demonstrated as follows:

   1. bitcoind and bitcoin-qt version v0.9.2.1 on Linux, disconnected from internet.
   2. create empty ~/.bitcoin directory and cd to it.
   3. bitcoind &
      • exits because no bitcoin.conf and default arg -server=1 and no rpc setup
   4. create bitcoin.conf with these entries: reindex=0 txindex=1 server=0
   5. bitcoin-qt &
      • Click OK in popup about rebuilding block database
      • File->exit
   6. copy beginning ~1kB of bootstrap.dat to ~/.bitcoin
   7. bitcoin-qt &
      • File->exit ~/.bitcoin/debug.log should contain this: 2014-09-30 03:52:40 LoadBlockIndexDB(): last block file = 0 2014-09-30 03:52:40 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=2, size=586, heights=0…0, time=2009-01-03…2009-01-03) 2014-09-30 03:52:40 LoadBlockIndexDB(): transaction index enabled

   $ od -A x -t x1 blocks/blk00000.dat | head -40 000000 f9 be b4 d9 1d 01 00 00 01 00 00 00 00 00 00 00 000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000020 00 00 00 00 00 00 00 00 00 00 00 00 3b a3 ed fd 000030 7a 7b 12 b2 7a c7 2c 3e 67 76 8f 61 7f c8 1b c3 000040 88 8a 51 32 3a 9f b8 aa 4b 1e 5e 4a 29 ab 5f 49 000050 ff ff 00 1d 1d ac 2b 7c 01 01 00 00 00 01 00 00 000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff ff 000080 ff ff 4d 04 ff ff 00 1d 01 04 45 54 68 65 20 54 000090 69 6d 65 73 20 30 33 2f 4a 61 6e 2f 32 30 30 39 0000a0 20 43 68 61 6e 63 65 6c 6c 6f 72 20 6f 6e 20 62 0000b0 72 69 6e 6b 20 6f 66 20 73 65 63 6f 6e 64 20 62 0000c0 61 69 6c 6f 75 74 20 66 6f 72 20 62 61 6e 6b 73 0000d0 ff ff ff ff 01 00 f2 05 2a 01 00 00 00 43 41 04 0000e0 67 8a fd b0 fe 55 48 27 19 67 f1 a6 71 30 b7 10 0000f0 5c d6 a8 28 e0 39 09 a6 79 62 e0 ea 1f 61 de b6 000100 49 f6 bc 3f 4c ef 38 c4 f3 55 04 e5 1e c1 12 de 000110 5c 38 4d f7 ba 0b 8d 57 8a 4c 70 2b 6b f1 1d 5f 000120 ac 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * 000240 00 00 00 00 00 00 00 00 00 00 f9 be b4 d9 d7 00 000250 00 00 01 00 00 00 6f e2 8c 0a b6 f1 b3 72 c1 a6 000260 a2 46 ae 63 f7 4f 93 1e 83 65 e1 5a 08 9c 68 d6 000270 19 00 00 00 00 00 98 20 51 fd 1e 4b a7 44 bb be 000280 68 0e 1f ee 14 67 7b a1 a3 c3 54 0b f7 b1 cd b6 000290 06 e8 57 23 3e 0e 61 bc 66 49 ff ff 00 1d 01 e3 0002a0 62 99 01 01 00 00 00 01 00 00 00 00 00 00 00 00 0002b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0002c0 00 00 00 00 00 00 00 00 ff ff ff ff 07 04 ff ff 0002d0 00 1d 01 04 ff ff ff ff 01 00 f2 05 2a 01 00 00 0002e0 00 43 41 04 96 b5 38 e8 53 51 9c 72 6a 2c 91 e6 0002f0 1e c1 16 00 ae 13 90 81 3a 62 7c 66 fb 8b e7 94 000300 7b e6 3c 52 da 75 89 37 95 15 d4 e0 a6 04 f8 14 000310 17 81 e6 22 94 72 11 66 bf 62 1e 73 a8 2c bf 23 000320 42 c8 58 ee ac 00 00 00 00 f9 be b4 d9 d7 00 00 Note: zero-padding from 293(0x125) to 585(0x249)

2. 
   ghost commented at 4:50 am on October 2, 2014: none

   This is known behavior, there’s nothing really saying that a block can’t have huge gaps filled with Shania Twain, the software is designed to handle that if necessary. I’d call this a block parser bug rather than a Bitcoin Core one.
3. 
   sipa commented at 5:41 am on October 2, 2014: member

   As the person who implemented the current block file storage logic, I consider this a bug.
4. 
   ghost commented at 6:08 am on October 2, 2014: none

   Even if it was a bug that caused some zero bytes to be written to the file, so what? The parser is going to have to be altered to handle non-contiguous files anyway, I hardly think that every single node is going to rewrite their blocks on disk to get rid of 293 bytes.
5. 
   sipa commented at 6:11 am on October 2, 2014: member

   Nothing needs to be rewritten. It is working is a fully compatible way.

   But if this report is correct, the code is not doing what it is intended to do. That’s a bug.

6. 
   trevelocity commented at 0:49 am on October 4, 2014: none

   The two application programs that I use, which both failed to load my blk00000.dat file, have now been fixed. I think that bitcoind functionality takes precedence here, so I do not expect to see any changes to the bitcoind code. However, to avoid this turning into a recurring issue, could someone please add a few comment lines at the start of main.cpp:LoadExternalBlockFile() to inform future code sleuths that this is the expected behavior.
7. 
   trevelocity commented at 2:30 pm on October 19, 2014: none

   I got a third application that failed, so I decided it was just easier to fix the block file. dd bs=1 if=blk00000.dat of=genesis.dat count=293 dd bs=586 if=blk00000.dat of=tail.dat skip=1 cat genesis.dat genesis.dat tail.dat >blk00000.dat
8. trevelocity closed this on Oct 19, 2014

   ---

9. 
   TheBlueMatt commented at 7:24 pm on October 27, 2014: contributor

   @trevelocity Please do not close this bug if you have fixed it yourself, it is a still a bug in Bitcoin Core, even if a low priority one with the workaround you gave.
10. laanwj reopened this on Oct 27, 2014

    ---

11. laanwj added the label Block storage on Feb 16, 2016
12. laanwj referenced this in commit df50fd194f on Oct 8, 2019
13. sidhujag referenced this in commit 2624540adc on Oct 8, 2019
14. 
    adamjonas commented at 1:21 pm on April 30, 2020: member

    A note from the author of #16802 - that PR solved #14986 by skipping odd bytes until it finds the magic bytes indicating the start of the next block. To solve this issue, blockparser needs to do the same.
15. PastaPastaPasta referenced this in commit 2e6335c475 on Sep 11, 2021
16. PastaPastaPasta referenced this in commit 158ca47c3c on Sep 11, 2021
17. PastaPastaPasta referenced this in commit c928eae616 on Sep 12, 2021
18. PastaPastaPasta referenced this in commit debed8d819 on Sep 12, 2021
19. PastaPastaPasta referenced this in commit 0ab338a7bd on Sep 12, 2021
20. PastaPastaPasta referenced this in commit e5c7de4e87 on Sep 14, 2021
21. PastaPastaPasta referenced this in commit c9f2068cde on Sep 14, 2021
22. PastaPastaPasta referenced this in commit bba59cabc3 on Sep 15, 2021
23. 
    mruddy commented at 12:52 pm on April 18, 2022: contributor

    note that this particular issue may have been fixed by #5864. i mention this because my PR #24858 updates the code of #5864 due to a closely related bug. i think this issue should be closed because it is very outdated compared to the current functionality. it probably should have been closed by #5864 and just wasn’t noticed.
24. 
    adamjonas commented at 2:02 pm on August 9, 2022: member

    Closing based on mruddy’s comment and the age of the ticket without further reports. Can re-open if it resurfaces.
25. adamjonas closed this on Aug 9, 2022

    ---

26. bitcoin locked this on Aug 9, 2023

Contributors
trevelocity ghost sipa TheBlueMatt adamjonas mruddy

Labels
Block storage