Non-latest rev files are sometimes re-opened but never re-flushed #17890

issue kallewoof openend this issue on January 7, 2020
  1. kallewoof commented at 3:41 pm on January 7, 2020: member

    The code currently assumes that once a block file is ’left’, it and its corresponding rev file can be flushed, and won’t be written to any further.

    https://github.com/bitcoin/bitcoin/blob/e8e79958a7b2a0bf1b02adcce9f4d811eac37dfc/src/validation.cpp#L1742-L1743

    (Flushing here additionally means to get rid of pre-allocated unused space.)

    This is the case for the bl(oc)k files, but it turns out to not be the case for the rev files. During an IBD up to ~200k blocks, the following excerpt (one of several) was encountered (grepping out irrelevancy):

     0[...]
     12020-01-07T14:19:09Z Pre-allocating up to position 0x7000000 in blk00013.dat
     22020-01-07T14:19:13Z Pre-allocating up to position 0x8000000 in blk00013.dat
     32020-01-07T14:19:14Z Pre-allocating up to position 0xf00000 in rev00013.dat
     42020-01-07T14:19:16Z Leaving block file 13: CBlockFileInfo(blocks=1099, size=134068483, heights=183531...185655, time=2012-06-08...2012-06-22)
     52020-01-07T14:19:16Z Pre-allocating up to position 0x1000000 in blk00014.dat
     62020-01-07T14:19:17Z Pre-allocating up to position 0x100000 in rev00014.dat
     72020-01-07T14:19:20Z Pre-allocating up to position 0x1000000 in rev00013.dat
     82020-01-07T14:19:20Z Pre-allocating up to position 0x2000000 in blk00014.dat
     92020-01-07T14:19:24Z Pre-allocating up to position 0x1100000 in rev00013.dat
    102020-01-07T14:19:26Z Pre-allocating up to position 0x3000000 in blk00014.dat
    112020-01-07T14:19:27Z Pre-allocating up to position 0x1200000 in rev00013.dat
    122020-01-07T14:19:30Z Pre-allocating up to position 0x200000 in rev00014.dat
    132020-01-07T14:19:33Z Pre-allocating up to position 0x300000 in rev00014.dat
    142020-01-07T14:19:35Z Pre-allocating up to position 0x400000 in rev00014.dat
    152020-01-07T14:19:37Z Pre-allocating up to position 0x500000 in rev00014.dat
    162020-01-07T14:19:39Z Pre-allocating up to position 0x600000 in rev00014.dat
    172020-01-07T14:19:42Z Pre-allocating up to position 0x700000 in rev00014.dat
    182020-01-07T14:19:43Z Pre-allocating up to position 0x4000000 in blk00014.dat
    192020-01-07T14:19:45Z Pre-allocating up to position 0x800000 in rev00014.dat
    202020-01-07T14:19:49Z Pre-allocating up to position 0x900000 in rev00014.dat
    21[...]
    

    But the only time we flush (and truncate the files to their correct size) is when we leave a block, and we already left block 13 here, so the rev files above end up with pre-allocated space that is never used. This is related to #17827.

    This seems to repeat itself for every file that ended up larger than expected in #17887 (comment). (Note: #17887 is not a fix for this issue. #17887 makes the impact much smaller by fixing a separate blow-up bug, but it does not address re-flushing re-opened rev files.)

  2. kallewoof added the label Bug on Jan 7, 2020
  3. laanwj closed this on Jun 4, 2020

  4. sidhujag referenced this in commit 3f5d6102a6 on Jun 4, 2020
  5. PastaPastaPasta referenced this in commit d4a9c7d6d4 on Sep 17, 2021
  6. PastaPastaPasta referenced this in commit 480fc90ef7 on Sep 19, 2021
  7. thelazier referenced this in commit f2f86ccf33 on Sep 25, 2021
  8. MarcoFalke locked this on Feb 15, 2022


kallewoof

Labels
Bug


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-11-21 12:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me