Block pre-allocation appears to have surprisingly poor performance

wpeckr commented at 9:34 am on February 24, 2021: none

When syncing Bitcoin Core with extremely fast storage the pre-allocation of block files with fallocate looks to be a surprising portion for the wall time for IBD and appears to sometimes slow down block relay at the tip of the chain. It’s not completely clear why this would be the case, 300ms to write 16MB to tmpfs would be excessively slow.

Is this an artifact of the logging (altered in #21041), or an actual performance concern?

 02021-02-24T08:49:34.637365Z UpdateTip: new best=00000000000000000020fc15108523bcb10b5b76a3fd61ce57fb8460f666f845 height=578868 version=0x20000000 log2_work=90.697662 tx=420028811 date='2019-06-01T23:46:52Z' progress=0.679947 cache=74.0MiB(372110txo)
 12021-02-24T08:49:34.828910Z Pre-allocating up to position 0x6000000 in blk01663.dat
 2
 3...
 4
 52021-02-24T08:49:42.683484Z Pre-allocating up to position 0x1000000 in blk01667.dat
 62021-02-24T08:49:42.951362Z Pre-allocating up to position 0x2000000 in blk01667.dat
 72021-02-24T08:49:43.257131Z Pre-allocating up to position 0x3000000 in blk01667.dat
 82021-02-24T08:49:43.510151Z Pre-allocating up to position 0x4000000 in blk01667.dat
 92021-02-24T08:49:43.752604Z Pre-allocating up to position 0x5000000 in blk01667.dat
102021-02-24T08:49:44.074489Z Pre-allocating up to position 0x6000000 in blk01667.dat
112021-02-24T08:49:44.341478Z Pre-allocating up to position 0x7000000 in blk01667.dat
122021-02-24T08:49:44.655581Z Pre-allocating up to position 0x8000000 in blk01667.dat
13
14...
15
162021-02-24T08:49:44.886586Z Pre-allocating up to position 0x1000000 in blk01668.dat
172021-02-24T08:49:45.148651Z Pre-allocating up to position 0x2000000 in blk01668.dat
182021-02-24T08:49:45.515480Z Pre-allocating up to position 0x100000 in rev01668.dat
192021-02-24T08:49:45.525122Z UpdateTip: new best=0000000000000000000c7af325f257778530cd2127b0ceacb1f56ce4c98fa4dc height=578869 version=0x20c00000 log2_work=90.697685 tx=420031549 date='2019-06-02T00:18:16Z' progress=0.679951 cache=29.9MiB(12647txo)

wpeckr added the label Bug on Feb 24, 2021

sipa commented at 6:54 pm on January 8, 2022: member

Just because the “Pre-allocating …” lines are 300 ms apart doesn’t mean that that’s the only thing it’s doing during that time. It only adds another allocation once the previous one is filled up, so it has downloaded, verified, and stored 16 MB worth of block data in that time too.

Also, during IBD blocks aren’t being relayed at all, normally. I’m not sure where you’re getting that impression that it’s being slowed down.

MarcoFalke removed the label Bug on Jan 9, 2022

MarcoFalke added the label Questions and Help on Jan 9, 2022

MarcoFalke added the label Block storage on Jan 9, 2022

sipa commented at 3:59 pm on January 9, 2022: member

Let me explain what is happening.

Bitcoin Core downloads blocks out-of-order (up to 1024 blocks ahead of where your current tip is). However, the tip cannot progress past block N until all blocks up to N have been downloaded and stored. Sometimes, when blocks arrive significantly out-of-order, that can mean you receive lots of blocks, which get stored on disk (causing preallocations), but the tip doesn’t progress because one straggler block is lagging behind. Once that block is received, you suddenly are able to progress a lot past it.

adamjonas commented at 5:09 pm on August 4, 2022: member

@sipa’s answer above answers the question in the OP.

adamjonas closed this on Aug 4, 2022

bitcoin locked this on Aug 4, 2023

Block pre-allocation appears to have surprisingly poor performance #21291