Add parallelism for downloading the blockchain #27442

issue Farxial openend this issue on April 9, 2023
  1. Farxial commented at 12:08 pm on April 9, 2023: none

    Please add parallelism (at least, multithreading; multithreading + some kind of clustering are better) for downloading blockchain by client/wallet (required by getblocktemplate). The speed of downloading the blockchain of Bitcoin is very slow, so much time is required, it’s absolutely awful. And the further, the worse. As I can understand, it happens because of nature of blockchain processing: first, client must check block N contents (transactions), then it can proceed to check of block N+1. But we can behave like modern CPUs branch predictors. Why not? Just pretend block N is correct for thread M and let thread M+1 check block N+1. Then let thread M+1 pass necessary information to thread M+2 for checking block N+2. And so on. So all threads can check it’s blocks, doing CPUs utilization. Threads can be reused as they become free. Just break processing at some points and do rollback, if some previous blocks (N, N+1, …) gets corrupted. As far I can see by CPUs usage, Bitcoin Core/Wallet does not behave like that. My Internet connection is not such slow and unstable; I once experimented with connection limit options, it didn’t work for me. The behavior I suggest has some performance risk, so it can be explicitly controlled by the user with launch option[s] (command line, …).

    (I aware about the security considerations, but this is not a issue if the user just wants to try mining or something.)

  2. Farxial added the label Feature on Apr 9, 2023
  3. pinheadmz commented at 2:19 pm on April 9, 2023: member
    This might be a better question for https://bitcoin.stackexchange.com, I think if you search that knowledgebase for topics like assumevalid and headers first sync (download blocks from multiple peers) you’ll find that IBD already has many optimizations of the kind you are describing. Assume valid in particular is optional because it skips validation on most blocks in the chain “even though” that carries a security compromise with it.
  4. sipa commented at 2:36 pm on April 9, 2023: member

    There are already two types of parallel processing involved in the initial block download mechanism:

    • Blocks are downloaded from multiple peers in parallel, fetching one up to a few blocks per peer at a time, from a moving window of the next 1024 blocks.
    • Script validation of blocks is performed in parallel on multiple CPU cores (splitting the work for any one single block). This only has an effect during the final blocks in the chain after the assumevalid point, because before, no script validation takes place at all.

    Going beyond these would require very invasive changes to the protocol, as it’s not possible to validate the inputs of block N+1 without having the outputs of block N, as validation depends on the outputs being spent.

    In practice, initial download speed is primarily limited by the cost of maintaining the UTXO set database, not downloading and validation itself. To speed things up, your best bet is increasing the database cache size.

    As @pinheadmz says, for more information on how things work, https://bitcoin.stackexchabge.com is a better resource than the bug tracker.

  5. fanquake closed this on Apr 9, 2023

  6. Farxial commented at 2:55 pm on April 9, 2023: none

    @pinheadmz Sometime I tried assumevalid, but it didn’t help me. May be I tried wrong? OK then, I will try this on next full synching, if you insist. Does this work with prune blockchain synching? I can’t find how to download from multiple peers on request headers first sync. Can you tell a launch option, please? And, of course, it would be nice to find such an option in the output of bitcoind --help. @sipa I see…

    Thanks for the info both of you.

  7. pinheadmz commented at 3:02 pm on April 9, 2023: member

    By default, the first 751565 blocks are assumed valid, without any user setting:

    https://github.com/bitcoin/bitcoin/blob/db720b5a703c90625fa7a4773bd2db5672427cbe/src/kernel/chainparams.cpp#L107

    Parallel download is also the default, see FindNextBlocksToDownload():

    https://github.com/bitcoin/bitcoin/blob/master/src/net_processing.cpp#L1298

  8. bitcoin locked this on Apr 8, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-09-19 10:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me