Instead of using "getblocks" to learn about the hashes of blocks peers have,
use "getheaders" to retrieve the headers of those blocks entirely. This
allows the client to immediately build an internal representation of the
block tree. This means the best chain (or at least its most likely candidate,
disregarding invalid transactions) is almost instantly known, and block download
can proceed as a secondary (though in practice simultaneous) and very robust
process to acquire the transaction data along this already-known best chain.
Advantages include:
* No more vulnerabilities related to orphan blocks, as they simply don't exist
anymore. We only fetch blocks along the best chain; any other block we're not
interested in.
* No more need for checkpoints to determine bounds on difficulties of received
blocks, as we never receive a block without knowing (the headers of) its
ancestry. For a new, smaller vulnerability (sending many long low-difficulty
header chains, consuming memory), checkpoints are still useful.
* No more need for checkpoints for safety when disabling signature check. A rule
like "Do not check signatures in blocks covered by N weeks worth of hash
power" could be used, though this is currently not yet implemented.
* The IsInitialBlockDownload() become very precises and safe: we're
syncing a backlog as long as there are headers in our best chain for which
blocks are missing. There is no need for relying on peers' reported number
of blocks anymore.
* It can serve as a basis to implement SPV mode, which is basically the same
logic, but with the blockchain-driven block fetching replaced by wallet-driven
filtered block fetching.
* Efficient parallel block download, and generally avoiding many edge cases
that were possible during the previous getblocks-based synchronization
(including frequent duplicate downloads, orphans and and needing to wait for
the next block to be announced before resuming). Additionally, good
synchronization speed does not rely on guessing a good initial peer to fetch
from anymore - a single slow peer does not hurt you anymore. This also means
that throttling upload speeds will no longer be a problem to the network.
For now, block sync is only done from outbound peers, to retain the
"no listening == less bandwidth" property that currently exist. As soon as
enough nodes upgrade to headers-first sync, this can be removed.
Potentially significant semantical changes:
* In case a new best chain is known, but its tip blocks are not yet known, the
best known ancestor of the new tip is used instead of the old tip. On the
other hand, during this period it is known that we're not up to date. The
effects of this on mining should be considered, probably.
More in detail, changes include:
* All redundant globals are gone (nBestHeight, nBestChainWork, nBestInvalidWork,
hashBestChain) - instead, we just have 4 pointers into the block tree
(pindexGenesisBlock, pindexBest, pindexBestHeader, pindexBestInvalid).
* ProcessBlock and AcceptBlock are split into ProcessBlockHeader and
AcceptBlockHeader.
* Instead of 'getblocks', push out 'getheaders' to the sync node.
* Implement 'headers' P2P handler, causing headers to be included in the block
index.
* pindexBest remains as pointer to the "current state of the UTXO set"
(chainstate), meaning the point until where we are synchronized.
pindexBestHeader is added as a pointer to the tip of the currently best known
(and validated, except for transaction validity) header. pindexBest is
guaranteed to be always an ancestor of (or equal to) pindexBestHeader.
* A new function SwitchToBestHeader finds the best header in the database, and
switches to it. SwitchToBestBlock connects blocks along this best chain. These
replace ConnectBestBlock.
* The block tree internally can deal with orphan blocks (whose parents are
unknown), but this mechanism is only used during reindexing, as block may be
stored out-of-order. A new function LinkOrphans connects orphans to the tree,
and replaces some code in LoadBlockIndex.
* Actual blocks are only fetched along the known-best chain, using parallel
block downloading logic in SendMessages. For each peer, a (bounded) list of
requested blocks is maintained. New requests are pipelined, as the requested
blocks arrive. As the tree they belong to is already known, we can receive and
store them out-of-order. This also means that the concept of orphan blocks
disappears. If one peer stalls us, it is disconnected and the blocks requested
from it are sent elsewhere. This is nicer both for us and for them.
* One exception to the headers-first syncing, is when we are fully synced
(pindexBest==pindexBestHeader), and a new block 'inv' is announced. In this
case, the block is fetched immediately ('getdata') after asking for the headers
preceding it, to avoid the extra roundtrip.
* SetBestChain (which handled reorganizations) is no longer necessary, as
SwitchToBestHeader and SwitchToBestBlock only ever connect/disconnect
individual blocks. It is therefore replaced by two simpler functions,
ConnectTip and DisconnectTip, implented using helpers WriteChainState and
UpdateTip.
* In several places code is added to deal with the case that a CBlockIndex
entry does not have corresponding block data available.
