Erlay Project Tracking: #30249
This is a full implementation of Erlay. Its purpose is to check the integrity and correctness of the implementation against changes/additions that may originate from the review process and/or rebases on top of newer functionality.
This is not to be merged. Functionality will be spread across multiple smaller PRs to ease the review process.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
For detailed information about the code coverage, see the test coverage report.
See the guideline for information on the review process. A summary of reviews will appear here.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.
Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
Leave a comment here, if you need help tracking down a confusing failure.
Feel free to grab https://github.com/hebasto/bitcoin/commit/ffed2ab27465e4e3888b4139ca50db1226a66ec5 to fix the MSVC build.
`m_is_inbound` cannot be changed throughout the life of a `Peer`. However, we
are currently storing it in `CNodeState`, which requires locking `cs_main` in
order to access it. This can be moved to the outside scope and only require
`m_peer_mutex`.
This is a refactor in preparation for Erlay reworks.
These comments became irrelevant in one of the previous code changes.
They simply don't make sense anymore.
They will be used later on.
Transactions eligible for reconciliation are added to the
reconciliation sets. For the remaining txs, low-fanout is used.
Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
It helps to avoid recomputing every time we consider
a transaction for fanout/reconciliation.
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
If a wtxid to be added to a peer's recon set has a shot id collisions (a previously
added wtxid maps to the same short id), both transaction should be fanout, given
our peer may have added the opposite transaction to our recon set, and reconciliation
will fail. Moreover, all descendants of the previously added transaction that were in
the recon set should also be removed and fanout.
If a transaction to be reconciled has in mempool ancestors, we need to be consistent
with the way in which we announce this transaction (either by fanning out or reconciling
all the ancestor set). In order for Erlay to work best, a minimum amount of fanout is required.
Therefore, we cannot simply reconcile with everyone in this case either. Find the subset of peers
that have the least ancestors to be reconciled and send all of them via fanout
For a subset of reconciling peers we announce transactions
via low fanout. We need to set lower intervals for that to
achieve lower relay latency.
Note that for privacy reasons the ratio between inbound and outbound
delays matter much more than the actual delays. That ratio is preserved
here, so it is not a privacy degradation.
When we're finalizing negotiation, we should add the peers
for which we will initiate reconciliations to the queue.
We initiate reconciliation by looking at the queue periodically
with equal intervals between peers to achieve efficiency.
This will be later used to see whether it's time to initiate.
When the time comes for the peer, we send a
reconciliation request with the parameters which
will help the peer to construct a (hopefully) sufficient
reconciliation sketch for us. We will then use that
sketch to find missing transactions.
Store the parameters the peer sent us inside the
reconciliation request.
For initial reconciliatin requests we track whether was the last time
we responded to them, and if it was too recent, we respond after
a small delay.
When the time comes, we should send a sketch of our
local reconciliation set to the reconciliation initiator.
When the sketches from both sides are combined successfully,
the diff is produced. Then this diff can (together with the local txs)
be used to identified which transactions are missing locally and remotely.
This will help to reuse the code later on in the function
to announce transactions.
If after decoding a reconciliation sketch it turned out
to be insufficient to find set difference, request extension.
Store the initial sketches so that we are able to process
extension sketch while avoiding transmitting the same data.
To be ready to respond to a sketch extension request
from our peer, we should store a snapshot of our state
and capacity of the initial sketch, so that we compute
extension of the same size and over the exact same
transactions.
Transactions arriving during this reconciliation will
be instead stored in the regular set.
If peer failed to reconcile based on our initial response sketch,
they will ask us for a sketch extension. Store this request to respond later.
Sending an extension may allow the peer to reconcile
transactions, because now the full sketch has twice
as much capacity.
If a peer sent us an extension sketch, we should
reconstruct a full sketch from it with the snapshot
we stored initially, and attempt to decode the difference.
p2p: Handle sketch extension
If a peer sent us an extension sketch, we should
reconstruct a full sketch from it with the snapshot
we stored initially, and attempt to decode the difference.
This currently unused function is supposed to be used once
a reconciliation round is done. It cleans the state corresponding
to the passed reconciliation.
Once a peer tells us reconciliation is done, we should behave as follows:
- if it was successful, just respond them with the transactions they asked
by short ID.
- if it was a full failure, respond with all local transactions from the reconciliation
set snapshot
- if it was a partial failure (only low or high part was failed after a bisection),
respond with all transactions which were asked for by short id,
and announce local txs which belong to the failed chunk.
I'm assuming Gleb didn't want to have transactions collisions in this
tests, but some are manifesting in CI, this is due to how coins are selected.
Fixing this plus making it a bit more idiomatic. This can be squashed to the
previous commit
First, extend the timeout to facilitate reconciliation
relay which might take longer than previously.
Second, since reconciliation can't guarantee that
descendants of same-degree (from the parent) are
relayed in the same order they were received, loosen
the check for descendant limit.
This is the case because, if the parent is flooded,
the children might be both flooded and reconciled,
and the decision is made independently. So even if
one arrived earlier, it can be relayed later because
flooding.
Note, it's possible to make txreconciliation.cpp be aware
of the order of adding transactions to the set, but this
won't help to solve the above issue.
🐙 This pull request conflicts with the target branch and needs rebase.
⌛ There hasn’t been much activity lately and the patch still needs rebase. What is the status here?
Labels
Needs rebase