Erlay Project Tracking: #30249
This is a full implementation of Erlay. Its purpose is to check the integrity and correctness of the implementation against changes/additions that may originate from the review process and/or rebases on top of newer functionality.
This is not to be merged. Functionality will be spread across multiple smaller PRs to ease the review process.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/30277.
See the guideline for information on the review process. A summary of reviews will appear here.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
Possible typos and grammar issues:
In src/node/txreconciliation.cpp comment:
“Allows to infer capacity of a reconciliation sketch based on it’s char[] representation”
should be
“its char[] representation”
In test/functional/p2p_reqtxrcncl.py comment:
“Each peer is reconcile every 1-nth”
should be
“Each peer is reconciled every 1-nth”
In test/functional/p2p_reqtxrcncl.py log message:
“REQTXRCNCL sent to an peer 1 (outbound)”
should be
“REQTXRCNCL sent to a peer 1 (outbound)”
In test/functional/p2p_reqtxrcncl.py comment:
“No transaction were created, so we expect the set size to be 0”
should be
“No transactions were created, so we expect the set size to be 0”
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.
Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
Leave a comment here, if you need help tracking down a confusing failure.
Feel free to grab https://github.com/hebasto/bitcoin/commit/ffed2ab27465e4e3888b4139ca50db1226a66ec5 to fix the MSVC build.
🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/40538402994
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
These comments became irrelevant in one of the previous code changes.
They simply don't make sense anymore.
They will be used later on.
It helps to avoid recomputing every time we consider
a transaction for fanout/reconciliation.
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
Fanout or reconciliation is decided on a transaction basis, based on the following criteria:
If the peer is inbound, we fanout to a pre-defined subset of peers (which is rotated periodically).
If the peer is outbound, we will reconcile the transaction if we received it via reconciliation, or
defer the decision to relay time otherwise. At relay time, we will fanout to outbounds until a threshold is met
(selecting peers in the order their timers go off) and reconcile with the rest.
With this approach we try to fanout when we estimate to be early in the propagation of the transaction,
and reconcile otherwise. Notice these heuristics don't apply to inbound peers, since they would be easily
exploitable. For inbounds we just aim for a target subset picked at random.
Transactions eligible for reconciliation are added to the reconciliation sets. For the remaining txs, low-fanout is used.
Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
This can be squashed into the previous commit, it is split for now to ease review
When scheduling the relay of a transaction (RelayTransaction) we should consider
whether it is worth fanning it out, or only reconciling it. This depends, partially,
on how the transaction was received.
For non-Erlay peers, we always consider_fanout (in fact, we only fanout).
For Erlay peers, if the peer is inbounds, we always consider fanout, and deffer the
decision of to whom to relay time. If the peer is outbound, we consider fanout if the
transaction was received via fanout, and only reconcile if it was received via reconciliation
(who to fanout to is also deferred to relay time).
Until the Erlay P2P flow is merged, consider_fanout is always true.
If a transaction to be added to a peer's recon set has a shot id collisions (a previously
added wtxid maps to the same short id), both transaction should be fanout, given
our peer may have added the opposite transaction to our recon set, and these two
transaction won't be reconciled.
When we're finalizing negotiation, we should add the peers
for which we will initiate reconciliations to the queue.
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
We initiate reconciliation by looking at the queue periodically
with equal intervals between peers to achieve efficiency.
This will be later used to see whether it's time to initiate.
When the time comes for the peer, we send a
reconciliation request with the parameters which
will help the peer to construct a (hopefully) sufficient
reconciliation sketch for us. We will then use that
sketch to find missing transactions.
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
Store the parameters the peer sent us inside the
reconciliation request.
🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/40799139881
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
When the time comes, we should send a sketch of our
local reconciliation set to the reconciliation initiator.
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
When the sketches from both sides are combined successfully,
the diff is produced. Then this diff can (together with the local txs)
be used to identified which transactions are missing locally and remotely.
This will help to reuse the code later on in the function
to announce transactions.
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
If after decoding a reconciliation sketch it turned out
to be insufficient to find set difference, request extension.
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
Store the initial sketches so that we are able to process
extension sketch while avoiding transmitting the same data.
Co-authored-by: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
To be ready to respond to a sketch extension request
from our peer, we should store a snapshot of our state
and capacity of the initial sketch, so that we compute
extension of the same size and over the exact same
transactions.
Transactions arriving during this reconciliation will
be instead stored in the regular set.
Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
If peer failed to reconcile based on our initial response sketch,
they will ask us for a sketch extension. Store this request to respond later.
Sending an extension may allow the peer to reconcile
transactions, because now the full sketch has twice
as much capacity.
If a peer sent us an extension sketch, we should
reconstruct a full sketch from it with the snapshot
we stored initially, and attempt to decode the difference.
p2p: Handle sketch extension
If a peer sent us an extension sketch, we should
reconstruct a full sketch from it with the snapshot
we stored initially, and attempt to decode the difference.
This currently unused function is supposed to be used once
a reconciliation round is done. It cleans the state corresponding
to the passed reconciliation.
Once a peer tells us reconciliation is done, we should behave as follows:
- if it was successful, just respond them with the transactions they asked
by short ID.
- if it was a full failure, respond with all local transactions from the reconciliation
set snapshot
- if it was a partial failure (only low or high part was failed after a bisection),
respond with all transactions which were asked for by short id,
and announce local txs which belong to the failed chunk.
We may still need to add more tests, specially around extensions (if we keep them)
Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
🐙 This pull request conflicts with the target branch and needs rebase.
Labels
Needs rebase