[RFC] Package-relay: sender-initiated

ariard commented at 7:55 pm on July 29, 2020: member

This is demo code for a sender-initiated version of package-relay. This is not proposed for merging, only to illustrate package relay design with trade-offs compared to #16401. See my update on #14895 to advance discussions on package-relay.

Few design choices of interest:

introduce new p2p messages, sendpackages, MSG_PACKAGE, package and new package_id identifier incorporated to the INV/GETDATA control logic : https://gist.github.com/ariard/10924249b3b7f6e239d80895372839b8
add a new sendpackage rpc to let a higher-application signals a chain of transactions and such start a propagation on the p2p network
integrate a PackageCache to internally cache the mapping package_id -> txids/wtxids, as they need to be persistent across a round-trip between upstream/downstream peers, it must be DoS resistant
build on #16401 AcceptMultipleTransactions with a package policy restriction to 2-tx only, make it easier to reason on for replacement
allow for package replacement based on newer feerate higher that the union of feerate of all transactions conflicted, no dedup for now between conflicted set
don’t redundantly send a transaction via INV(MSG_TX) if we know an associated package and peer signal package relay support
TODO: integrate package_id with overhaul transaction request/AlreadyHave to save further on bandwidth

Add package-acceptance logic to mempool

Accepting a single transaction to the mempool only succeeds if (among other
things) the feerate of the transaction is greater than both the min relay fee
and the mempool min fee. Consequently, a transaction below the minimum fee
may not be accepted to the mempool, even if we later learn of a transaction
with a high relay fee that depends on it.

This commit adds a function that will accept a package of transactions to the
mempool, with the following restrictions:

- All package transactions must be direct parents of the final transaction.
  This is a simple heuristic for ensuring that a candidate list of transactions
  is in fact a package (we wouldn't want arbitrary transactions to be paying
  for random low feerate transactions)

- The feerate of the package, as a whole, exceeds the mempool min fee and the
  min relay fee.

- No transactions in the mempool conflict with any transactions in the package.
  This is a simplification that makes the logic easier to write. Without this
  requirement, we would need to do additional checks to ensure that no parent
  transaction would evict a transaction from the mempool that some other child
  depends on.

- The ancestor/descendant size limits are calculated assuming that any mempool
  ancestor of any candidate transaction is an ancestor of all the candidate
  transactions.
  This allows for doing simpler calculations to ensure that we're staying
  within the mempool's package limits. If we eliminated this, we would need to
  do much more careful package calculations for each candidate transaction and each
  in-mempool ancestor.

This commit does not include any accessor function for utilizing this logic (eg
by exposing this function at the p2p or rpc layer).

e77fab05a0

Restrain package size to 2 transactions directly dependent

In next commit, we relax restriction on conflict replacement at package
acceptance. Package transactions may depend on conflicted ones replaced
by other package elements, thus making its evaluation expensive. To avoid
DoS risk while allowing feerate-increasing packages limit theirs composition
to 2 transactions, the latter only dependent on the former.

71c1451a36

Allow a package tx to bypass PreChecks() already-in/feerate-conflicts checks

A package may bundle a newer CPFP with an already-in subgraph. To
evaluate replacement correctly, collect older fees of each conflicted
transactions while omitting per-transaction fee/feerate checks in
PreChecks().

For now, we don't evict conflicting duplicate, as a transaction might be in
conflict with multiple package elements.

XXX: refactor PreChecks() to dedup in AcceptMultipleTransactions ?

aca8c361cc

Add PackageCache, a package_id -> set of wtxids map

To reduce bandwidth consumption due to package announcement, only
announce a 32-byte package_id, sha256 of all package transactions.

To avoid this cache being a DoS risk, allocate a number of package
announcements to each upstream peer. To guarantee flush, prune old
entries after a delay (PACKAGE_CACHE_DELAY), thus requiring downstream
package round-trip to achieve before.

dbb426bc26

Add p2p message PACKAGE support

This new p2p message contains a serialized list of transactions, which
upon receipt feerate of the whole must determine mempool acceptance.

1debaa50d1

Add CNodeState::m_package_relay 11efd4662f

Add a new inv type MSG_PACKAGE

The hash being referenced is a package_id.

d9a5ce1778

Integrate package-relay announcement

At INV sending, announce packages unknown to the processed downstream
peer. Any transaction part of a mapped package, won't be announced
redundantly.

e13889242b

Introduce package-relay getdata reply

Disallow getdata reply if this peer isn't assumed to have been
announced this package yet.

13955ea386

Add RelayPackage ace58efc3c

Introduce package processing by receiver 2f0e359f16

Add p2p message sendpackage

XXX: add byte flag to signal package policy enforced

657588cf11

Add BroadcastPackage bba7bfbe88

Add sendpackage RPC command

This command let caller submit a bundle of transactions to the local
mempool and p2p network, ensuring their summed feerate is going to be evaluated
as a package.

b3abd2fb0b

[qa] Exercise package relay e62c0a6e35

ariard renamed this:
~~[RFC] Package-relay : sender-initiated~~
[RFC] Package-relay: sender-initiated
on Jul 29, 2020

DrahtBot added the label Build system on Jul 29, 2020

DrahtBot added the label Mempool on Jul 29, 2020

DrahtBot added the label P2P on Jul 29, 2020

DrahtBot added the label RPC/REST/ZMQ on Jul 29, 2020

DrahtBot added the label Tests on Jul 29, 2020

DrahtBot added the label Validation on Jul 29, 2020

DrahtBot commented at 11:12 pm on July 29, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#19611 (p2p: refactor CInv::type from public int to private uint32_t by jonatack)
#19610 (p2p, refactor: add CInv block message helpers; use in net processing by jonatack)
#19607 ([p2p] Add Peer struct for per-peer data in net processing by jnewbery)
#19590 (p2p, refactor: add CInv transaction message helpers; use in net processing by jonatack)
#19573 ([WIP] Replace unused BIP 9 logic with BIP 8 by luke-jr)
#19569 (Enable fetching of orphan parents from wtxid peers by sipa)
#19438 (Introduce deploymentstatus by ajtowns)
#19339 (validation: re-delegate absurd fee checking from mempool to clients by gzhao408)
#19306 (refactor: Replace RecursiveMutex with Mutex in CTxMemPool by hebasto)
#18766 (Disable fee estimation in blocksonly mode by darosior)
#18261 (Erlay: bandwidth-efficient transaction relay protocol by naumenkogs)
#16365 (Log RPC parameters (arguments) if -debug=rpcparams by LarryRuane)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Needs rebase on Jul 30, 2020

DrahtBot commented at 2:28 pm on July 30, 2020: member

🐙 This pull request conflicts with the target branch and needs rebase.

Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a “draft”.

ariard marked this as a draft on Aug 6, 2020

DrahtBot commented at 11:21 am on December 15, 2021: member

Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
Is it no longer relevant? ➡️ Please close.
Did the author lose interest or time to work on this? ➡️ Please close it and mark it ‘Up for grabs’ with the label, so that it can be picked up in the future.

DrahtBot commented at 1:07 pm on March 21, 2022: member

Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
Is it no longer relevant? ➡️ Please close.
Did the author lose interest or time to work on this? ➡️ Please close it and mark it ‘Up for grabs’ with the label, so that it can be picked up in the future.

MarcoFalke commented at 1:37 pm on March 21, 2022: member

Closing for now. You can open a new pull request if you are working on this again, since this one doesn’t have any discussion.

MarcoFalke closed this on Mar 21, 2022

ariard commented at 7:14 pm on April 3, 2022: member

I confirm, this work has been superseded by the advances with package acceptance (https://gist.github.com/glozow/dc4e9d5c5b14ade7cdfac40f43adb18a). For the p2p part, the sender-initiated approach and the package ids caching might be of interest for the conceptual discussion when we introduce p2p packages. I don’t have interest to keep working on that version of package relay but I’m staying interested to review any work or draft in that area.

DrahtBot locked this on Apr 3, 2023

[RFC] Package-relay: sender-initiated #19621

Conflicts