Test Package Accept #32160

1. 
   glozow commented at 2:48 pm on March 28, 2025: member

   Please describe the feature you’d like to see added.

   Description of how the feature should work: The testmempoolaccept RPC should continue to take a list of hex transactions with minimal restrictions (no duplicates, no conflicting transactions, not too many). It should allow singleton transactions, transactions already in mempool, and any topology (multiple disconnected components should be ok). It should attempt to validate as much as possible and return each transaction’s validation result. Of course, it should not modify mempool contents while doing so, and most importantly it should simulate the fee-bumping policies like package CPFP and package RBF.

   Problems today: We added multi-transaction support in testmempoolaccept in #20833, before package policies were decided. It enabled some of the validation functionality that package validation uses, but ultimately has a different codepath and thus different interface. It has roughly the input requirements that we want (no duplicates, no conflicts, max 25, topological). Its results array format is also fine in my opinion. The biggest problem is that it doesn’t apply package policies. So if you have a 1p1c package with the parent bumped by the child, testmempoolaccept says that the parent has too low feerate, but submitpackage says it’s accepted. This is confusing.

   Why fixing this is hard: (1) Our method of “splitting” a package is not compatible with test acceptance because it is trial-and-error based. Instead, it should decide what subpackages will be up front. (2) We can’t stage and unstage the changes made by each subpackage on top of the changes made by previous subpackages without applying the changes to mempool.

   Longer form explanation:

   • Supporting test package acceptance arbitrary lists of transactions, or indeed any list of transactions that isn’t a two-generation tree, requires we first analyze the package and decide what subpackages or chunks to submit together, and in what order. These decisions can very much affect whether/which transactions get in. The decision should involve the fee and vsize of these transactions, which requires fetching UTXOs and linearizing them.
   • Today, we achieve “splitting” through a trial-and-error process. We continuously attempt to submit package subsets in increasing size order (i.e. starting individually), excluding things once they have been successfully submitted. This was the convenient way to implement it because AcceptPackage, and it’s not a terrible way to split two generation-packages.
   • Given our current setup, package transactions are either in the mempool or we haven’t yet decided if they’re valid; we don’t have an intermediate stage of transactions that have been fully “approved” but not yet submitted. Once something fails, we just quit out and don’t try the rest of the transactions in the package. Often, they get a “missing inputs” error, which is equivalent to “depends on an invalid transaction”.
   • In git terms: We want the ability to git commit each subpackage, and then merge the branch with multiple commits into master, or just look at the branch log and use it to produce the RPC results. Today, we can’t git branch, we can only stage changes and either discard all of them or commit them directly on master.

   “But if it’s so hard to keep going after a failure, why do it? How much do we care about continuing to validate the package after a subpackage has failed?” For a package we receive over p2p, we should keep going instead of wasting bandwidth downloading the same transactions until we get the exact right combination.

   “Does it make things easier or more sensible to not support disconnected transactions / distinct clusters?” After deduplication, it is possible that our package contains disconnected transactions, even if we define packages as ancestors or prefixes of some target transaction (i.e. the protocol requires the package to be connected when we receive it over p2p). As an example, imagine it’s us + our parent + a sibling in the same chunk, and the parent is deduplicated. Another reason is that we’d like package test acceptance to be as helpful as possible, and potentially accept lists of raw transactions that aren’t all connected. DepGraph handles disconnected transactions just the same, so the linearization part is not more difficult.

   Describe the solution you’d like

   Here is a proposed solution, which builds off of the package RBF outline in this delving post, simplifying out some of the RBF details and focusing on the changeset staging.

   1. Deduplication: remove transactions that are already in mempool.
   2. Topological linearization: sort it topologically.
   3. UTXO fetching, in which we learn the fee and virtual sizes of these transactions. This allows us to build a standalone instance of DepGraph or a TxGraph of the package, i.e. not connected to mempool transactions. This is probably done through a PreChecks call, so we can save and exclude any standardness failures.
   4. Pre-linearization: linearize the package transactions. This is without mempool transactions. Decide on what the chunks i.e. subpackages will be.
   5. Use TxGraph::StartStaging to create a level 1.
   6. Splitting for each chunk CNK:
      1. Use TxGraph::StartStaging to create a level 2.
      2. Validate: Limiting, Conflict-finding, Replacement checks, Verification, up to and including PolicyScriptChecks. When doing replacement and diagram checks, always compare the top level with the one just below it, not with the Main level.
      3. Commit these changes using TxGraph::CommitStaging to level 1, not level 0 which represents what is in the mempool.
      4. The chunk can also be discarded if it is invalid or doesn’t pass its RBF requirements, i.e. TxGraph::AbortStaging. The other chunks’ changes in the level 1 are retained.
   7. Full Addition and Eviction should happen at the end, i.e. ConsensusScriptChecks and ChangeSet::Apply and TxGraph::CommitStaging applying the changes from level 1 to level 0.

   Please leave any additional context

   Takeaways and open questions:

   • I think we could do an extremely limited version now by rerouting test-accepts through AcceptPackage and making a few logical tweaks (AcceptSingleTransaction(test_accept=true) followed by and AcceptMultipleTransactions(test_accept=true) works for 1p1c). But I don’t think this is worth it.
   • Is there a simple way to implement the full feature without doing the package validation restructuring described above? I don’t really think so.
   • Like a lot of things, implementing will be way easier with the cluster mempool changes merged. However, we need to modify TxGraph to support up to 3 levels.
     • An alternative is to try to build an “UndoSubpackageChanges” external to TxGraph. I think that will be pretty complex and more levels seems more natural.
   • I think we can delete MiniMiner::Linearize since it was written for this purpose, and we can now use cluster_linearize instead.
   • Do we need to make a new RPC given we’re changing the interface? I’d lean towards no since this is a loosening and basically playing catchup for supporting packages.
2. glozow added the label Feature on Mar 28, 2025
3. glozow added the label Mempool on Mar 28, 2025

Contributors
glozow

Labels
Feature Mempool