This is a followup to #32631.
Addresses: #32631 (review) #32631 (review) #32631 (review) #32631 (review) #32631 (comment)
This is a followup to #32631.
Addresses: #32631 (review) #32631 (review) #32631 (review) #32631 (review) #32631 (comment)
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33005.
See the guideline for information on the review process.
Type | Reviewers |
---|---|
ACK | glozow, stickies-v, maflcko |
Approach ACK | ajtowns |
Stale ACK | hodlinator, janb84 |
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
@JeremyRubin in your comment you mentioned that there are multiple data structures that should be switched to homogeneous. I was only able to spot m_tx_inventory_to_send
as one that should be changed. Are there others that I’m missing?
If it is only the one, then I’m not sure about making such a generic container. Might be better to just keep it specific to Txid/Wtxid or have a class for m_tx_inventory_to_send
only.
🚧 At least one of the CI tasks failed.
Task lint
: https://github.com/bitcoin/bitcoin/runs/46205797248
LLM reason (✨ experimental): The CI failure is caused by a missing include guard in src/util/variantset.h, as reported by the lint check.
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
302@@ -302,7 +303,7 @@ struct Peer {
303 * non-wtxid-relay peers, wtxid for wtxid-relay peers). We use the
304 * mempool to sort transactions in dependency order before relay, so
305 * this does not have to be sorted. */
306- std::set<GenTxid> m_tx_inventory_to_send GUARDED_BY(m_tx_inventory_mutex);
307+ VariantSet<Txid, Wtxid> m_tx_inventory_to_send GUARDED_BY(m_tx_inventory_mutex);
VariantSet
solely for this purpose? We can just keep a set of Wtxid
s and, if the peer does not support wtxidrelay, grab the txid from info.tx
(which we have to look up anyway) to construct the inv message. It’s also going to be Wtxids pretty much 100% of the time.
5762 peer->m_wtxid_relay ? MSG_WTX : MSG_TX,
5763- peer->m_wtxid_relay ?
5764- txinfo.tx->GetWitnessHash().ToUint256() :
5765- txinfo.tx->GetHash().ToUint256(),
5766+ peer->m_wtxid_relay ? wtxid.ToUint256() : txid.ToUint256(),
5767 };
nit: Could be made more consistent with: https://github.com/bitcoin/bitcoin/blob/e991cac062ca644f3c4ac5c5144eb0daf445fd09/src/net_processing.cpp#L5815-L5817
0diff --git a/src/net_processing.cpp b/src/net_processing.cpp
1index ebb3db39a5..f38dc4b90d 100644
2--- a/src/net_processing.cpp
3+++ b/src/net_processing.cpp
4@@ -5758,10 +5758,9 @@ bool PeerManagerImpl::SendMessages(CNode* pto)
5 for (const auto& txinfo : vtxinfo) {
6 const auto& txid{txinfo.tx->GetHash()};
7 const auto& wtxid{txinfo.tx->GetWitnessHash()};
8- CInv inv{
9- peer->m_wtxid_relay ? MSG_WTX : MSG_TX,
10- peer->m_wtxid_relay ? wtxid.ToUint256() : txid.ToUint256(),
11- };
12+ CInv inv = peer->m_wtxid_relay ?
13+ CInv{MSG_WTX, wtxid.ToUint256()} :
14+ CInv{MSG_TX, txid.ToUint256()};
15 tx_relay->m_tx_inventory_to_send.erase(wtxid);
16
17 // Don't send transactions that peers will not put into their mempool
18@@ -5813,8 +5812,8 @@ bool PeerManagerImpl::SendMessages(CNode* pto)
19 continue;
20 }
21 const auto inv = peer->m_wtxid_relay ?
22- CInv(MSG_WTX, wtxid.ToUint256()) :
23- CInv(MSG_TX, txinfo.tx->GetHash());
24+ CInv{MSG_WTX, wtxid.ToUint256()} :
25+ CInv{MSG_TX, txinfo.tx->GetHash()};
26 // Check if not in the filter already
27 if (tx_relay->m_tx_inventory_known_filter.contains(inv.hash)) {
28 continue;
crACK e991cac062ca644f3c4ac5c5144eb0daf445fd09
Thanks for doing the follow-up!
The major change is that Peer::TxRelay::m_tx_inventory_to_send
now uses Wtxid
instead of GenTxid
while the Peer::TxRelay::m_tx_inventory_known_filter
bloom filter still uses either Wtxid
or Txid
consistently depending on the type of peer.
Only one style-nit in case you re-touch.
97@@ -98,7 +98,7 @@ class GenTxid : public std::variant<Txid, Wtxid>
98
99 friend auto operator<=>(const GenTxid& a, const GenTxid& b)
100 {
101- return std::tuple(a.IsWtxid(), a.ToUint256()) <=> std::tuple(b.IsWtxid(), b.ToUint256());
102+ return std::tuple<bool, const uint256&>(a.IsWtxid(), a.ToUint256()) <=> std::tuple<bool, const uint256&>(b.IsWtxid(), b.ToUint256());
const&
and the reason for it?
2401- auto txinfo{std::visit(
2402- [&](const auto& id) EXCLUSIVE_LOCKS_REQUIRED(NetEventsInterface::g_msgproc_mutex) {
2403- return m_mempool.info_for_relay(id, tx_relay.m_last_inv_sequence);
2404- },
2405- gtxid)};
2406+ auto txinfo{m_mempool.info_for_relay(id, tx_relay.m_last_inv_sequence)};
GenTxid
and keeping the visit
inside the function (rather than an if
outside it along with templatizing) seems nicer?
Fair, it does feel cleaner using GenTxid and avoiding the branch in ProcessGetData
. I chose to templatize in response to this comment from the original PR. The templating takes care of everything at compile-time, which is nice. But m_most_recent_block_txs
holds GenTxids, so in the templated version we’re creating a temporary GenTxid anyway for that map lookup vs creating a GenTxid once upfront.
I think it’s largely a stylistic choice and don’t have a strong preference either way. @stickies-v @dergoegge if you have anything to add, feel free.
edit: Of course, if there’s something I’m missing as to why one approach is clearly better, let me know
CInv&
. Seems fine to at least restore this as an Assume?
You might consider:
0
1diff --git a/src/net_processing.cpp b/src/net_processing.cpp
2index 10fbae5e7ed..42102973c31 100644
3--- a/src/net_processing.cpp
4+++ b/src/net_processing.cpp
5@@ -947,7 +947,7 @@ private:
6 std::atomic<std::chrono::seconds> m_last_tip_update{0s};
7
8 /** Determine whether or not a peer can request a transaction, and return it (or nullptr if not found or not allowed). */
9- CTransactionRef FindTxForGetData(const Peer::TxRelay& tx_relay, const CInv& inv)
10+ CTransactionRef FindTxForGetData(const Peer::TxRelay& tx_relay, const GenTxid& gtxid)
11 EXCLUSIVE_LOCKS_REQUIRED(!m_most_recent_block_mutex, NetEventsInterface::g_msgproc_mutex);
12
13 void ProcessGetData(CNode& pfrom, Peer& peer, const std::atomic<bool>& interruptMsgProc)
14@@ -2391,15 +2391,10 @@ void PeerManagerImpl::ProcessGetBlockData(CNode& pfrom, Peer& peer, const CInv&
15 }
16 }
17
18-CTransactionRef PeerManagerImpl::FindTxForGetData(const Peer::TxRelay& tx_relay, const CInv& inv)
19+CTransactionRef PeerManagerImpl::FindTxForGetData(const Peer::TxRelay& tx_relay, const GenTxid& gtxid)
20 {
21- auto gtxid{ToGenTxid(inv)};
22 // If a tx was in the mempool prior to the last INV for this peer, permit the request.
23- auto txinfo{std::visit(
24- [&](const auto& id) EXCLUSIVE_LOCKS_REQUIRED(NetEventsInterface::g_msgproc_mutex) {
25- return m_mempool.info_for_relay(id, tx_relay.m_last_inv_sequence);
26- },
27- gtxid)};
28+ auto txinfo = m_mempool.info_for_relay(gtxid, tx_relay.m_last_inv_sequence);
29 if (txinfo.tx) {
30 return std::move(txinfo.tx);
31 }
32@@ -2442,7 +2437,7 @@ void PeerManagerImpl::ProcessGetData(CNode& pfrom, Peer& peer, const std::atomic
33 continue;
34 }
35
36- if (auto tx{FindTxForGetData(*tx_relay, inv)}) {
37+ if (auto tx{FindTxForGetData(*tx_relay, ToGenTxid(inv))}) {
38 // WTX and WITNESS_TX imply we serialize with witness
39 const auto maybe_with_witness = (inv.IsMsgTx() ? TX_NO_WITNESS : TX_WITH_WITNESS);
40 MakeAndPushMessage(pfrom, NetMsgType::TX, maybe_with_witness(*tx));
41diff --git a/src/txmempool.h b/src/txmempool.h
42index 687c474f8e0..31e168bc277 100644
43--- a/src/txmempool.h
44+++ b/src/txmempool.h
45@@ -652,12 +652,13 @@ public:
46 }
47
48 /** Returns info for a transaction if its entry_sequence < last_sequence */
49- template <TxidOrWtxid T>
50- TxMempoolInfo info_for_relay(const T& id, uint64_t last_sequence) const
51+ TxMempoolInfo info_for_relay(const GenTxid& gtxid, uint64_t last_sequence) const
52 {
53- LOCK(cs);
54- auto i{GetIter(id)};
55- return (i.has_value() && i.value()->GetSequence() < last_sequence) ? GetInfo(*i) : TxMempoolInfo{};
56+ return std::visit([&](const auto& id) {
57+ LOCK(cs);
58+ auto i{GetIter(id)};
59+ return (i.has_value() && i.value()->GetSequence() < last_sequence) ? GetInfo(*i) : TxMempoolInfo{};
60+ }, gtxid);
61 }
62
63 std::vector<CTxMemPoolEntryRef> entryAll() const EXCLUSIVE_LOCKS_REQUIRED(cs);
Or adding GetIter(GenTxid)
could work too:
0 std::optional<txiter> GetIter(const GenTxid& gtxid) const EXCLUSIVE_LOCKS_REQUIRED(cs)
1 {
2 return std::visit([&](const auto& id) EXCLUSIVE_LOCKS_REQUIRED(cs) {
3 return GetIter(id);
4 }, gtxid);
5 }
6
7 /** Returns info for a transaction if its entry_sequence < last_sequence */
8 TxMempoolInfo info_for_relay(const GenTxid& gtxid, uint64_t last_sequence) const
9 {
10 LOCK(cs);
11 auto i{GetIter(gtxid)};
12 return (i.has_value() && i.value()->GetSequence() < last_sequence) ? GetInfo(*i) : TxMempoolInfo{};
13 }
I opted for passing in a GenTxid to FindTxForGetData
and doing the visit there over the templated version for a couple of reasons:
m_most_recent_block_txs
.ToGenTxid
.I’m keeping the functions in txmempool
Txid/Wtxid, as after this PR there won’t be any GenTxids at all in the current mempool code, which is nice.
below you are removing the assert on the
CInv&
. Seems fine to at least restore this as an Assume?
This approach takes care of this as well.
Sounds fine.
Another option would be to move info_for_relay
from txmempool.h into net_processing.cpp:
0/** Returns info for a transaction if its entry_sequence < last_sequence */
1static TxMempoolInfo info_for_relay(const CTxMemPool& mempool, const GenTxid& gtxid, uint64_t last_sequence)
2{
3 return std::visit([&](const auto& id) {
4 LOCK(mempool.cs);
5 auto i{mempool.GetIter(id)};
6 return (i.has_value() && i.value()->GetSequence() < last_sequence) ? mempool.GetInfo(*i) : TxMempoolInfo{};
7 }, gtxid);
8}
9
10// FindTxForGetData:
11 auto txinfo = info_for_relay(m_mempool, gtxid, tx_relay.m_last_inv_sequence);
That removes p2p-specific code from the mempool (and kernel), keeps the mempool with specific Txid/Wtxid things, and simplifies the locking logic a little. Requires exposing CTxMemPool::GetInfo()
as public
, though that doesn’t seem terrible afaics.
info_for_relay
to net_processing.cpp
absolutely makes sense to me, that logic should not be in mempool interface. However, leaking mempool internals into p2p (through mempool.GetIter(id)
) I really don’t like.
GetIter
is already exposed, it’s used in validation.cpp for looking up txs. This would also expose GetInfo
which converts an iterator into a index-neutral summary of info about the tx.
tl;dr - I think the current implementation a9819b0e9d3c74970a94cb674fe8fd771e60f6df (i.e. ajtown’s suggestion in #33005 (review)) is pragmatically the best way forward.
GetIter
is already exposed
I think “still” is more appropriate than “already”, see e.g. work done in #28391. Non-mempool code using mapTx
iterators imo remains an anti-pattern and we should strive to reduce, not increase it.
That removes p2p-specific code from the mempool (and kernel)
Looking at the code in more detail, I actually think the only thing p2p-specific about the function is the name. E.g. the function still makes sense if we turn it into a TxMempoolInfo info(const T& id, uint64_t last_sequence) const
overload. Besides the sequence number, it doesn’t use any p2p symbols or logic, so I think keeping as-is makes sense for now.
The true p2p-specific code in the mempool appears to be the existence of sequence numbers, but getting rid of that seems like a much bigger code change than is warranted for this follow-up PR.
The true p2p-specific code in the mempool appears to be the existence of sequence numbers, but getting rid of that seems like a much bigger code change than is warranted for this follow-up PR.
I have a bunch of thoughts about separating “kernel” and “relay” mempool logic like this… hmm, maybe should just do a brain dump to a gist sometime.
Non-mempool code using
mapTx
iterators imo remains an anti-pattern
I think if they were (treated as) opaque handles and you had to use GetInfo()
or similar accessors to get information out from them, that would be fine fwiw.
hmm, maybe should just do a brain dump to a gist sometime.
Please cc me if you do, would be very interested to see that.
I think if they were (treated as) opaque handles and you had to use
GetInfo()
… that would be fine fwiw.
I agree. Although not opaque, I think that’s pretty much what the GetEntry()
methods and CTxMemPoolEntry
should support doing? But most of the mempool interface is still expecting iterators rather than entries, so it would require a bit of adding/changing methods. Might be a sensible project after cluster mempool?
Addresses a few comments from #32631:
https://github.com/bitcoin/bitcoin/pull/32631#discussion_r2199951996
https://github.com/bitcoin/bitcoin/pull/32631#discussion_r2201874252
https://github.com/bitcoin/bitcoin/pull/32631#discussion_r2201918072
Addresses
https://github.com/bitcoin/bitcoin/pull/32631#discussion_r2200291621
from #32631
301@@ -302,7 +302,7 @@ struct Peer {
302 * non-wtxid-relay peers, wtxid for wtxid-relay peers). We use the
303 * mempool to sort transactions in dependency order before relay, so
304 * this does not have to be sorted. */
305- std::set<GenTxid> m_tx_inventory_to_send GUARDED_BY(m_tx_inventory_mutex);
306+ std::set<Wtxid> m_tx_inventory_to_send GUARDED_BY(m_tx_inventory_mutex);
2165@@ -2166,9 +2166,9 @@ void PeerManagerImpl::RelayTransaction(const Txid& txid, const Wtxid& wtxid)
2166 // in the announcement.
2167 if (tx_relay->m_next_inv_send_time == 0s) continue;
2168
2169- const auto gtxid{peer.m_wtxid_relay ? GenTxid{wtxid} : GenTxid{txid}};
2170- if (!tx_relay->m_tx_inventory_known_filter.contains(gtxid.ToUint256())) {
2171- tx_relay->m_tx_inventory_to_send.insert(gtxid);
2172+ const uint256& hash{peer.m_wtxid_relay ? wtxid.ToUint256() : txid.ToUint256()};
2173+ if (!tx_relay->m_tx_inventory_known_filter.contains(hash)) {
m_tx_inventory_known_filter
should contain sometimes-txids instead of wtxids?
concept ACK 8e19a4375e1db6e59cdcb38e0cc3e1dab79e88ec
This PR is a followup of #32631 which addresses some of the NITS / comments that were placed in that PR. Additionally the PR simplifies m_tx_inventory_to_send
a bit to make it std::set<Wtxid>
based in stead of std::set<GenTxid>
imho a good change.
5819 if (!txinfo.tx) {
5820 continue;
5821 }
5822+ const auto inv = peer->m_wtxid_relay ?
5823+ CInv{MSG_WTX, wtxid.ToUint256()} :
5824+ CInv{MSG_TX, txinfo.tx->GetHash().ToUint256()};
m_tx_inventory_known_filter
hashes are based on the wtxidrelay-ness of the peer, so it’s important that we compare the hash from inv
and not the wtxid that was in m_tx_inventory_to_send
.
Simplifies `m_tx_inventory_to_send` a bit by making it a set of Wtxids.
Wtxid relay is prevalent throughout the network, so the complexity of
dealing with a GenTxid in this data structure isn't necessary.
For peers that aren't wtxid relay, the txid is now retrieved from our
mempool entry when the inv is constructed.
Most recent push updates the m_tx_inventory_to_send
comment and adds a clarifying comment as to why we create the inv before checking m_tx_inventory_known_filter
.
I’m still considering #33005 (review).
5764- tx_relay->m_tx_inventory_to_send.erase(ToGenTxid(inv));
5765+ const Txid& txid{txinfo.tx->GetHash()};
5766+ const Wtxid& wtxid{txinfo.tx->GetWitnessHash()};
5767+ const auto inv = peer->m_wtxid_relay ?
5768+ CInv{MSG_WTX, wtxid.ToUint256()} :
5769+ CInv{MSG_TX, txid.ToUint256()};
nit: txid
declaration not really helpful imo
0 const Wtxid& wtxid{txinfo.tx->GetWitnessHash()};
1 const auto inv = peer->m_wtxid_relay ?
2 CInv{MSG_WTX, wtxid.ToUint256()} :
3 CInv{MSG_TX, txinfo.tx->GetHash().ToUint256()};
5780@@ -5783,9 +5781,9 @@ bool PeerManagerImpl::SendMessages(CNode* pto)
5781 // Determine transactions to relay
5782 if (fSendTrickle) {
5783 // Produce a vector with all candidates for sending
5784- std::vector<std::set<GenTxid>::iterator> vInvTx;
5785+ std::vector<std::set<Wtxid>::iterator> vInvTx;
nit: this is touching a lot more code than you are, but modernizing this bit of code would make it more readable:
0diff --git a/src/net_processing.cpp b/src/net_processing.cpp
1index 165f6e822f..a9dafb7160 100644
2--- a/src/net_processing.cpp
3+++ b/src/net_processing.cpp
4@@ -5780,28 +5780,23 @@ bool PeerManagerImpl::SendMessages(CNode* pto)
5
6 // Determine transactions to relay
7 if (fSendTrickle) {
8- // Produce a vector with all candidates for sending
9- std::vector<std::set<Wtxid>::iterator> vInvTx;
10- vInvTx.reserve(tx_relay->m_tx_inventory_to_send.size());
11- for (std::set<Wtxid>::iterator it = tx_relay->m_tx_inventory_to_send.begin(); it != tx_relay->m_tx_inventory_to_send.end(); it++) {
12- vInvTx.push_back(it);
13- }
14 const CFeeRate filterrate{tx_relay->m_fee_filter_received.load()};
15 // Topologically and fee-rate sort the inventory we send for privacy and priority reasons.
16- // A heap is used so that not all items need sorting if only a few are being sent.
17 CompareInvMempoolOrder compareInvMempoolOrder(&m_mempool);
18- std::make_heap(vInvTx.begin(), vInvTx.end(), compareInvMempoolOrder);
19+ auto tx_inv_iters{std::views::iota(tx_relay->m_tx_inventory_to_send.begin(),
20+ tx_relay->m_tx_inventory_to_send.end())};
21+ // A priority queue is used so that not all items need sorting if only a few are being sent.
22+ std::priority_queue tx_inv_to_send{tx_inv_iters.begin(), tx_inv_iters.end(),
23+ compareInvMempoolOrder};
24 // No reason to drain out at many times the network's capacity,
25 // especially since we have many peers and some will draw much shorter delays.
26 unsigned int nRelayedTransactions = 0;
27 LOCK(tx_relay->m_bloom_filter_mutex);
28 size_t broadcast_max{INVENTORY_BROADCAST_TARGET + (tx_relay->m_tx_inventory_to_send.size()/1000)*5};
29 broadcast_max = std::min<size_t>(INVENTORY_BROADCAST_MAX, broadcast_max);
30- while (!vInvTx.empty() && nRelayedTransactions < broadcast_max) {
31- // Fetch the top element from the heap
32- std::pop_heap(vInvTx.begin(), vInvTx.end(), compareInvMempoolOrder);
33- std::set<Wtxid>::iterator it = vInvTx.back();
34- vInvTx.pop_back();
35+ while (!tx_inv_to_send.empty() && nRelayedTransactions < broadcast_max) {
36+ std::set<Wtxid>::iterator it = tx_inv_to_send.top();
37+ tx_inv_to_send.pop();
38 auto wtxid = *it;
39 // Remove it from the to-be-sent set
40 tx_relay->m_tx_inventory_to_send.erase(it);
2399 auto txinfo{std::visit(
2400 [&](const auto& id) EXCLUSIVE_LOCKS_REQUIRED(NetEventsInterface::g_msgproc_mutex) {
2401 return m_mempool.info_for_relay(id, tx_relay.m_last_inv_sequence);
2402 },
2403 gtxid)};
2404+
review ACK 94b39ce73831acc4c94c7f0d1347d5991b27ef0b 🎲
Signature:
0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: review ACK 94b39ce73831acc4c94c7f0d1347d5991b27ef0b 🎲
3oeMvE0HV8g7Aop9tgh9W10FYJLCk8QCUAj7Iy4uReCklfeTSpvizOzp9hdytYtSC1Y+hsNfZUHITsxIooV24Dw==
2390@@ -2391,15 +2391,14 @@ void PeerManagerImpl::ProcessGetBlockData(CNode& pfrom, Peer& peer, const CInv&
2391 }
2392 }
2393
2394-CTransactionRef PeerManagerImpl::FindTxForGetData(const Peer::TxRelay& tx_relay, const CInv& inv)
2395+CTransactionRef PeerManagerImpl::FindTxForGetData(const Peer::TxRelay& tx_relay, const GenTxid& gtxid)
2396 {
2397- auto gtxid{ToGenTxid(inv)};
2398- // If a tx was in the mempool prior to the last INV for this peer, permit the request.
marcofleon
DrahtBot
JeremyRubin
glozow
hodlinator
ajtowns
maflcko
stickies-v
janb84
Labels
Refactoring
Milestone
30.0