Remove the Broadcast()
/ResendWalletTransactions()
notification from the Validation interface.
Closes #15619. See that issue for discussion.
Obviously requires careful testing. We don’t currently have any automated testing for wallet rebroadcasts (wallet_resendwallettransactions.py()
basically only tests the resendwallettransactions
rpc method, and doesn’t test whether:
ResendWalletTransactions()
is called on a timerResendWalletTransactions()
actually causes txs to be rebroadcastThe following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
This seems to add a lot of lines of code for something intended to be a simplification?
Only the last 3 commits are relevant here. The first 4 commits come from #10973 which would conflict with this PR, so John chose it as a base, but is actually independent. This PR could just as easily be based on master.
The first 4 commits come from #10973 which would conflict with this PR
Indeed. It didn’t make sense to make or review changes in the node<->wallet interface while Russ’s PR was open. It’s merged now so I’ve rebased on master.
Changes are now +28/-35
CScheduler
and SendMessages()
code well enough to be extremely confident that this change doesn’t break anything, but assuming CWallet::ResendWalletTransactions()
is still called when it needs to be called, everything else looks correct to me and the changes are a nice simplification.
2137@@ -2138,8 +2138,13 @@ std::vector<uint256> CWallet::ResendWalletTransactionsBefore(interfaces::Chain::
2138 return result;
2139 }
2140
2141-void CWallet::ResendWalletTransactions(interfaces::Chain::Lock& locked_chain, int64_t nBestBlockTime)
2142+// Resend wallet transactions that haven't gotten in a block yet
0// Resend wallet transactions that haven't gotten in a block yet;
2137@@ -2138,8 +2138,13 @@ std::vector<uint256> CWallet::ResendWalletTransactionsBefore(interfaces::Chain::
2138 return result;
2139 }
2140
2141-void CWallet::ResendWalletTransactions(interfaces::Chain::Lock& locked_chain, int64_t nBestBlockTime)
2142+// Resend wallet transactions that haven't gotten in a block yet
2143+// Except during reindex, importing and IBD, when old wallet
2144+// transactions become unconfirmed and spams other nodes.
0// transactions become unconfirmed and spam other nodes.
utACK 3e7bea52fa76d522ce5befb2546cab6959b7aed5
SendMessages
is called for every “active” peer after a sleep of 1000ms, so if the scheduler does it for the wallets once every 1000ms, that should still be more than sufficient.
IIUC before this change ResendWalletTransactionsBefore
could be called more frequently than once per second.
But considering https://github.com/bitcoin/bitcoin/blob/656a15e5394d8d841619b5b186fa0f9c8be0322b/src/wallet/wallet.cpp#L2152-L2155 It would wait for a new block to resend wallet transactions.
So I don’t see a reason to not increase the scheduler period from 1 second to, for instance, 1 minute?
Edit: maybe the reason is to rush for the next block?
utACK 3e7bea5, I think this is a valid refactor: remove Broadcast
signal from SendMessages
and related cleanup.
2153@@ -2149,21 +2154,31 @@ void CWallet::ResendWalletTransactions(interfaces::Chain::Lock& locked_chain, in
2154 if (fFirst)
2155 return;
2156
2157+ // Get the best block time
2158+ auto locked_chain = chain().lock();
2159+ Optional<int> tip_height = locked_chain->getHeight();
2160+ int64_t best_block_time = tip_height ? locked_chain->getBlockTime(*tip_height) : 0;
I think you’re right that we should use the local time of the block received, but only in order to maintain existing behaviour.
Can you explain how using block time would be brittle?
It will hurt privacy even for tx that have a sufficient fee to get into the next block (+5 min). They will be rebroadcast regardless because a miner might have set the the block time to at least 5 minutes into the future.
Similarly, if the block time is set into the past, you might not rebroadcast transactions for quite some time, even if a block has been mined without your txs in the meantime.
Not consistently worse, but in some race-conditions, one of which is enough to give an attacker useful observations
GetTime()
for the block time)UpdateBlockTip
to interfaces::Chain::Notifications
to avoid scheduler timer, maybe follow up?
Could add UpdateBlockTip to interfaces::Chain::Notifications to avoid scheduler timer, maybe follow up?
I don’t understand this. How would this avoid using the schedule timer?
Thanks @promag . Your change is definitely a simplification, but I think it makes the already bad privacy of rebroadcasts even worse.
Currently, wallet rebroadcasts are pretty bad for privacy: If you connect to a peer and they send you the same broadcast repeatedly, then it’s their tx. Furthermore, they’ll send all of their txs together, so it’s even easier to identify which txs are rebroadcast.
The fact that we randomize when we rebroadcast transactions makes this slightly less bad. Your suggested change would be to only rebroadcast your wallet txs immediately after receiving a block. That would pretty obviously identify which txs are yours.
Ways we could improve rebroadcast privacy:
I think I don’t change the behavior?
Your change would mean that wallet txs are only ever rebroadcast immediately after a block is received. In the existing code, and in this PR, wallet txs are rebroadcast at random times.
@jnewbery I think you are missing this check:
0 // Only do it if there's been a new block since last time
1 if (best_block_time < nLastResend)
2 return;
3 nLastResend = GetTime();
so in the existing behavior (your change too) txs are broadcast only once shortly after a new block.
nNextResend
will be pushed into the future if there was not block
best_block_time
is the same from the last ResendWalletTransactions
then nothing happens.
2137@@ -2138,8 +2138,13 @@ std::vector<uint256> CWallet::ResendWalletTransactionsBefore(interfaces::Chain::
2138 return result;
2139 }
2140
2141-void CWallet::ResendWalletTransactions(interfaces::Chain::Lock& locked_chain, int64_t nBestBlockTime)
2142+// Resend unconfirmed wallet transactions
2143+void CWallet::ResendWalletTransactions()
Greg’s comments about this in IRC were very non-obvious to me. It would be good if the information could be added as a code comment:
http://www.erisian.com.au/bitcoin-core-dev/log-2019-03-27.html#l-219
<gmaxwell> jnewbery: the idea behind the current rebroadcast behavior is that unless the tip has changed we don’t have any reason to expect our transaction to have been mined, so the rebroadcast is pointless, since the txn might just be sitting waiting to be mined and all the rebroadcasts in the world won’t speed that up. <gmaxwell> (I mean the idea behind why it watches the tip) <gmaxwell> jnewbery: But thats still too rebroadcast heavy. If the tip changed but at our transaction wasn’t near the top of our mempool, well it shouldn’t have been mined then either. <gmaxwell> jnewbery: so right now if you make a txn that won’t get mined for 12 blocks, you’ll end up potentially rebroadcasting 12 times, even though it was in miners mempools the whole time. I’ve also seen network spies that appear to be attempting to exploit the rebroadcasting. <gmaxwell> So ideally we’d suppress rebroadcasting during periods when the txn is nowhere near the top. Rebroadcasting then is pointless and only serves to deanon us.
UpdateBlockTip
. As @MarcoFalke pointed out (https://github.com/bitcoin/bitcoin/pull/15632#discussion_r269639165), we should use the local time of the tip updated to maintain existing behaviour. To do that, the wallet should receive an UpdateBlockTip notification and store the local time (currently nTimeBestReceived
is stored in validation, and should be removed by this PR).
UpdateBlockTip
(https://github.com/bitcoin/bitcoin/pull/15632#issuecomment-477353148) and add comments (https://github.com/bitcoin/bitcoin/pull/15632#discussion_r269754733)
New branch pushed. Changes:
nNextResend
was not being set.Should be ready for review now.
nit in commit 94344f23ac2f914866f3f440e56dd4840344ff17:
Should also remove nTimeBestReceived
in here, since the other changes make it unused? That way it is easier to see that they are the same value, just passed around differently.
Same goes for the next commit 82b2f8c5cf8bce0202c8911f6a57b97b87dc35a1:
The hunks that are removed in later commits should maybe already be removed in this commit. That way it is clearer that the code logic was moved and the only difference is the scheduling.
656@@ -657,6 +657,8 @@ class CWallet final : public CCryptoKeyStore, private interfaces::Chain::Notific
657 int64_t nNextResend = 0;
658 int64_t nLastResend = 0;
659 bool fBroadcastTransactions = false;
660+ // Local time that the tip block was received. Used to schedule wallet rebroadcasts.
661+ int64_t m_best_block_time {0};
38@@ -39,6 +39,10 @@ def run_test(self):
39 self.log.info("Create a new transaction and wait until it's broadcast")
40 txid = int(node.sendtoaddress(node.getnewaddress(), 1), 16)
41
42+ # Sleep for just over a second so the rebroadcast scheduler can get
wait_until()
below can return in less than one second. If that happens then the rebroadcast timer won’t have popped for the first time, and so nNextResend
won’t have been set. We then setmocktime, and when the rebroadcast timer does pop, nNextResend
is still zero and ResendWalletTransactions()
exits early.
100@@ -101,7 +101,7 @@ void RegisterValidationInterface(CValidationInterface* pwalletIn) {
101 conns.BlockDisconnected = g_signals.m_internals->BlockDisconnected.connect(std::bind(&CValidationInterface::BlockDisconnected, pwalletIn, std::placeholders::_1));
102 conns.TransactionRemovedFromMempool = g_signals.m_internals->TransactionRemovedFromMempool.connect(std::bind(&CValidationInterface::TransactionRemovedFromMempool, pwalletIn, std::placeholders::_1));
103 conns.ChainStateFlushed = g_signals.m_internals->ChainStateFlushed.connect(std::bind(&CValidationInterface::ChainStateFlushed, pwalletIn, std::placeholders::_1));
104- conns.Broadcast = g_signals.m_internals->Broadcast.connect(std::bind(&CValidationInterface::ResendWalletTransactions, pwalletIn, std::placeholders::_1, std::placeholders::_2));
105+ conns.Broadcast = g_signals.m_internals->Broadcast.connect(std::bind(&CValidationInterface::ResendWalletTransactions, pwalletIn, std::placeholders::_1));
In commit b53a127ec02fa42e87cb4151e0529fbf3226187a:
Would have to adjust boost::signals2::signal<void (int64_t nBestBlockTime, CConnman* connman)> Broadcast;
a few lines up as well
2142 nLastResend = GetTime();
2143
2144 int relayed_tx_count = 0;
2145
2146+ auto locked_chain = chain().lock();
2147 { // cs_wallet scope
style-nit in f7790b3ad4c0cd5318841e30224c0d3dea0a3dca:
any reason to scope cs_wallet, but not cs_main?
38@@ -39,6 +39,10 @@ def run_test(self):
39 self.log.info("Create a new transaction and wait until it's broadcast")
40 txid = int(node.sendtoaddress(node.getnewaddress(), 1), 16)
41
42+ # Sleep for just over a second so the rebroadcast scheduler can get
43+ # warmed up (see fNextResend in ResendWalletTransactions())
style-nit in commit 9ad047d936c5601ca3841303595368e3c23e50b9:
0 # warmed up (see nNextResend in ResendWalletTransactions(), which is called first scheduled 1sec after startup)
38@@ -39,6 +39,11 @@ def run_test(self):
39 self.log.info("Create a new transaction and wait until it's broadcast")
40 txid = int(node.sendtoaddress(node.getnewaddress(), 1), 16)
41
42+ # The rebroadcast scheduler can is first called 1 sec after startupp
0 # The rebroadcast is first scheduled 1 sec after startup
38@@ -39,6 +39,11 @@ def run_test(self):
39 self.log.info("Create a new transaction and wait until it's broadcast")
40 txid = int(node.sendtoaddress(node.getnewaddress(), 1), 16)
41
42+ # The rebroadcast scheduler can is first called 1 sec after startupp
43+ # (see fNextResend in ResendWalletTransactions()). Sleep for just over
0 # (see nNextResend in ResendWalletTransactions()). Sleep for just over
some typos
Sorry. Now fixed.
38@@ -39,9 +39,9 @@ def run_test(self):
39 self.log.info("Create a new transaction and wait until it's broadcast")
40 txid = int(node.sendtoaddress(node.getnewaddress(), 1), 16)
41
42- # The rebroadcast scheduler can is first called 1 sec after startupp
43- # (see fNextResend in ResendWalletTransactions()). Sleep for just over
44- # a second to be certain that it's been called.
45+ # Wallet rebroadcast is first scheduled 1 sec after startup (see
46+ # nNextResend in ResendWalletTransactions()). Sleep for just over a
47+ # second to be certain that it has been called.
re: #15632 (review)
Maybe add “to be certain that it has been called before the first setmocktime call below”.
Normally when you see a sleep in a test, it’s inserted right before the thing that needs to be delayed, but in this case, the thing that needs to be delayed is further down, so it would be helpful to point out.
38@@ -39,9 +39,9 @@ def run_test(self):
39 self.log.info("Create a new transaction and wait until it's broadcast")
40 txid = int(node.sendtoaddress(node.getnewaddress(), 1), 16)
41
42- # The rebroadcast scheduler can is first called 1 sec after startupp
43- # (see fNextResend in ResendWalletTransactions()). Sleep for just over
44- # a second to be certain that it's been called.
45+ # Wallet rebroadcast is first scheduled 1 sec after startup (see
In commit “[wallet] Remove unnecessary Chain::Lock parameter from ResendWalletTransactions” (4aa062760e5b6d91e3528833eff81d6a6cf96ff5)
Comment change should go in the previous commit which adds the comment and the sleep: “[wallet] Schedule tx rebroadcasts in wallet.” (b1d1c414889aa2573806fe2d925651be66335bc4), since this isn’t related to removing chain_lock.
re-utACK 28c75c0b5e
Only change is in a comment
201@@ -202,16 +202,20 @@ class NotificationsHandlerImpl : public Handler, CValidationInterface
202 {
203 m_notifications->BlockDisconnected(*block);
204 }
205+ void UpdatedBlockTip(const CBlockIndex *pindexNew, const CBlockIndex *pindexFork, bool fInitialDownload) override
*
, snake case, and unused symbols.
1228@@ -1229,6 +1229,8 @@ class CWallet final : public CCryptoKeyStore, private interfaces::Chain::Notific
1229 friend struct WalletTestingSetup;
1230 };
1231
1232+void MaybeResendWalletTxs();
2151@@ -2151,7 +2152,7 @@ void CWallet::ResendWalletTransactions(interfaces::Chain::Lock& locked_chain)
2152 // only rebroadcast unconfirmed txes older than 5 minutes before the
2153 // last block was found
2154 if (wtx.nTimeReceived > m_best_block_time - 5 * 60) continue;
2155- relayed_tx_count += wtx.RelayWalletTransaction(locked_chain) ? 1 : 0;
2156+ relayed_tx_count += wtx.RelayWalletTransaction(*locked_chain) ? 1 : 0;
if (wtx.RelayWalletTransaction(*locked_chain)) relayed_tx_count++;
🎉
2151@@ -2151,7 +2152,7 @@ void CWallet::ResendWalletTransactions(interfaces::Chain::Lock& locked_chain)
2152 // only rebroadcast unconfirmed txes older than 5 minutes before the
2153 // last block was found
2154 if (wtx.nTimeReceived > m_best_block_time - 5 * 60) continue;
2155- relayed_tx_count += wtx.RelayWalletTransaction(locked_chain) ? 1 : 0;
2156+ relayed_tx_count += wtx.RelayWalletTransaction(*locked_chain) ? 1 : 0;
2157 }
2158 } // cs_wallet
;
.
;
removal.
Move nTimeBestReceived (which is only used for wallet
rebroadcasts) into the wallet.
Removes the now-unused Broadcast/ResendWalletTransactions interface from
validationinterface.
The wallet_resendwallettransactions.py needs a sleep added at the start
to make sure that the rebroadcast scheduler is warmed up before the next
block is mined.
Apologies for forcing another rebase
Don’t apologise. This is what we all signed up for! Thanks for the review.
Rebased on master.
jnewbery
DrahtBot
gmaxwell
ryanofsky
jamesob
MarcoFalke
promag
meshcollider
Labels
Wallet
Validation
Milestone
0.19.0