Could the wallet count unconfirmed non-mempool change?

morcos commented at 5:53 pm on December 13, 2017: member

Just writing down some thoughts on this.

I find it quite counter-intuitive that if you have a 1 BTC output and make a transaction spending 0.01 BTC and sending 0.99 BTC back to yourself in change, that unless that transaction is in your mempool, your balance drops from 1 to 0. Unconfirmed change in the mempool appears in available balance (assuming we can spend 0-conf change), but if it is not in the mempool it is not reflected at all, but the output spent in the transaction is still spent.

This can occur because walletbroadcast=0 and you’re delaying broadcast or using another broadcast method or because the initial transaction was evicted. In either case though it doesn’t make sense to count the entire input as spent but not credit the change output.

I believe it would make sense to include the unconfirmed non-mempool change in the pending balance.

Unfortunately this is non-trivial. This can best be seen by the example of two transactions in the wallet that spend the same input. Naively you’d double count the change, which is clearly wrong. It’s non-obvious which change you should count nor how you would go about implementing that if you did have a plan. One idea would be if you had a mempool package acceptance test (ala @sdaftuar) you could try to add all non-mempooled wallet transactions incrementally in time order, and take the resulting putative mempool state as what would count. But its starting to get a bit cumbersome. Another idea would be to check the unconfirmed non-mempool change for potential conflicts and only count the smallest change.

There are other aspects of trickiness such as mixed debit transactions.

ryanofsky commented at 6:33 pm on December 13, 2017: contributor

This can best be seen by the example of two transactions in the wallet that spend the same input.

It seems like one way to handle this would be to store more transaction metadata. We already store “replaces_txid” and “replaced_by_txid” values for bumpfee transactions. If we started assigning these values in general when adding new transactions that spend the same inputs as existing transactions, we could avoid double counting change outputs by not including change from transactions with a “replaced_by” property in the unconfirmed balance.

fanquake added the label Brainstorming on Dec 13, 2017

fanquake added the label Wallet on Dec 13, 2017

ryanofsky commented at 8:16 pm on January 9, 2018: contributor

#11020 obliquely addresses this issue by adding include_unavailable options to getbalance and getunconfirmedbalance RPCs.

maflcko referenced this in commit 064c729a96 on Nov 25, 2021

sidhujag referenced this in commit 95074ec543 on Nov 25, 2021

glozow commented at 4:10 pm on March 8, 2022: member

Now that we have package testmempoolaccept (since #20833), would it be sufficient to test accept all the non-mempool wallet transactions to get this information when we need it?

AdamISZ commented at 6:15 am on May 14, 2023: none

Thanks to @harding for pinging us on this, but: in Joinmarket, we have had a number of user reports (via our Telegram support chat and a couple other places) for the last several weeks (while mempool sizes were trending higher ofc) of missing balance. It seems that it is related to what @morcos has described above here.

Context: Joinmarket uses an unusual, nowadays (and probably bad!) model for tracking its wallet: starting from a normal BIP84 seed, we are using address imports into a watch-only Core wallet and then, to track address usage we use listaddressgroupings, then we use listunspent on startup to get an up-to-date picture of currently unspent funds, and then during running we are using listtransactions to make updates to our JM wallet’s “utxo set”. Here is a description of some of that.

So you can see we are keeping track entirely via the wallet RPC calls to Core.

In practice in something like Joinmarket you have users on the maker- side that are obviously concerned to see the new funds disappear without the old funds reappearing (because the taker used a low fee). In many cases they are exporting their wallet to another client (e.g. Sparrow) that can “find” the funds, I’m not sure if it’s because that client accesses external Electrum servers (which might have bigger mempools ofc!), or something else is going on there.

Alternatively some of them just reconfigured their mempool to be bigger, and reported they could “rediscover” the funds that way within Joinmarket itself.

Apologies that the title of the issue refers to “address reuse” as well as missing balances, so I’ve linked you to the first part of the thread that focuses only on the missing balance part.

I think I understand (a) why the problem arises and (b) why it’s a problematic thing to solve (the TLDR seems to be that it’s tricky to know properly the actual functional state of a utxo which is not currently in mempool. Is the only natural stopgap (prior to things like testmempoolaccept described above), to use abandontransaction? Or is some strategy of periodic rebroadcast by remembering txs in our application itself, a good idea? ( i suppose that line of thinking resembles the testmempoolaccept idea).

willcl-ark commented at 2:51 pm on April 10, 2024: member

Related to private broadcast @vasild ?

vasild commented at 8:33 am on May 7, 2024: contributor

Related to private broadcast @vasild ?

Right. Mentioned at the bottom of OP in #29415:

Next step after this PR is done will be to have the wallet do the private broadcast as well, #11887 would have to be resolved.

Could the wallet count unconfirmed non-mempool change? #11887