After max chain of unconfirmed change transactions, last tx is missing from memory until rescan

pinheadmz commented at 8:28 pm on March 15, 2017: member

Summary:

https://github.com/bitcoin/bitcoin/blob/57b34599b2deb179ff1bd97ffeab91ec9f904d85/doc/release-notes/release-notes-0.12.0.md

“Bitcoin Core 0.12 also introduces new default policy limits on the length and size of unconfirmed transaction chains that are allowed in the mempool (generally limiting the length of unconfirmed chains to 25 transactions, with a total size of 101 KB).”

Generate a chain of transactions with unconfirmed change outputs until this maximum is hit.
The last tx in this chain does is not accessible from gettransaction, does not appear in listunspent and will not be mined into a block until the node is restarted with -rescan (luckily 0.14.0 saves mempool to disk so the tx is “found” on rescan)

Demonstration:

Start node, generate 101 blocks so only a single 50 BTC output is mature enough to spend:

 0$ ./bitcoind -daemon -regtest
 1Bitcoin server starting
 2$ ./bitcoin-cli -regtest getinfo
 3{
 4  "version": 140000,
 5  "protocolversion": 70015,
 6  "walletversion": 130000,
 7  "balance": 0.00000000,
 8  ...
 9}
10$ ./bitcoin-cli -regtest generate 101
11[
12  "271d2c725a2279976bee7650bf20f0a16ccbbc626e07150fcb5e8e8429909b09", 
13  ... 
14  "13e07e2376aa2ce1d161497e05156632262571a17bb6e1cbd333392e23b973cc"
15]
16$ ./bitcoin-cli -regtest getinfo
17{
18  "version": 140000,
19  "protocolversion": 70015,
20  "walletversion": 130000,
21  "balance": 50.00000000,
22  ...
23}

Now attempt to create 30 x 0.5 BTC transactions, forcing bitcoind to spend unconfirmed change outputs in a chain until the limit is hit:

 0$ for n in {1..30}; do ./bitcoin-cli -regtest sendtoaddress 2NCY1Q1Mu6V5uha7fDzGNYYPUs5F1zSdXCL 0.5; done
 1fb15572accafe0b6d2b7b2806f31bda16280c7d97a91d99ca929d4236e5099c1
 2c7cea6a964ab01f20bfc54e647fc2ea9b86f1a73f64ae547dcb5fb77a9071180
 31cca3556fa953a0826d34d5096353a6809df1479996caff9e47954b1f6f81dce
 42ae2141d550c8d290fdc0f0313684eac0c5813836b051ab275cc3663059ce3ef
 5b660559b13939d4d77e2ee01907841008b09b473ed6c215388f319692312ba4b
 6e79a92d89b027c064c0c24df438c795b3595627ee823e532ab7c18ea87f98434
 77383970de305e47f781de8451303ebe84469e31f75308f72672dd52dcdd3886b
 82355d572f3cfd28f6ecae42c6ec04a739283087091512aae1360880a4b4056fe
 9873cb3977698284a118ef29be6ed307badab9da635049ab5a6b0c9a253bf35a3
10bf66983fe782efbc6ba9ce36c5e6bfba691f8721e35a16bfcb4b35505903ebf5
11f80c7d4863bb9f14175f8915ced763176b7f6d4f18680793a5929ed2671d8e5b
1221fa4a20a41be5f64156ec9df037077611269fabd7de0703fa11c660014c0aa9
136aad90127ab5e3fe67fa8b9b1a13eb03892a0f0b73798ecd7cf8af53f3a59e2c
14e68af2ecb142efb3b5885b5fc885905c3e08a3ebd5cbeff9ace73893aeb9451f
1518980ff65d020f6e333d7562bb0cf77aa2a8dca7d3f3f53f5b3f5986cc5b43f5
168de3f866aa42c71836927d3db6e764fadb462c61c82f398a22c035104a7d5dee
17a9d139b2763c5ae4000cdfe78521d41c8c4647b50de3c86e2e2d0e7daa8cc97f
180ee976f345a5805e1a260a2f2d570edf3354cf0c434bdcf42ce16752ae6cc96f
196b5327eafcd979c3f63ecd8de1139b73651b21525157f2c681b81dfdd43acf9b
20acd5a269a60dc43622c30a0ecaa8b1e393f074efb61f00e225cac0c028e7809f
2152204cba5012dceb2f0bdc0b1dbec771d728cf1345da5545704c0ef004523d46
22824b8ab2bef5a742f0ca3fad8b58f7a226daf3c5565b0b364fdfe7013d98e8f5
23fbd88cdff1a7b648b5103699321c8d035698c867a39aa1a7b0c49c1a087ab879
244fa4e846deb03bbc7a7e07364c0087d6f43cd339792a220892c53e50eb60a463
25c3d33580c52cd4eee88a0d76b5df2c77653f92537a10c3d04d27c063b543d2d9
263a8b99ed748f3418a8bf9ef9ea3b167df2bfabb28d660c0027405769e1d67174
27error code: -6
28error message:
29Insufficient funds
30error code: -6
31error message:
32Insufficient funds
33error code: -6
34error message:
35Insufficient funds
36error code: -6
37error message:
38Insufficient funds

Ok looks like 26 transactions were generated. The first one spends from a coinbase and the next 25 spend unconfirmed change outputs until bitcoind errors out with “Insufficient funds”. That’s ok, I should still have some change left in my wallet balance right?

0$ ./bitcoin-cli -regtest listunspent 0
1[
2]
3$ ./bitcoin-cli -regtest getbalance
40.00000000
5$ ./bitcoin-cli -regtest getbalance "*" 0
636.99884200

Unexpected behavior 1: Unconfirmed change is not reflected in unspent output list, but unconfirmed balance seems correct. Let’s mine a block and see what happens to my transaction chain:

 0$ ./bitcoin-cli -regtest generate 1
 1[
 2  "5d2e7dd5bb7366fecfc36acfc6ca33346114f05d040ebb020b84465db2401e9f"
 3]
 4$ ./bitcoin-cli -regtest getblock 5d2e7dd5bb7366fecfc36acfc6ca33346114f05d040ebb020b84465db2401e9f
 5{
 6  "hash": "5d2e7dd5bb7366fecfc36acfc6ca33346114f05d040ebb020b84465db2401e9f",
 7  "confirmations": 1,
 8  "strippedsize": 5783,
 9  "size": 5783,
10  "weight": 23132,
11  "height": 102,
12  "version": 536870912,
13  "versionHex": "20000000",
14  "merkleroot": "f7692c07ffc26637b3f1df3dcbf9077f0a6292f8e445289fc7ce817ce4c3ae82",
15  "tx": [
16    "d2d05cc73f95abad38c8fa59af38629f04da8dc40d638ef46852aeb0604a20ed", 
17    "fb15572accafe0b6d2b7b2806f31bda16280c7d97a91d99ca929d4236e5099c1", 
18    "c7cea6a964ab01f20bfc54e647fc2ea9b86f1a73f64ae547dcb5fb77a9071180", 
19    "1cca3556fa953a0826d34d5096353a6809df1479996caff9e47954b1f6f81dce", 
20    "2ae2141d550c8d290fdc0f0313684eac0c5813836b051ab275cc3663059ce3ef", 
21    "b660559b13939d4d77e2ee01907841008b09b473ed6c215388f319692312ba4b", 
22    "e79a92d89b027c064c0c24df438c795b3595627ee823e532ab7c18ea87f98434", 
23    "7383970de305e47f781de8451303ebe84469e31f75308f72672dd52dcdd3886b", 
24    "2355d572f3cfd28f6ecae42c6ec04a739283087091512aae1360880a4b4056fe", 
25    "873cb3977698284a118ef29be6ed307badab9da635049ab5a6b0c9a253bf35a3", 
26    "bf66983fe782efbc6ba9ce36c5e6bfba691f8721e35a16bfcb4b35505903ebf5", 
27    "f80c7d4863bb9f14175f8915ced763176b7f6d4f18680793a5929ed2671d8e5b", 
28    "21fa4a20a41be5f64156ec9df037077611269fabd7de0703fa11c660014c0aa9", 
29    "6aad90127ab5e3fe67fa8b9b1a13eb03892a0f0b73798ecd7cf8af53f3a59e2c", 
30    "e68af2ecb142efb3b5885b5fc885905c3e08a3ebd5cbeff9ace73893aeb9451f", 
31    "18980ff65d020f6e333d7562bb0cf77aa2a8dca7d3f3f53f5b3f5986cc5b43f5", 
32    "8de3f866aa42c71836927d3db6e764fadb462c61c82f398a22c035104a7d5dee", 
33    "a9d139b2763c5ae4000cdfe78521d41c8c4647b50de3c86e2e2d0e7daa8cc97f", 
34    "0ee976f345a5805e1a260a2f2d570edf3354cf0c434bdcf42ce16752ae6cc96f", 
35    "6b5327eafcd979c3f63ecd8de1139b73651b21525157f2c681b81dfdd43acf9b", 
36    "acd5a269a60dc43622c30a0ecaa8b1e393f074efb61f00e225cac0c028e7809f", 
37    "52204cba5012dceb2f0bdc0b1dbec771d728cf1345da5545704c0ef004523d46", 
38    "824b8ab2bef5a742f0ca3fad8b58f7a226daf3c5565b0b364fdfe7013d98e8f5", 
39    "fbd88cdff1a7b648b5103699321c8d035698c867a39aa1a7b0c49c1a087ab879", 
40    "4fa4e846deb03bbc7a7e07364c0087d6f43cd339792a220892c53e50eb60a463", 
41    "c3d33580c52cd4eee88a0d76b5df2c77653f92537a10c3d04d27c063b543d2d9"
42  ],
43  "time": 1489607281,
44  "mediantime": 1489607020,
45  "nonce": 0,
46  "bits": "207fffff",
47  "difficulty": 4.656542373906925e-10,
48  "chainwork": "00000000000000000000000000000000000000000000000000000000000000ce",
49  "previousblockhash": "13e07e2376aa2ce1d161497e05156632262571a17bb6e1cbd333392e23b973cc"
50}

Unexpected behavior 2: There is a transaction missing! The last one I generated in my unspent-change chain (3a8b99ed748f3418a8bf9ef9ea3b167df2bfabb28d660c0027405769e1d67174) did not make it into the block. Hm, ok maybe that is correct miner policy, let’s see if it gets into the next block:

 0$ ./bitcoin-cli -regtest generate 1
 1[
 2  "2b842bed9f8e8d17b3026b8772416c6f015cfe434c0dc0ea24d8e4bfab6bc0fc"
 3]
 4$ ./bitcoin-cli -regtest getblock 2b842bed9f8e8d17b3026b8772416c6f015cfe434c0dc0ea24d8e4bfab6bc0fc
 5{
 6  "hash": "2b842bed9f8e8d17b3026b8772416c6f015cfe434c0dc0ea24d8e4bfab6bc0fc",
 7  "confirmations": 1,
 8  "strippedsize": 227,
 9  "size": 227,
10  "weight": 908,
11  "height": 103,
12  "version": 536870912,
13  "versionHex": "20000000",
14  "merkleroot": "6efa911037aae4e7b809374ab0f812894e9db446b5fd02da3606fc9f80bacf31",
15  "tx": [
16    "6efa911037aae4e7b809374ab0f812894e9db446b5fd02da3606fc9f80bacf31"
17  ],
18  "time": 1489607296,
19  "mediantime": 1489607021,
20  "nonce": 1,
21  "bits": "207fffff",
22  "difficulty": 4.656542373906925e-10,
23  "chainwork": "00000000000000000000000000000000000000000000000000000000000000d0",
24  "previousblockhash": "5d2e7dd5bb7366fecfc36acfc6ca33346114f05d040ebb020b84465db2401e9f"
25}

Nope, that tx is still not getting mined. Any luck with my balance or unspent output list?

 0$ ./bitcoin-cli -regtest listunspent 0
 1[
 2  {
 3    "txid": "e0aafa4fa10960b32a506d42777c48be646caa88aac223a5d3f4bee353a7187e",
 4    "vout": 0,
 5    "address": "myuKG99nWAVXUdePtE5V66jA4DQRkshUdY",
 6    "scriptPubKey": "2103bc5af0d827a5f6b9ae4911f343ef864f4040f078dad86059dce2a7a12e376b5eac",
 7    "amount": 50.00000000,
 8    "confirmations": 101,
 9    "spendable": true,
10    "solvable": true
11  }, 
12  {
13    "txid": "69274f8265b50907bb486a28a782562a41f2f1fc71e531e4806b9d16e3cc9ac5",
14    "vout": 0,
15    "address": "myuKG99nWAVXUdePtE5V66jA4DQRkshUdY",
16    "scriptPubKey": "2103bc5af0d827a5f6b9ae4911f343ef864f4040f078dad86059dce2a7a12e376b5eac",
17    "amount": 50.00000000,
18    "confirmations": 102,
19    "spendable": true,
20    "solvable": true
21  }
22]
23$ ./bitcoin-cli -regtest getbalance
24100.00000000
25$ ./bitcoin-cli -regtest getbalance "*" 0
26136.99884200

No! I am still missing coins! Where is the leftover change from my transaction chain? Ok, let’s restart the node, rescan, and then check my balance:

 0$ ./bitcoin-cli -regtest stop
 1Bitcoin server stopping
 2$ ./bitcoind -daemon -regtest -rescan
 3Bitcoin server starting
 4$ ./bitcoin-cli -regtest listunspent 0
 5[
 6  {
 7    "txid": "3a8b99ed748f3418a8bf9ef9ea3b167df2bfabb28d660c0027405769e1d67174",
 8    "vout": 1,
 9    "address": "mpNrtzsAU2eEeKk9NMjwMmKvvrkTBFwL8Q",
10    "scriptPubKey": "76a91461314c7a9c7e4481797b9be3a2754313a45314f888ac",
11    "amount": 36.99884200,
12    "confirmations": 0,
13    "spendable": true,
14    "solvable": true
15  }, 
16  {
17    "txid": "e0aafa4fa10960b32a506d42777c48be646caa88aac223a5d3f4bee353a7187e",
18    "vout": 0,
19    "address": "myuKG99nWAVXUdePtE5V66jA4DQRkshUdY",
20    "scriptPubKey": "2103bc5af0d827a5f6b9ae4911f343ef864f4040f078dad86059dce2a7a12e376b5eac",
21    "amount": 50.00000000,
22    "confirmations": 101,
23    "spendable": true,
24    "solvable": true
25  }, 
26  {
27    "txid": "69274f8265b50907bb486a28a782562a41f2f1fc71e531e4806b9d16e3cc9ac5",
28    "vout": 0,
29    "address": "myuKG99nWAVXUdePtE5V66jA4DQRkshUdY",
30    "scriptPubKey": "2103bc5af0d827a5f6b9ae4911f343ef864f4040f078dad86059dce2a7a12e376b5eac",
31    "amount": 50.00000000,
32    "confirmations": 102,
33    "spendable": true,
34    "solvable": true
35  }
36]

There it is! The missing tx and my chain’s final change output. Now can we mine it?

 0$ ./bitcoin-cli -regtest generate 1
 1[
 2  "01b49edd77e4c1a6eb19860013c8b93ea79cc4cb7c2f8f4739dc8f2478db41ba"
 3]
 4$ ./bitcoin-cli -regtest getblock 01b49edd77e4c1a6eb19860013c8b93ea79cc4cb7c2f8f4739dc8f2478db41ba
 5{
 6  "hash": "01b49edd77e4c1a6eb19860013c8b93ea79cc4cb7c2f8f4739dc8f2478db41ba",
 7  "confirmations": 1,
 8  "strippedsize": 450,
 9  "size": 450,
10  "weight": 1800,
11  "height": 104,
12  "version": 536870912,
13  "versionHex": "20000000",
14  "merkleroot": "810e3b7be253da6114b8d6e3b644f7dcfc467e0809bc3387d23a672234901b92",
15  "tx": [
16    "502876265ef9c5238331c1635d2c949de9d64f2bd880ede4660dd756cb2e6f71", 
17    "3a8b99ed748f3418a8bf9ef9ea3b167df2bfabb28d660c0027405769e1d67174"
18  ],
19  "time": 1489607415,
20  "mediantime": 1489607021,
21  "nonce": 0,
22  "bits": "207fffff",
23  "difficulty": 4.656542373906925e-10,
24  "chainwork": "00000000000000000000000000000000000000000000000000000000000000d2",
25  "previousblockhash": "2b842bed9f8e8d17b3026b8772416c6f015cfe434c0dc0ea24d8e4bfab6bc0fc"
26}

Yes we can!

pinheadmz renamed this:
~~After max chain of unspent change transactions, last tx is missing from memory until rescan~~
After max chain of unconfirmed change transactions, last tx is missing from memory until rescan
on Mar 15, 2017

jnewbery commented at 9:56 pm on March 15, 2017: member

A few comments here:

your node is probably running with the default limitancestorcount / limitdescendant count, which prevents chains of >25 transactions entering your mempool. You can override those with command line arguments.
when you send the 26th transaction, it gets added your wallet, but is rejected by your mempool. You should see something like this in debug.log:

02017-03-15 21:41:07 AddToWallet 4c1473ae23f8c087116beb04ccb8cf1ee01226bb5a77db4cba1c5a44c8b52de6  new
12017-03-15 21:41:07 CommitTransaction(): Transaction cannot be broadcast immediately, too-long-mempool-chain

after you generate the next block, the transaction does not get resubmitted to your mempool automatically. You can find the transaction in your wallet using gettransaction, but getrawtransaction won’t show it (because it’s not in the mempool):

 0[ubuntu] /home/ubuntu/bitcoin
 1→ bitcoin-cli getrawtransaction 4c1473ae23f8c087116beb04ccb8cf1ee01226bb5a77db4cba1c5a44c8b52de6
 2error code: -5
 3error message:
 4No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
 5[ubuntu] /home/ubuntu/bitcoin
 6→ bitcoin-cli gettransaction 4c1473ae23f8c087116beb04ccb8cf1ee01226bb5a77db4cba1c5a44c8b52de6
 7{
 8  "amount": -0.50000000,
 9 ...
10 "hex": "02000000015415b85a301a420bac34ec12d3786e1b9d8608c29c9a083306da5e0a0a065286010000006b483045022100ca16a86854d1c51c975bf3e55386bb9ca9929c252524eb1118cfb27121e4d97902206d6acedf4596b9edaea93f411474d39746cc3d5348c9de9f43d9c9c937310de5012103880da4e6b6451859b1eed794b76    7a2e5f2a8e7898aa4cc7a83f2aad8165faa13feffffff02a88e0492000000001976a9149e2f8afc219f0db384ce131d7c028753ac350a4a88ac80f0fa020000000017a914d396f368f3b4348ced0ed1f4138bb2906e12f7df8765000000"
11}

at this point, you could resubmit the transaction to your mempool by running sendrawtransaction <tx hex serialization>
persistent mempool has nothing to do with the transaction being readded after stop-start (since the transaction never made it into the mempool in the first place). The transaction is actually in your wallet file, so when you stop-start, that’s where it’s being found. You can prove this to yourself by deleting mempool.dat between stopping and starting.

I think that the correct behaviour here is that the wallet should reject the transaction if it’s in a chain that’s too long to get accepted by the node’s mempool.

pinheadmz commented at 10:23 pm on March 15, 2017: member

Ahhh thank you for the explanation! This makes sense. Yeah I suppose the expected behavior for me would be if sendtoaddress failed with a “chain too long” error before creating that last tx, leaving one last unspent output available for a retry or wait for confirmation…

sdaftuar commented at 1:59 am on March 16, 2017: member

FYI there’s some relevant context in #9262, #9302, and #9752.

laanwj added the label Mempool on Mar 16, 2017

nopara73 commented at 10:26 pm on November 17, 2017: none

your node is probably running with the default limitancestorcount / limitdescendant count, which prevents chains of >25 transactions entering your mempool @jnewbery Isn’t it 64? I don’t have that set and I just got this error: 64: too-long-mempool-chain, when I did this. (On the testnet.)

sipa commented at 10:29 pm on November 17, 2017: member

64 is the error code.

nopara73 commented at 10:52 pm on November 17, 2017: none

64 is the error code.

Thanks. That explains why the error message on my GUI had less than 64 transaction ids in it.

MarcoFalke commented at 6:40 pm on November 27, 2017: member

Linking to the relevant line in the code for reference: https://github.com/bitcoin/bitcoin/blob/a89221873a3ee2451c73b41bbe2d99d36f439d31/src/wallet/wallet.cpp#L3036-L3042

smartwombat commented at 8:14 pm on April 6, 2018: none

Is there a solution using LockUnspent ?

Generate 24 transactions with SendToAddress Using TXID of the 24th transaction GetRawTransaction txid, true (or 1 if version before 0.14.0)

examine the vout: array; in our case there should only be two, the spend and the change
find the vout: for the change the amount is != the amount spent the address is != the address the spend was to LockUnspent false, txid, vout
lock that vout: so it can not be used as the UTXO for any further spend.

Lather, rinse, repeat.

Seems to me a lot simpler than loading amounts into multiple addresses, using the deprecated Account feature to identify them, and then SetAccount every group of 24 to force the wallet to move onto a different (confirmed) UTXO so that a new chain will be built.

Will this approach work ?

YankeyTech commented at 9:57 pm on April 6, 2018: none

Stop On Apr 6, 2018 8:15 PM, “smartwombat” notifications@github.com wrote:

Is there a solution using LockUnspent ?

Generate 24 transactions with SendToAddress Using TXID of the 24th transaction GetRawTransaction txid, true (or 1 if version before 0.14.0)

examine the vout: array; in our case there should only be two, the spend and the change

find the vout: for the change the amount is != the amount spent the address is != the address the spend was to LockUnspent false, txid, vout

lock that vout: so it can not be used as the UTXO for any further spend.

Lather, rinse, repeat.

Seems to me a lot simpler than loading amounts into multiple addresses, using the deprecated Account feature to identify them, and then SetAccount every group of 24 to force the wallet to move onto a different (confirmed) UTXO so that a new chain will be built.

Will this approach work ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/10004#issuecomment-379364753, or mute the thread https://github.com/notifications/unsubscribe-auth/Ai3zhr9OqGO9n6GsIrtGPaI0xTDz86xcks5tl8zngaJpZM4Mee-O .

xcelr commented at 5:26 pm on August 4, 2018: none

CommitTransaction(): Transaction cannot be broadcast immediately, no-witness-yet anyone else getting this error

MrCryptoBillionaire commented at 4:54 pm on November 23, 2019: none

My question is how to catch this relays from my api service automatically?

MarcoFalke referenced this in commit f66c827c2d on Mar 25, 2022

MarcoFalke closed this on Mar 25, 2022

sidhujag referenced this in commit 92432625ce on Apr 2, 2022

DrahtBot locked this on Mar 25, 2023

After max chain of unconfirmed change transactions, last tx is missing from memory until rescan #10004

Summary:

Demonstration: