Fixes #27129
To avoid bad-txns-premature-spend-of-coinbase
error,
when getting a utxo (using get_utxo
) to create a new
transaction get_utxo
shouldn’t return (if possible)
by default immature coinbase.
feature_bip68_sequence
#27177
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
See the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.
No conflicts as of last run.
221@@ -222,7 +222,8 @@ def test_sequence_lock_unconfirmed_inputs(self):
222
223 # Create a mempool tx.
224 self.wallet.rescan_utxos()
225- tx1 = self.wallet.send_self_transfer(from_node=self.nodes[0])["tx"]
226+ utxo = self.wallet.get_utxos(include_immature_coinbase=False)[0]
227+ tx1 = self.wallet.send_self_transfer(from_node=self.nodes[0], utxo_to_spend=utxo)["tx"]
get_utxo
not return immature coinbases by default?
223 self._utxos = sorted(self._utxos, key=lambda k: (k['value'], -k['height'])) # Put the largest utxo last
224+ if not include_immature_coinbase:
225+ utxo_filter: Any = filter(lambda utxo: not utxo['coinbase'] or COINBASE_MATURITY <= utxo['confirmations'], self._utxos)
226 if txid:
227- utxo_filter: Any = filter(lambda utxo: txid == utxo['txid'], self._utxos)
228+ utxo_filter = filter(lambda utxo: txid == utxo['txid'], self._utxos)
Instead of overwriting and discarding the previous filter, it might be better to just set it once. This can be done by nesting the new if
under a new elif
.
Another alternative would be to always apply the filter.
220 Args:
221 txid: get the first utxo we find from a specific transaction
222 """
223 self._utxos = sorted(self._utxos, key=lambda k: (k['value'], -k['height'])) # Put the largest utxo last
224+ if not include_immature_coinbase:
225+ utxo_filter: Any = filter(lambda utxo: not utxo['coinbase'] or COINBASE_MATURITY <= utxo['confirmations'], self._utxos)
Any
here?
Any
it throw a lint error, I believe it’s because it can be any value from filter
or reversed
.
218 Returns a utxo and marks it as spent (pops it from the internal list)
219
220 Args:
221 txid: get the first utxo we find from a specific transaction
222 """
223 self._utxos = sorted(self._utxos, key=lambda k: (k['value'], -k['height'])) # Put the largest utxo last
reversed()
on L288 can just be factored in to this first sort? That might make the actual filtering logic cleaner.
txid
, it will not have to return the largest one.
concept ACK
I’m curious what tests require immature coinbase spending, I tried playing with the function to get other tests to fail and ultimately couldn’t get any
Concept ACK
If possible, would prefer to simply always apply the filter for excluding immature coinbases.
However, I realized we can’t always apply this coinbase filter because some tests need to use unconfirmed coinbase utxo (because it’s the only available ones).
I assume you meant immature coinbases, rather than unconfirmed ones. Which tests currently rely on those? Can’t really imagine a case where an unspendable UTXO would be of any use (only if mining of new blocks happen between fetching the UTXO and actually spending them, but then we can probably change those tests to fetch the UTXO later?).
I assume you meant immature coinbases, rather than unconfirmed ones. Which tests currently rely on those? Can’t really imagine a case where an unspendable UTXO would be of any use (only if mining of new blocks happen between fetching the UTXO and actually spending them, but then we can probably change those tests to fetch the UTXO later?).
One of the examples I found:
In p2p_segwit.py
:
https://github.com/bitcoin/bitcoin/blob/cb40639bdf04bab31bcdceb3bf941e9bade8317a/test/functional/p2p_segwit.py#L2009-L2010
0diff --git a/test/functional/p2p_segwit.py b/test/functional/p2p_segwit.py
1index b0900e49b..c502bac8c 100755
2--- a/test/functional/p2p_segwit.py
3+++ b/test/functional/p2p_segwit.py
4@@ -2006,6 +2006,7 @@ class SegWitTest(BitcoinTestFramework):
5 def serialize(self):
6 return serialize_with_bogus_witness(self.tx)
7
8+ self.log.info(self.wallet.get_utxos(include_immature_coinbase=True))
9 tx = self.wallet.create_self_transfer()['tx']
10 assert_raises_rpc_error(-22, "TX decode failed", self.nodes[0].decoderawtransaction, hexstring=serialize_with_bogus_witness(tx).hex(), iswitness=True)
11 with self.nodes[0].assert_debug_log(['Unknown transaction optional data']):
It will print a lot of utxos, but if you set include_immature_coinbase=False
, it will return an empty list.
Force-pushed changing the approach.
The logic:
if you don’t specify a txid
, it will return the largest utxo desconsidering immature coinbase. In case the only ones available are immature ones, it will return the largest of them.
Why? - We can’t filter the immature coinbase ones by default because in some tests they’re the only available ones.
Added a flag avoid_immature_coinbase
to not break some mempool tests, in that case, they need to fetch it considering the immature ones even having other (“normal”) utxos.
224+ coinbase_filter = filter(lambda utxo: not utxo['coinbase'] or COINBASE_MATURITY <= utxo['confirmations'], self._utxos)
225 if txid:
226 utxo_filter: Any = filter(lambda utxo: txid == utxo['txid'], self._utxos)
227+ elif len(list(coinbase_filter)) > 0 and avoid_immature_coinbase:
228+ utxo_filter = filter(lambda utxo: not utxo['coinbase'] or COINBASE_MATURITY <= utxo['confirmations'], self._utxos)
229+ utxo_filter = reversed(list(utxo_filter))
list(coinbase_filter)
exhausts the iterator so you need to start it over. What if coinbase_filter
was actually a list called mature_coins
from the beginning of the function? I dunno, just an idea.
avoid_immature_coinbase
at all which I feel should just be an implied default anyway.
Concept ACK
Regarding the approach, I think we should always apply the filter or allow the argument but have it default to True and only let you change it to false if you are calling get_utxo
directly (no argument forwarding).
I’m also a little confused by the mempool_package_limit.py
, because it looks like the test assumes it is already working with mature UTXOs:
https://github.com/bitcoin/bitcoin/blob/master/test/functional/mempool_package_limits.py#L26-L29
Using immature UTXOs in the mempool test feels like a hack, where we could just add more confirmations or figure out why the test is failing when there are supposed to already be mature UTXOs
EDIT: if you do end up needing to touch mempool_package_limits.py
with a fix, I’d also suggest adding it as its own commit
mempool_package_limits.py
.
Force-pushed changing the approach:
get_utxo
(avoid_immature_coinbase=True
), removed that in other functions.mempool_package_limits
from self.generate(self.nodes[0], COINBASE_MATURITY)
to self.generate(self.wallet, COINBASE_MATURITY)
.Seems like the fix in mempool_package_limits
was simpler than I thought.
interface_usdt_mempool.py
is now failing but the fix is very simple: just mine a block after each test: https://github.com/josibake/bitcoin/commit/85ec64b1eb3037e8bd9349242b05615b41b87261
Nice, this looks much simpler now. interface_usdt_mempool.py is now failing but the fix is very simple: just mine a block after each test: https://github.com/josibake/bitcoin/commit/85ec64b1eb3037e8bd9349242b05615b41b87261
Added a commit fixing it and added you as co-author, thank you! Force-pushed doing it!
@glozow want to take a look at the mempool tests?
LGTM, makes sense to generate 1 to clear mempool after subtests. changing who calls generate
shouldn’t impact mempool_package_limits.
I haven’t taken the time to verify that this fixes the test intermittent failure.
25@@ -26,7 +26,7 @@ def run_test(self):
26 self.wallet = MiniWallet(self.nodes[0])
27 # Add enough mature utxos to the wallet so that all txs spend confirmed coins.
28 self.generate(self.wallet, 35)
29- self.generate(self.nodes[0], COINBASE_MATURITY)
30+ self.generate(self.wallet, COINBASE_MATURITY)
feel-free-to-ignore tiny-nit: could just do it in one call
0 self.generate(self.wallet, 35 + COINBASE_MATURITY)
Two little questions about this:
COINBASE_MATURITY + 35
if those 35 blocks even needed anymoreThe MiniWallet does not update the confirmation counts of its UTXOs when new blocks are generated. When the blocks are generated with a different generator, it will still think that its generated coins are immature even if enough blocks have been mined to make them mature. When this behavior is coupled with the change to prefer matured coins, the MiniWallet ends up later having a very restricted UTXO set as it will have a few mature coins that are the result of making transaction chains. This occurs because of the fallback to use immature coins when no mature coins are available.
By generating the maturity blocks with the MiniWallet, we end up having it rescan for its UTXOs which makes it update the confirmation counts of its coins thereby actually making the mature coins set as full as expected. It is possible to achieve the same result by explicitly calling self.wallet.rescan_utxos()
after generating the blocks.
Incidentally, this indicates that there might be an intermittent failure in this test as it may randomly select some UTXOs for package building that were spent by chain building loops. This is readily apparent when the mature UTXO pool is small.
213@@ -214,16 +214,19 @@ def get_address(self):
214 assert_equal(self._mode, MiniWalletMode.ADDRESS_OP_TRUE)
215 return self._address
216
217- def get_utxo(self, *, txid: str = '', vout: Optional[int] = None, mark_as_spent=True) -> dict:
218+ def get_utxo(self, *, txid: str = '', vout: Optional[int] = None, mark_as_spent=True, avoid_immature_coinbase=True) -> dict:
False
anymore, does it make sense to remove the argument and just filter out immature coins all the time no matter what anyway?
I agree that this option seems pointless. It’s never used currently, and I don’t think it makes sense to ever use it in the tests unless specifically testing immature transactions, in which case the utxo should be known explicitly by the caller.
In addition to removing the parameter, I think it would also make sense to remove the fallback to use any UTXO when there are no mature coins.
diff example:
0index 0cde72c82..62a64b9dd 100644
1--- a/test/functional/test_framework/wallet.py
2+++ b/test/functional/test_framework/wallet.py
3@@ -218,13 +218,12 @@ class MiniWallet:
4 txid: get the first utxo we find from a specific transaction
5 """
6 self._utxos = sorted(self._utxos, key=lambda k: (k['value'], -k['height'])) # Put the largest utxo last
7- mature_coins = list(filter(lambda utxo: not utxo['coinbase'] or COINBASE_MATURITY <= utxo['confirmations'], self._utxos))
8+ block_height = self._test_node.getblockchaininfo()['blocks']
9+ mature_coins = list(filter(lambda utxo: not utxo['coinbase'] or COINBASE_MATURITY - 1 <= block_height - utxo['height'], self._utxos))
10 if txid:
11 utxo_filter: Any = filter(lambda utxo: txid == utxo['txid'], self._utxos)
12- elif len(mature_coins) > 0:
13- utxo_filter = reversed(mature_coins)
14 else:
15- utxo_filter = reversed(self._utxos) # By default the largest utxo
16+ utxo_filter = reversed(mature_coins) # By default the largest utxo
17 if vout is not None:
18 utxo_filter = filter(lambda utxo: vout == utxo['vout'], utxo_filter)
19 index = self._utxos.index(next(utxo_filter))
ACK f8ebcb823fc5f39adc87ea6cf079f3e72ee44f30
(Didn’t run interface_usdt_mempool.py locally, as I’m currently not having a build with tracepoints available, but the changes look reasonable.)
221
222 Args:
223 txid: get the first utxo we find from a specific transaction
224 """
225 self._utxos = sorted(self._utxos, key=lambda k: (k['value'], -k['height'])) # Put the largest utxo last
226+ mature_coins = list(filter(lambda utxo: not utxo['coinbase'] or COINBASE_MATURITY <= utxo['confirmations'], self._utxos))
In 894b2531991f8179022c158c2c15c06f0b774f0e “test: fix intermittent issue in feature_bip68_sequence
”
The confirmation counts for the UTXOs can be very outdated when blocks are generated by things other than the MiniWallet. It would be nice if these could be kept up to date so that this filter is accurate.
Thanks everyone for reviewing it! Here’s what have changed:
get_utxo
will not return immature coinbase utxos anymore for any case.confirmation
count from utxo object since it may have outdated. We will use current block height + utxo’s height for it.get_utxos
to use current block height to compute the confirmation count instead of using the value from utxo object, same logic from get_utxo
.190@@ -191,6 +191,7 @@ def run_test(self):
191 def test_persist_unbroadcast(self):
192 node0 = self.nodes[0]
193 self.start_node(0)
194+ self.start_node(2)
rescan_utxos
with any already-running node to update utxo confirmation count, but I guess this is probably the cleanest way
234@@ -233,7 +235,8 @@ def get_utxo(self, *, txid: str = '', vout: Optional[int] = None, mark_as_spent=
235 def get_utxos(self, *, include_immature_coinbase=False, mark_as_spent=True):
236 """Returns the list of all utxos and optionally mark them as spent"""
237 if not include_immature_coinbase:
238- utxo_filter = filter(lambda utxo: not utxo['coinbase'] or COINBASE_MATURITY <= utxo['confirmations'], self._utxos)
239+ blocks_height = self._test_node.getblockchaininfo()['blocks']
block_height
in the other function. no need to fix unless you have to retouch
ACK ee018b25af9477050839826c27a885e7211eed14
Great work on this, nice and clean now ;-)
0-----BEGIN PGP SIGNED MESSAGE-----
1Hash: SHA256
2
3ACK ee018b25af9477050839826c27a885e7211eed14
4-----BEGIN PGP SIGNATURE-----
5
6iQIzBAEBCAAdFiEE5hdzzW4BBA4vG9eM5+KYS2KJyToFAmRmTNQACgkQ5+KYS2KJ
7yTqJsBAAqV8ZWwsCyWxK8THFr3lc5gQWcgpERH4DHAXWFpMn1WGMBLn0aXs6VTMh
8V5qXjM76piqQsQcJyLEli5r4zgW0hDF7YdNtLTwPNPEnMYqvI6uxj1mUMochZEXn
9dgk9vhhQzRV4s/sixPNolV6Pt6+1hCIRDRKWY6f/8r1caGoUYp4X8zQXeUkkn1Xg
10vvvAr277CFJCWr/wXTWtgBJUUM+s06e2l34f+sXNJIX9bIQR239ClvTkOh2Gg4OA
11z7V/L1l81tzKCUs5Ze81VczjotrO9709445P/zQa9/XTXPTyr27EW2ZkPpNqALlh
12yv0xYxnogM2tM527rRA5isuhLgd+HXdrIfIm6xiZD7Uj49WgqDOzz2LHM1AYSeh8
13bWBOiYaVccVagkXEh2F0I/cb0AxBf8oVnUtVwifJtcwx/++atKIfptXOWTd7PPRo
14W8mEw7C462YyZQaErHx3a/QKTnKq208raedpjNwQWvgtmS5cLP1Ang+RpttDghUR
15EZt6+SNzoYL0jO51K9AYdJQ75mVOXBR0cu5VDVYNkqA9ZGbEDutKOMOwYZ/JOVcs
16JSSKIzIM0CyqK3JDg8yXGdsWYC2z/sjpWkxXsZjFJDcUkDGkn2bvOuOzfoaFhmaw
17KpgfBB92wZsAqTrpzhw2T5thAFjDRcsULS8ad8AofiS39vFTeAg=
18=vGaL
19-----END PGP SIGNATURE-----
pinheadmz’s public key is on keybase
Looks like the tests fail in CI, even on a re-run?
Yes, needs to fix mempool_compatibility
.
Force-pushed:
mempool_compatibility
to stop node1 after calling send_self_transfer
, not before: 0--- a/test/functional/mempool_compatibility.py
1+++ b/test/functional/mempool_compatibility.py
2@@ -47,12 +47,12 @@ class MempoolCompatibilityTest(BitcoinTestFramework):
3 # unbroadcasted_tx won't pass old_node's `MemPoolAccept::PreChecks`.
4 self.connect_nodes(0, 1)
5 self.sync_blocks()
6- self.stop_node(1)
7
8 self.log.info("Add a transaction to mempool on old node and shutdown")
9 old_tx_hash = new_wallet.send_self_transfer(from_node=old_node)["txid"]
10 assert old_tx_hash in old_node.getrawmempool()
11 self.stop_node(0)
12+ self.stop_node(1)
To avoid `bad-txns-premature-spend-of-coinbase` error,
when getting a utxo (using `get_utxo`) to create a new
transaction `get_utxo` shouldn't return by default
immature coinbase.
Co-authored-by: josibake <josibake@protonmail.com>
Use current block height to compute the confirmation count
instead of using the value from utxo object
Before I re-ACK, are you gonna squash any commits?
Done!
re-ACK 272eb5561667482f8226bcf98eea00689dccefb8
confirmed trivial rebase and modifictaion to one test to pass CI, since last review
0-----BEGIN PGP SIGNED MESSAGE-----
1Hash: SHA256
2
3ACK 272eb5561667482f8226bcf98eea00689dccefb8
4-----BEGIN PGP SIGNATURE-----
5
6iQIzBAEBCAAdFiEE5hdzzW4BBA4vG9eM5+KYS2KJyToFAmRo3i4ACgkQ5+KYS2KJ
7yTq+bw//bLFMWw6VL7OnEzumxPTLIFRHW6ol5Wsk8nIkgvwDmviLXRjU0Gh2K0G5
8fMWUq0t2lrRqWzZgobcWfTG9+HNRn2J3Y/WuR+PeUfsOk7lEKhztUN6ROo9O86EC
9j/e709RfVk2bpNLHeST864ih6B9x0lFSvbikgI3CLOa2oQVuMc2u8Yz96RNGLE15
10XSNjiO/C4svU18EtNdFN5cYI7Uj0Ysz2z+XeMo2818DZ1/yINcCU4ehFJFQk90bF
11jC8u8UzghZyXgucJ93nj18ihp/gZuLntPtyffT+dKr/duTOJSYTMDMgnE3wtV5H4
12UhZ/mc+fmb2NKvr1L9N5sFe+OZAnq762RqfkkudQI5qNYJFh7iX8OxiL4XFuq/A3
13DuNRRf+tig61iUSX7VnFDhwAkMIeYm2FTaCg6EnqduuaO5XxgQlGsRAJCmQY1d4G
14waMGThUPyvoHIFCn1cWLmwChcsbcrVjjOL3Bi4F0DHNFOiTtFcekMSjbyL7UnR8w
15hc/jygVw5h3UOaob1yn37u+iLfa6ELBZM7wc3WZIvVOuh61YiMetWEs2LR5Yn29M
16o3NxH9+aGFo09uFv0NAxP7ulLShj7YIuOaJovzGQpUXae475YGNsJpxYlDNO50RX
17MzEPQ/f71xFiyWU0mHHod+If2ds/N5z/3IpWdEx4FliZF2XS+fk=
18=Sqe2
19-----END PGP SIGNATURE-----
pinheadmz’s public key is on keybase