27.0 RC Testing Guide Feedback

marcofleon commented at 2:53 am on March 21, 2024: contributor

This issue is to discuss the 27.0 Release Candidate Testing Guide. If you have any issues with or feedback on the document, please leave a comment here.

Note: this is for feedback on the document, not on Bitcoin Core or on the 27.0 changes. Please see #29697 for instructions on how to report bugs/results.

Thank you for your feedback.

marcofleon commented at 3:06 am on March 21, 2024: contributor

Great work @tdb3 @davidgumberg @cbergqvist!

davidgumberg commented at 3:13 am on March 21, 2024: contributor

Great job @marcofleon!

willcl-ark added the label Brainstorming on Mar 21, 2024

maflcko added this to the milestone 27.0 on Mar 21, 2024

rkrux commented at 1:39 pm on March 23, 2024: none

Hey folks I’ve been testing using the guide, and am facing few issues while testing net: enable v2transport by default.

There is no signet directory.
I could not find corresponding logs for the grep command in the root debug file too.

0➜  rctesting cat $DATA_DIR_27/signet/debug.log | grep "v2\|peer=0"
1cat: /tmp/27-rc-test/signet/debug.log: No such file or directory
2➜  rctesting cat $DATA_DIR_27/debug.log | grep "v2\|peer=0"
32024-03-23T12:11:54Z Bitcoin Core version v27.0.0rc1 (release build)

The transport_protocol_type field of the newly added node doesn’t show v2 - shows detecting instead.

 0➜  rctesting bcli getpeerinfo | grep transport_protocol_type
 1    "transport_protocol_type": "v1",
 2    "transport_protocol_type": "v1",
 3    "transport_protocol_type": "v1",
 4    "transport_protocol_type": "v1",
 5    "transport_protocol_type": "v1",
 6    "transport_protocol_type": "v1",
 7    "transport_protocol_type": "v1",
 8    "transport_protocol_type": "v1",
 9    "transport_protocol_type": "v1",
10    "transport_protocol_type": "v1",
11    "transport_protocol_type": "detecting",

Also while testing v3 Transaction Policy, I’m seeing a Input not found or already spent error while signing the child transaction.

 0➜  rctesting bcli signrawtransactionwithwallet 02000000013a5f7d6c206993161998b9f380b4034e80efff83b0ebd5b982b76e3b5e5525240100000000fdffffff0200e1f5050000000016001459b511334c26ae60531dd9a8a486e4b114671b6ac022171e0100000016001498a8ca326b45666899c0b1a96b1b008880361ed100000000
 1{
 2  "hex": "02000000013a5f7d6c206993161998b9f380b4034e80efff83b0ebd5b982b76e3b5e5525240100000000fdffffff0200e1f5050000000016001459b511334c26ae60531dd9a8a486e4b114671b6ac022171e0100000016001498a8ca326b45666899c0b1a96b1b008880361ed100000000",
 3  "complete": false,
 4  "errors": [
 5    {
 6      "txid": "2425555e3b6eb782b9d5ebb083ffef804e03b480f3b99819169369206c7d5f3a",
 7      "vout": 1,
 8      "witness": [
 9      ],
10      "scriptSig": "",
11      "sequence": 4294967293,
12      "error": "Input not found or already spent"
13    }
14  ]
15}

tdb3 commented at 1:58 pm on March 23, 2024: contributor

27.0 Release Candidate Testing Guide

Thank you for looking at the guide @rkrux!

For the v2transport steps, it looks like your machine is syncing on mainnet rather than signet.

Can you verify that /tmp/27-rc-test/bitcoin.conf contains signet=1?

Could you also verify that echo $DATA_DIR_27 results in /tmp/27-rc-test. If the guide was run over multiple sessions to the machine (e.g. SSH into the machine, run a section, end SSH session, re-SSH again later), then the environment variables would not persist between sessions.

A step in the beginning of the section creates a bitcoin.conf with the following:

0{
1echo "signet=1"
2echo "dnsseed=0"
3echo "fixedseeds=0"
4echo "debug=net"
5} > $DATA_DIR_27/bitcoin.conf

Does the bitcoin.conf in your environment match? We’re open to thoughts on how to make steps clearer, so if there’s anything confusing, please don’t hesitate to point it out.

tdb3 commented at 2:07 pm on March 23, 2024: contributor

Also while testing v3 Transaction Policy, I’m seeing a Input not found or already spent error while signing the child transaction.

 0➜  rctesting bcli signrawtransactionwithwallet 02000000013a5f7d6c206993161998b9f380b4034e80efff83b0ebd5b982b76e3b5e5525240100000000fdffffff0200e1f5050000000016001459b511334c26ae60531dd9a8a486e4b114671b6ac022171e0100000016001498a8ca326b45666899c0b1a96b1b008880361ed100000000
 1{
 2  "hex": "02000000013a5f7d6c206993161998b9f380b4034e80efff83b0ebd5b982b76e3b5e5525240100000000fdffffff0200e1f5050000000016001459b511334c26ae60531dd9a8a486e4b114671b6ac022171e0100000016001498a8ca326b45666899c0b1a96b1b008880361ed100000000",
 3  "complete": false,
 4  "errors": [
 5    {
 6      "txid": "2425555e3b6eb782b9d5ebb083ffef804e03b480f3b99819169369206c7d5f3a",
 7      "vout": 1,
 8      "witness": [
 9      ],
10      "scriptSig": "",
11      "sequence": 4294967293,
12      "error": "Input not found or already spent"
13    }
14  ]
15}

For this one, could you specify the corresponding transaction in the guide, so we can help pinpoint the step? Is this the corresponding example transaction in the guide?

0bcli signrawtransactionwithwallet 0200000001bcffc6b754dcf1160d9fec14eb836483f9246db603ea194d5ee3c9dcbc1eb8b60100000000fdffffff0200e1f50500000000160014623affcf6334d7e50a9757b80f702e4129aa02d0c022171e010000001600146b458f0507bc79e85774bb2db7e820adb557e95800000000 '[{"txid":"b6b81ebcdcc9e35e4d19ea03b66d24f9836483eb14ec9f0d16f1dc54b7c6ffbc", "vout": 1, "scriptPubKey": "00146b458f0507bc79e85774bb2db7e820adb557e958", "witnessScript": "02fa562c1af027c39ee03a9892b388f854904c99fbf80b77f69aba3f48305f2aa9", "amount": 48.999}]'

If so, it seems that the bcli signrawtransactionwithwallet 02000000013a5f7d6c206993161998... is missing the content at the end which specifies txid, spk, witnessScript, etc.

rkrux commented at 9:21 am on March 25, 2024: none

If so, it seems that the bcli signrawtransactionwithwallet 02000000013a5f7d6c206993161998… is missing the content at the end which specifies txid, spk, witnessScript, etc.

Yes @tdb3, I’m sorry that I didn’t scroll to the right in the code block to see the second argument that’s needed for the parent txs that are not in the blockchain, trying this step again.

Does the bitcoin.conf in your environment match?

Yes, it did contain signet=1, I will try that one again after covering the other tests.

rkrux commented at 9:38 am on March 25, 2024: none

0➜  rctesting coinbase_txid=$(bcli listunspent | jq -r .[0].txid)
1zsh: no matches found: .[0].txid

This command needs to be replaced with the following instead. coinbase_txid=$(bcli listunspent | jq -r '.[0].txid')

Similarly for the following:

0➜  rctesting parent_spk=$(echo $decoded_parent_tx | jq -r .vout[1].scriptPubKey.hex)
1zsh: no matches found: .vout[1].scriptPubKey.hex

0parent_witness_script=$(echo $decoded_parent_tx | jq -r .vin[0].txinwitness[1])

0child_spk=$(echo $decoded_child_tx | jq -r .vout[1].scriptPubKey.hex)
1child_witness_script=$(echo $decoded_child_tx | jq -r .vin[0].txinwitness[1])

There are 5-6 other similar occurrences as well below in the guide.

rkrux commented at 9:55 am on March 25, 2024: none

This command throws this error, so I bypassed it by passing the actual hex values instead.

0➜  rctesting bcli testmempoolaccept [\"$signed_parent_tx\",\"$signed_child_tx\",\"$signed_grandchild_tx\"]
1
2zsh: no matches found: ["0200000000010141708e2954c9f0e6fda6500ec23325b96522d90e63aa7264bbda1324cf0e46cb0000000000fdffffff0200e1f50500000000160014fc7705f21f583a3fcfbb61ec203de7a999acfe04608a0e2401000000160014b0e9d9d52a4fdd87a8e0cadc2c95d9fb37a1efd70247304402201ad2eb4f4b78bd289a2fa0891343ba421ce8f7d746fefa95584e223ed5ef15020220295fee1d1a4675fa744293ed661a3db7a14b9f610ed08e194c6b28ba2ef27b98012102e6374c796c14e20c7a8315d74b42f721979a49df27ed87f30b9f4b015001bd1c00000000","0200000000010115bf3fd8c2285fe85de939fca533bd6a00970585d1292e2e3aa4a39c788faa4e0100000000fdffffff0200e1f50500000000160014fc7705f21f583a3fcfbb61ec203de7a999acfe04c022171e01000000160014b0e9d9d52a4fdd87a8e0cadc2c95d9fb37a1efd702473044022072cb174cc92a36b900f794fc320a15ed49e9240f628bfc0cde2dd02e25c50e7a0220639c964cc3bf5b2b8bcfe6e836a8c31d3d0d23db4b8fe73dbad3610d4e613efb012102e6374c796c14e20c7a8315d74b42f721979a49df27ed87f30b9f4b015001bd1c00000000","02000000000101fbd69b4d90dd95c72e2778ee1efd17a4b30613d633e0e2aa0151998b33d6e2a80100000000fdffffff0200e1f50500000000160014fc7705f21f583a3fcfbb61ec203de7a999acfe0420bb1f1801000000160014b0e9d9d52a4fdd87a8e0cadc2c95d9fb37a1efd702473044022033560fe31ed823e83d5ad82af79aee73b714410981387a605c99a8d8807cc89602201a8889bc76e77905d9dac6d4f28a53e127fa912bedc31e470dc54873081f2f20012102e6374c796c14e20c7a8315d74b42f721979a49df27ed87f30b9f4b015001bd1c00000000"]

With this command to be precise:

0➜  rctesting bcli testmempoolaccept '["0200000000010141708e2954c9f0e6fda6500ec23325b96522d90e63aa7264bbda1324cf0e46cb0000000000fdffffff0200e1f50500000000160014fc7705f21f583a3fcfbb61ec203de7a999acfe04608a0e2401000000160014b0e9d9d52a4fdd87a8e0cadc2c95d9fb37a1efd70247304402201ad2eb4f4b78bd289a2fa0891343ba421ce8f7d746fefa95584e223ed5ef15020220295fee1d1a4675fa744293ed661a3db7a14b9f610ed08e194c6b28ba2ef27b98012102e6374c796c14e20c7a8315d74b42f721979a49df27ed87f30b9f4b015001bd1c00000000","0200000000010115bf3fd8c2285fe85de939fca533bd6a00970585d1292e2e3aa4a39c788faa4e0100000000fdffffff0200e1f50500000000160014fc7705f21f583a3fcfbb61ec203de7a999acfe04c022171e01000000160014b0e9d9d52a4fdd87a8e0cadc2c95d9fb37a1efd702473044022072cb174cc92a36b900f794fc320a15ed49e9240f628bfc0cde2dd02e25c50e7a0220639c964cc3bf5b2b8bcfe6e836a8c31d3d0d23db4b8fe73dbad3610d4e613efb012102e6374c796c14e20c7a8315d74b42f721979a49df27ed87f30b9f4b015001bd1c00000000","02000000000101fbd69b4d90dd95c72e2778ee1efd17a4b30613d633e0e2aa0151998b33d6e2a80100000000fdffffff0200e1f50500000000160014fc7705f21f583a3fcfbb61ec203de7a999acfe0420bb1f1801000000160014b0e9d9d52a4fdd87a8e0cadc2c95d9fb37a1efd702473044022033560fe31ed823e83d5ad82af79aee73b714410981387a605c99a8d8807cc89602201a8889bc76e77905d9dac6d4f28a53e127fa912bedc31e470dc54873081f2f20012102e6374c796c14e20c7a8315d74b42f721979a49df27ed87f30b9f4b015001bd1c00000000"]'

Alternatively, following also works:

0➜  rctesting bcli testmempoolaccept "[\"$signed_parent_tx\",\"$signed_child_tx\",\"$signed_grandchild_tx\"]"

Similarly for the following too:

0bcli submitpackage [\"$v3_parent_signed_tx\",\"$high_fee_child_signed_tx\"]

rkrux commented at 11:33 am on March 25, 2024: none

While testing the migratewallet RPC, I realised highlighting the descriptors field in the response of getwalletinfo might make the aim of this activity more apparent to the testers. Also, running the same command before the migration and comparing both the outputs would also be helpful IMO.

Sample output after the migration:

 0➜  rctesting wallet getwalletinfo
 1
 2{
 3  "walletname": "legacy_wallet",
 4  "walletversion": 169900,
 5  "format": "sqlite",
 6  "balance": 149.99997810,
 7  "unconfirmed_balance": 0.00000000,
 8  "immature_balance": 0.00000000,
 9  "txcount": 4,
10  "keypoolsize": 4000,
11  "keypoolsize_hd_internal": 4000,
12  "paytxfee": 0.00000000,
13  "private_keys_enabled": true,
14  "avoid_reuse": false,
15  "scanning": false,
16  "descriptors": true,
17  "external_signer": false,
18  "blank": false,
19  "birthtime": 0,
20  "lastprocessedblock": {
21    "hash": "5f7ed9c408fbec18dcfbfdb36152b3e67bcc79f7c1023896a86a853c6c5c4665",
22    "height": 103
23  }
24}

rkrux commented at 12:16 pm on March 25, 2024: none

@tdb3 I retested the v2 transport feature again, and this time I can see v2 connections to 2 of the peers - we can add one line in the guide mentioning which nodes these must be. I can confirm that one of the nodes is 74.208.205.30:38333 and the other one resolves to an IP that I don’t recognise - neither it matches the IP bitcoin.achow101.com resolves to using nslookup, nor it’s an inbound connection - instead it uses P2P_V2 in the servicenames, which I believe seems to agree with what’s mentioned in the PR description.

 0➜  rctesting bcli getpeerinfo | jq '.[] | "\(.transport_protocol_type) | \(.subver) | \(.connection_type) | \(.inbound) | \(.servicesnames) | \(.lastrecv)"'
 1"v1 | /Satoshi:26.0.0/ | outbound-full-relay | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\"] | 1711368816"
 2"v1 | /Satoshi:26.0.0/ | outbound-full-relay | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\"] | 1711368797"
 3"v2 | /Satoshi:26.99.0/ | outbound-full-relay | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\",\"P2P_V2\"] | 1711368831"
 4"v2 | /Satoshi:26.0.0/ | manual | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\",\"P2P_V2\"] | 1711368826"
 5"v1 | /Satoshi:26.0.0/ | outbound-full-relay | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\"] | 1711368798"
 6"v1 | /Satoshi:26.0.0/ | outbound-full-relay | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\"] | 1711368814"
 7"v1 | /Satoshi:26.0.0/ | outbound-full-relay | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\"] | 1711368826"
 8"v1 | /Satoshi:24.0.1/ | outbound-full-relay | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\"] | 1711368792"
 9"v1 | /Satoshi:26.0.0/ | outbound-full-relay | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\"] | 1711368833"
10"v1 | /Satoshi:25.0.0(inquisition)/ | block-relay-only | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\"] | 1711368835"
11"v1 | /Satoshi:24.99.0/ | block-relay-only | false | [\"NETWORK\",\"WITNESS\",\"NETWORK_LIMITED\"] | 1711368806"

cbergqvist commented at 4:16 pm on March 25, 2024: contributor

0➜  rctesting coinbase_txid=$(bcli listunspent | jq -r .[0].txid)
1zsh: no matches found: .[0].txid
This command needs to be replaced with the following instead. coinbase_txid=$(bcli listunspent | jq -r '.[0].txid')

This probably comes from differences between zsh and bash. But if it’s just a case of adding some quotes I agree it’s worth doing to remove stumbling blocks.

davidgumberg commented at 6:36 pm on March 25, 2024: contributor

@rkrux

While testing the migratewallet RPC, I realised highlighting the descriptors field in the response of getwalletinfo might make the aim of this activity more apparent to the testers. Also, running the same command before the migration and comparing both the outputs would also be helpful IMO.

Good idea, I’ve updated the test instructions to make more detailed use of getwalletinfo

I retested the v2 transport feature again, and this time I can see v2 connections to 2 of the peers - we can add one line in the guide mentioning which nodes these must be. I can confirm that one of the nodes is 74.208.205.30:38333 and the other one resolves to an IP that I don’t recognise - neither it matches the IP bitcoin.achow101.com resolves to using nslookup, nor it’s an inbound connection […]

That is expected, if you used bitcoin.achow101.com as the seed node, a short lived addrfetch connection is made and terminated once you receive an ADDR from them. The other v2 node is probably someone that the seed node told you about!

itornaza commented at 7:19 pm on March 25, 2024: none

Replicated all tests as described in the test guide and all works fine!

However, when I use the ADDRESS in the following commands:

export ADDRESS=$(bcli26 -rpcwallet=test getnewaddress) bcli26 -rpcwallet=test -named send outputs="{\"$ADDRESS\": 1}" fee_rate=10 bcli26 -rpcwallet=test -named send outputs="{\"$ADDRESS\": 1}" fee_rate=10

I am getting an error, probably because \"$ADDRESS\" is not recognised:

0$ bcli26 -rpcwallet=test -named send outputs="{\"$ADDRESS\": 1}" fee_rate=10
1error code: -5
2error message:
3Invalid Bitcoin address:

I circumvented that by manually creating and using two fresh addresses. Just leaving that note here in case someone has the same issue.

Just a few tips for macos:

ls --full-time is not supported but we can use ls -lT instead to get the mempool.dat last edit timestamp.
bcli testmempoolaccept [\"...\"] does not work, but wrapping the bracketed section with quotes bcli testmempoolaccept "[\"...\"]" does work.

davidgumberg commented at 7:44 pm on March 25, 2024: contributor

@itornaza

Thanks, I’ve changed the guide to use a shell variable instead of an environment variable, which probably fixes that issue.

0address=$(bcli26 -rpcwallet=test getnewaddress)
1bcli26 -rpcwallet=test -named send outputs="{\"$address\": 1}" fee_rate=10
2bcli26 -rpcwallet=test -named send outputs="{\"$address\": 1}" fee_rate=10

cbergqvist commented at 9:58 pm on March 25, 2024: contributor

0➜  rctesting coinbase_txid=$(bcli listunspent | jq -r .[0].txid)
1zsh: no matches found: .[0].txid
This command needs to be replaced with the following instead. coinbase_txid=$(bcli listunspent | jq -r '.[0].txid')
This probably comes from differences between zsh and bash. But if it’s just a case of adding some quotes I agree it’s worth doing to remove stumbling blocks.

bcli testmempoolaccept ["…"] does not work, but wrapping the bracketed section with quotes bcli testmempoolaccept “["…"]” does work. @rkrux & @itornaza Thanks for the reports! I have now gone through the v3 txn policy section with zsh as that seemed to be the major offender and updated the guide. Let me know if you encountered issues in any of the other tests, but at least now we know the pattern.

cbergqvist commented at 10:14 pm on March 25, 2024: contributor

ls --full-time is not supported but we can use ls -lT instead to get the mempool.dat last edit timestamp. @itornaza added note about alternative ls parameters (seems to be fairly XOR, one working on my Mac (BSD) and the other on Linux (GNU)).

itornaza commented at 11:39 am on March 26, 2024: none

I think that our designated offender zsh does it again on the following script:

Now, we’ll send 200 UTXOs of 0.003 tBTC to our test wallet. We send them in 10 batches of 20 transactions, mining a block in between to confirm each batch.

 0num_batches=10
 1tx_per_batch=20
 2
 3for batch in $(seq 1 $num_batches); do
 4    echo "Creating batch $batch of $tx_per_batch transactions..."
 5    for tx in $(seq 1 $tx_per_batch); do
 6        bcli -named -rpcwallet=coinbasewallet sendtoaddress address=$(bcli -rpcwallet=testwallet getnewaddress) amount=0.003 fee_rate=15
 7    done
 8    echo "Generating 1 block to confirm the batch..."
 9    bcli -rpcwallet=coinbasewallet generatetoaddress 1 $(bcli -rpcwallet=coinbasewallet getnewaddress)
10done

returning an error like:

0Creating batch 10 of 20 transactions...
1./batch.zsh:11: command not found: -rpcwallet=testwallet
2./batch.zsh:11: command not found: -named

Using a shell variable as @davidgumberg suggested in one of the previous fixes, works here as well. I am including a possible fix in the script below, using bcli_local to distinguish from the bcli alias we have defined before.

 0#!/usr/bin/env zsh
 1
 2num_batches=10
 3tx_per_batch=20
 4
 5bcli_local=($BINARY_PATH_27/bitcoin-cli -datadir=$DATA_DIR_27)
 6
 7for batch in $(seq 1 $num_batches); do
 8    echo "Creating batch $batch of $tx_per_batch transactions..."
 9    for tx in $(seq 1 $tx_per_batch); do
10        $bcli_local -named -rpcwallet=coinbasewallet sendtoaddress address=$($bcli_local -rpcwallet=testwallet getnewaddress) amount=0.003 fee_rate=15
11    done
12    echo "Generating 1 block to confirm the batch..."
13    $bcli_local -rpcwallet=coinbasewallet generatetoaddress 1 $($bcli_local -rpcwallet=coinbasewallet getnewaddress)
14done

cbergqvist commented at 2:09 pm on March 26, 2024: contributor

I think that our designated offender zsh does it again on the following script: @itornaza Weird, I did not run into issues using zsh in this case actually. Maybe I’m doing something slightly different? I dropped the defining of $DATA_DIR... etc into a test.sh file the I execute through:

0source test.sh

I dropped this CoinGrinder script in its original form into a second file and ran it with…

0source generate_utxos.sh

…both under the same zsh session and it worked fine. Same result after adding #!/usr/bin/env zsh at the top.

I was using zsh 5.9 under NixOS.

itornaza commented at 3:59 pm on March 27, 2024: none

@itornaza Weird, I did not run into issues using zsh in this case actually. Maybe I’m doing something slightly different? @cbergqvist you are right! I was not sourcing the script because I have defined the aliases and exports within .zshrc, however, I only now realize that aliases are not expanded in non-interactive shells by default!

AngusP commented at 5:27 pm on March 27, 2024: contributor

A little lazy, as it’s not that hard to look up myself, but linking to the PR(s) that made the change being tested would be handy, if not too difficult (like a stack of 20 different PRs)

davidgumberg commented at 5:28 pm on March 27, 2024: contributor

A little lazy, as it’s not that hard to look up myself, but linking to the PR(s) that made the change being tested would be handy, if not too difficult (like a stack of 20 different PRs)

There are links at the top of the guide in the test index to all of the relevant PR’s.

alfonsoromanz commented at 1:52 pm on March 31, 2024: none

Hello! Thanks for putting this together. I’m almost finished testing using the guide. A small feedback: In the section “Migrate a Legacy Wallet” it says:

Take note of the ‘format’ (bdb), ‘descriptors’ (true), and balance values in the output of getwalletinfo.

Maybe you meant ‘descriptors’ (false)? Given that it was created with the flag descriptors=false, and it also shows false when running getwalletinfo

davidgumberg commented at 6:04 pm on April 1, 2024: contributor

@alfonsoromanz Thanks for catching this! I’ve corrected the mistake.

fanquake closed this on Apr 15, 2024

27.0 RC Testing Guide Feedback #29685