Change address selected randomly causes problem with hw. wallet #19570

Using Core with hardware wallets (ColdCard in this case) comes with a problem: When you create a PSBT with walletcreatefundedpsbt it pulls a random change address. Problem is: ColdCard (I think justly) says: it is out of order and displays a strong warning. Like in our case in a pool of 1000 addresses ColdCard scans and accepts the first 200.

Solution: It would be nice to have an option (default?) to pull change addresses in the derivation order.

Alternative: Right now we can importmulti 200 addresses, so any random address falls into the accepted first 200. Problem is: it's easy to run out of that 200.

I believe pulling random change addresses form a HD wallet gives no extra privacy.

I'm sorry what does "random" mean in this context?

Like, an unrelated address? An address with "too high" of a BIP32 index for ColdCard to accept it?

Yes, too high in the derivation path. Not the first not used.

We imported an XPUB and filled the wallet with addresses (1,000 "real" and 1,000 change addresses using importmulti as ColdCard user's manual says). After this the change address is selected randomly from the 1,000 available addresses. Not in order as the derivation path would "dictate". Because of the large gap (617 in our case) ColdCard alerts "Change fraud".

Seems like an overly-tight "change fraud" heuristic. Ledger allows the first 50k or something without warning the user since scanning that space is pretty trivial it cannot be made a profitable ransom script. Either way, a gap that large sounds like 616-ish transactions with change were made between the two transactions you're looking at? I believe they are grabbed sequentially. @achow101 am I missing something?

They should be imported into the change keypool in order already...

Descriptor wallets fix this.

Are you doing something that is "burning" change addresses like funding a bunch of transactions without signing them?

They should be imported into the change keypool in order already...

Descriptor wallets fix this.

Are you doing something that is "burning" change addresses like funding a bunch of transactions without signing them?

No, nothing unusual. We are just testing ColdCard at the moment and this came up. The first transaction and the hw. wallet doesn't like the gap. ColdCard "likes" 200 or less gap.

They should be imported into the change keypool in order already...

Descriptor wallets fix this.

Are you doing something that is "burning" change addresses like funding a bunch of transactions without signing them?

We don't really import addresses (I believe). We provide the XPUB (from ColdCard) and Bitcoin Core derives/generates the given count of addresses (as far as I understand).

This is unexpected behavior. We have tests making sure they are drawn in order, so my assumption is you're hitting some case where CreateTransaction is succeeding(consuming a change script) but for some reason doesn't broadcast/commit to it. A single *funded* call should never do this afaik

To be clear: 10 calls to walletcreatefundedpsbt will consume up to 10 change scripts, even if you don't end up submitting the transactions.

This is unexpected behavior. We have tests making sure they are drawn in order, so my assumption is you're hitting some case where CreateTransaction is succeeding(consuming a change script) but for some reason doesn't broadcast/commit to it. A single *funded* call should never do this afaik

To be clear: 10 calls to walletcreatefundedpsbt will consume up to 10 change scripts, even if you don't end up submitting the transactions.

I see, I am positive we didn't call walletcreatefundedpsbt (or other "consumers") too many times (definitely not 617 times ;))

Just an idea: is there a way to dump addresses (in order) from Core (both internal and not). That way I could check if the import/generation was correct and how addresses are selected.

Would you be willing to share logs? If not, you'll likely have to give reproducible steps in regtest.

is there a way to dump addresses (in order) from Core (both internal and not). That way I could check if the import/generation was correct and how addresses are selected.

You can call getnewaddress and getrawchangeaddress as many times as you like, and check the index for each using getaddressinfo.

Hmm, I just tried it on a newly created wallet and now and it seems to be working normally, I mean selecting [0] index change address as it should. Sorry, this looks like a false flag. I will try to reproduce and will send the log if it comes up again.

getrawchangeaddress

tb1qy50697m53z6m7ctf574zh3ch5wdu03ywt9s8ws

getaddressinfo tb1qy50697m53z6m7ctf574zh3ch5wdu03ywt9s8ws { "address": "tb1qy50697m53z6m7ctf574zh3ch5wdu03ywt9s8ws", "scriptPubKey": "0014251fa2fb7488b5bf6169a7aa2bc717a39bc7c48e", "ismine": false, "solvable": true, "desc": "wpkh([c09dbfcb/84'/1'/0'/1/0]03969e4c3552234c7c84470ed8d15d03e9ec898e94744a94c2ec77b4aac936d25d)#t5ayjxfr", "iswatchonly": true, "isscript": false, "iswitness": true, "witness_version": 0, "witness_program": "251fa2fb7488b5bf6169a7aa2bc717a39bc7c48e", "pubkey": "03969e4c3552234c7c84470ed8d15d03e9ec898e94744a94c2ec77b4aac936d25d", "ischange": true, "timestamp": 0, "hdkeypath": "m/84'/1'/0'/1/0", "hdseedid": "0000000000000000000000000000000000000000", "hdmasterfingerprint": "c09dbfcb", "labels": [ ] }

Ok, going forward I'd try to use real descriptor wallets(released in 0.21 soonish, or try in master).

Those will be more tested/sane especially going forward..

Will do, thank you! ;)

I can confirm with both segwit and legacy addresses that address selection works fine (in derivation path order). It was our mistake, we probably did use up that many change addresses with failed calls.

Previously we had bugs where it would reuse the same change key so as a precaution it burns it every time the main transaction creation function. Could be another bug you hit but that means it did it's job I guess. On Sat, Jul 25, 2020, 8:07 AM GhostOfSatoshi notifications@github.com wrote:

I can confirm with both segwit and legacy addresses that address selection works fine (in derivation path order). It was our mistake, we probably did use up that many change addresses with failed calls.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/19570#issuecomment-663848017, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMAFU3RSIZ5QHRHAXKVQFDR5LDGLANCNFSM4PFOOG2A .