Bech32 address generation in regtest does not have the prefix tb #12314

issue pedrobranco openend this issue on January 31, 2018
  1. pedrobranco commented at 4:58 pm on January 31, 2018: contributor

    Steps to reproduce:

    0❯ bitcoin-cli --rpcuser=x --rpcpassword=y -regtest getnewaddress "" "bech32"
    1bcrt1q6rhpng9evdsfnn833a4f4vej0asu6dk5srld6x
    

    By BIP173 Segwit address format bitcoind should return a tb prefixed address for testnet, and by developer examples, regtest is a blockchain with the same basic rules as testnet.

    Expected behavior

    bitcoind should return a bech32 address with the same prefix as testnet.

    Bitcoind version

    0.16.0rc1

  2. achow101 commented at 5:05 pm on January 31, 2018: member
    To distinguish regtest from testnet and mainnet, the human readable part of a bech32 address on regtest is bcrt. Not everything with regtest follows that of testnet, nor does it need to.
  3. sipa commented at 5:11 pm on January 31, 2018: member
    Yes, this is intentional. I don’t see a reason why regtest should produce addresses that can be confused with testnet.
  4. pedrobranco commented at 5:15 pm on January 31, 2018: contributor

    Not everything with regtest follows that of testnet, nor does it need to.

    Agree on that. By reading the developer examples and the BIP173 i was lead to expecting a tb prefixed address.

    Yes, this is intentional. I don’t see a reason why regtest should produce addresses that can be confused with testnet.

    Thanks for confirming 👍

  5. pedrobranco closed this on Jan 31, 2018

  6. stevenroose commented at 9:07 am on March 22, 2018: contributor
    There is no mention of the bcrt prefix in BIP 173 though, neither does it mention regtest mode.
  7. stevenroose referenced this in commit 78140d5fa3 on Mar 22, 2018
  8. sipa commented at 9:31 am on March 22, 2018: member
    regtest is a Bitcoin Core specific test network. It’s not mentioned in the BIP because it’s not something that affects Bitcoin.
  9. stevenroose commented at 10:09 am on March 22, 2018: contributor

    I do agree that regression testing should primarily be done within one repository, but it does make sense to have a similar structure for integration testing between implementations. Next to the official test network, I think it makes sense to have some standards on a single regression test network to facilitate cross-implementation testing.

    I realize that standards in Bitcoin currently mostly go hand in hand with Core development, but with Lightning the process has been way more diverse, creating a need for some form of standard integration testing.

    So, while the integration efforts could specify a custom regtest chain, it would be way easier if different implementations could already provide a compatible one.

  10. maflcko commented at 4:45 pm on March 22, 2018: member

    We tend to change the regtest parameter whenever it makes our testing or maintaining easier, so even if it is unlikely that the bech32 prefix will be changed, I am slightly -0 on adding it to the BIP.

    Also see #8994, which makes it easier to launch regtest like chains and thus impossible to documents all of them in the BIP.

  11. ejose19 commented at 4:40 pm on September 26, 2019: none
    Is there a way to differentiate between a regtest and testnet for non bech32 addresses?
  12. dpc commented at 6:11 pm on January 31, 2020: none

    @sipa:

    Yes, this is intentional. I don’t see a reason why regtest should produce addresses that can be confused with testnet.

    Reasons:

    • It introduces additional complexity. And therefore good reasons to actually introduce such distinction should be given, and not good reasons not to introduce it. What practical problems does having this distinction solve?
    • For good or bad, the existing address formats were the same on testnet vs regtest. There is a lot of existing code out there that expect no difference between tesnets and regtests addresses. Such distinction breaks some long established assumptions.
    • It creates confusion. Many places are not even mentioning this prefix when listing details of bech32. Example: https://en.bitcoin.it/wiki/List_of_address_prefixes . From what I’ve discovered in the discussion above, not even BIP-173 mentions it. :confused:
    • (I think,) the reason to have distinction between mainnet and testnet is to protect end users from making mistakes by confusing them, which could lead to losing real mainnet bitcoin. Such confusion between regtest and testnet is benign. Protecting users from loosing testnet coins on regtest, or tegtest coins on testnet is pointless.
    • One way to test complex systems integrating Bitcoin, is to swap a normal node with a regtest node, and manually drive it to exercise certain scenarios (say: reorgs) and drive the block creation much faster than a real testnet/mainnet (so your test suite can finish quickly). That was all great before, but now with this change the “compex system” under test has to be altered (possibly in many places) just to make it support another “network”. A network that is exactly like testnet network, but with one little difference that does not seem to have much reason to exist in the first place.

    There is still not too late to get rid of bcrt. It’s not a consensus change (since regtests are in consensus with themselves anyway), and I doubt anyone would miss it.

  13. murchandamus commented at 6:28 pm on January 31, 2020: contributor
    Can confirm that this has cost us several hours of troubleshooting and writing extra code. I agree with @dpc that I don’t quite see the advantages of a different hrp.
  14. sipa commented at 6:44 pm on January 31, 2020: member

    I think part of the issue here is the question whether regtest is an internal Bitcoin Core thing, or a framework for testing infrastructure on top.

    Historically at least, regtest was purely an internal thing for Bitcoin Core’s own tests. There is no standardization, and it has undergone rule changes several times whenever it was convenient.

    On the other hand, testnet was explicitly designed for applications to test integration with, is dealt with in BIPs, though I can see why it may be hard to use in some cases.

    I don’t know what the solution is. I’m opposed to adding regtest specifications to BIPs in general, as this would make it harder to adapt to changes that need testing in Core itself. We could consider changing the regtest prefix to something else; I don’t think it’s a strong reduction in test capability.

    Overall, the expectation of using regtest externally may bite us again in other situations. Maybe we need even more networks? (signet comes to mind, though global networks aren’t what you always need either).

  15. dpc commented at 7:35 pm on January 31, 2020: none

    @sipa I guess I can use this opportunity to share some thought of nature of Bitcoin “nets” from my experiences at Bitgo (thought are mine though). Please excuse if it is a bit scattered and random.

    I’ll start with the main point:

    On the IDs, hashes, addresses, etc. level there’s no reason to make any distinctions between testnet, regtest, simnet or whatever other non-mainnets. Conceptually the are just two types of networks: real one, and not real one. Mostly for the end user not to ever get confused. If the software needs to know if it is running against testnet vs regtest vs simnet, it can carry that information out of band - that’s not a problem. It doesn’t have to be embedded in every single address. Quite the opposite: having to potentially come up with additional logic in every single piece of code handling Bitcoin, to support every type of “net” that we could come up with is just a hindrance.

    General perspective on nets:

    There’s mainnet - the real deal. Everybody understands mainnet.

    Then there’s a testnet. What’s not-obvious is that testnet is not a test network for companies integrating Bitcoin. It’s a test network for end users of Bitcoin. It’s global, shared and behaves almost exactly like a mainnet, so users can do testing of the whole ecosystem, send funds between different companies and so on. From the perspective of a company that integrates with Bitcoin, testnet is actually what I call a “prod-net” - production network, exposed to end users. When testnet systems are having problems, users report normal support cases to get it fixed, just like a mainnet prod systems.

    Testnet is also not really suitable for Bitcoin integration testing. It is just too slow and does not give enough control. Any practical yet extensive test suite needs to have an ability to control mining and what exactly goes on the block. Regtest is perfect for that. I don’t know if it was intended for it or not, but in practice it works great.

    Regtest node is not a mock, fake or whatever. It’s a real Bitcoin Core node. From an outside perspective of a blockchain data consumer it looks very much like a normal testnet node. But you can drive it manually like a mock. You can setup a test instance of your whole system, swap a node for a regtest node, drive it with rpc commands in your test suite and see if what comes in and out is what you expect. You don’t have to wait for blocks being mined, and you can simulate stuff like reorgs. Consensus with anyone else irrelevant, because there’s no way to coordinate anyway.

  16. dpc referenced this in commit de210ecc92 on Feb 3, 2020
  17. dpc referenced this in commit 99e63500f7 on Feb 4, 2020
  18. dpc commented at 10:07 pm on February 4, 2020: none

    For anyone in a similar situation: I eventually figured out that it is probably easier to patch bitcoin core node to just not use this custom prefix, than try to introduce and maintain all the code to support it.

    Indeed the fix is simple: https://github.com/dpc/bitcoin/commit/99e63500f767f5dbf966280e9b2a27fb4aecb213 , though from what I can tell it will break a lot of existing tests that expect bcrt.

    I asked around and I still haven’t seen a single practical problem that different prefixes would solve, so I hope the whole idea can still be revised, especially that it is not baked into any consensus or even the BIP itself.

  19. maflcko commented at 11:05 am on February 5, 2020: member

    I don’t think the Bitcoin Core test suite cares what prefix is used for the regtest chain, so I don’t mind changing it to tb. There shouldn’t be any downsides and if it helps testing of 3rd party applications, why not?

    If someone wants to pick this up, a good start would be to cherry-pick https://github.com/dpc/bitcoin/commit/99e63500f767f5dbf966280e9b2a27fb4aecb213 and then fixup the test from there.

  20. maflcko reopened this on Feb 5, 2020

  21. maflcko added the label Utils/log/libs on Feb 5, 2020
  22. dpc commented at 7:35 pm on February 5, 2020: none

    I just hope there’s an agreement on the reasoning here. I just point out that there is a substantial cost in making every downstream consumer of Bitcoin Core RPC deal specifically with minor details of distinct “testnets”. If there’s a good practical reason to have such a distinction that have benefits out-weighting the costs, than maybe there are alternative approaches like making it configurable, etc. But if no one has any use for it, than it seems simplest to just get rid of it.

    I have also not played with signet or any other potential testnet-like modes/setups, so if someone is familiar with them - it might be worthwhile to rethink if there are any cases like that there. Ideally, if a downstream consumer decides to e.g. switch from testnet to signet for a better user-experience, it would be great if that did not require any alterations.

  23. sipa commented at 8:37 pm on February 5, 2020: member

    @dpc I don’t think there is any reason why having a separate prefix for regtest is advantageous to us, but at the time it was introduced, it was also assumed to be of no cost. I had no idea people were using regtest for testing other things than Bitcoin Core itself.

    As I said, if there’s a good reason (like you seem to have), changing it to ’tb’ is fine by me.

    However, I am worried about encouraging people using regtest for external testing in general. We can’t guarantee that regtest won’t undergo radical changes as our test needs change. This makes me wonder if we shouldn’t define a standard “external regtest” network instead, with an intention if it being as close as possible to testnet (or mainnet?) as possible while remaining usable for testing.

    PS: why ’tb’ and not ‘bc’? Regtest is closer to mainnet than to testnet in many aspects.

  24. dpc commented at 9:21 pm on February 5, 2020: none

    @sipa

    We can’t guarantee that regtest won’t undergo radical changes as our test needs change. This makes me wonder if we shouldn’t define a standard “external regtest” network instead

    At least from our perspective, anything like this works. We can pass different flags, change config, etc. That would be a local change, and not introduce any work downstream of the rpc interface.

    I could imagine, that in the future there could be a mode even more tailored to external integration testing, separate from regtest, but as things are right now it’s not really neccessary. I realize that the reward schedule is different on regtest, and probably things like difficulty, but these are not very important - a downstream service just takes the data returned by Bitcoin Core at its face value, and is not in the business of second guessing the consensus rules. If Bitcoin Core says the coinbase tx had a 25 BTC coinbase output at height 50, then that’s what must have really happened.

    PS: why ’tb’ and not ‘bc’? Regtest is closer to mainnet than to testnet in many aspects.

    Seems that testing against “testnet” is the most natural. If someone has any sort of service that needs additional testing APIs etc. they will turn such things on in the configs of their non-production env. which would typically be a regtest/testnet.

  25. ejose19 commented at 9:48 pm on February 5, 2020: none

    @sipa

    This makes me wonder if we shouldn’t define a standard “external regtest” network instead

    That indeed sounds like a good idea, as regtest is currently used to do any kind of tests and not only Bitcoin Core tests.

    Whenever to use tb or bc, I think it’s important to stick to the same configuration on all types of addresses for regtest and not just bech32, since in current implementation legacy regtest and testnet are exactly the same (even bitcoinrpc says one is valid in the other network), which is not the case for bech32 (and additionals checks needs to be made for properly testing)

  26. murchandamus commented at 0:51 am on February 8, 2020: contributor

    PS: why ’tb’ and not ‘bc’? Regtest is closer to mainnet than to testnet in many aspects.

    We just really need to distinguish mainnet from testing funds, and regtest is not mainnet. :) Regnet is expedient to test e.g. behaviour around confirmations, channel open and closes, and making indexers safe through chainforks. I’d actually bet that a ton of other Bitcoin projects use regtest for some controlled environment end-to-end tests.

  27. maflcko commented at 4:29 pm on February 9, 2020: member

    However, I am worried about encouraging people using regtest for external testing in general. We can’t guarantee that regtest won’t undergo radical changes as our test needs change. This makes me wonder if we shouldn’t define a standard “external regtest” network instead, with an intention if it being as close as possible to testnet (or mainnet?) as possible while remaining usable for testing.

    We include regtest “hardforks” and other breaking changes in the release notes (e.g. https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.16.0.md#testing-changes or https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.19.0.1.md#tests), so I think regtest is already meant to be also used externally.

  28. maflcko commented at 8:11 am on August 28, 2020: member
    According to #18267 (review) rust-bitcoin requires the prefixes to be unique per network. Not sure if that is something to consider here, but I wanted to mention it.
  29. dr-orlovsky commented at 12:53 pm on September 21, 2020: none

    It’s not that rust-bitcoin can’t work with the same prefix for multiple networks; it can: https://github.com/rust-bitcoin/rust-bitcoin/pull/291/files#diff-7769ce7612a45cca417dce18a5aa6a50R388

    The problem is that when rust-bitcoin (or, I assume any other library) parses address, it needs somehow to distinguish whether tb comes from testnet, regtest or signet (like here: https://github.com/rust-bitcoin/rust-bitcoin/pull/291/files#diff-7769ce7612a45cca417dce18a5aa6a50R417) - and it’s not clear how to do that.

  30. dpc commented at 5:50 pm on September 21, 2020: none

    it needs somehow to distinguish whether tb comes from testnet, regtest or signet

    Why?

  31. dr-orlovsky commented at 6:20 pm on September 21, 2020: none
    @dpc not sure I am getting your question. Because the code is written in that way… If you reconstruct network type from bech32 string you need to return some enum variant.
  32. sipa commented at 6:23 pm on September 21, 2020: member

    It’s very surprising to me that you at any point in time would want to decode an address without knowing what network it’s for.

    I also don’t see why that isn’t a problem already - P2PKH and P2SH addresses for regtest are the same as for testnet.

  33. dr-orlovsky commented at 9:45 am on September 22, 2020: none
  34. maflcko added this to the milestone 0.22.0 on Sep 22, 2020
  35. rustyrussell commented at 9:29 pm on September 22, 2020: contributor

    Just to confirm, at p2pkh and p2sh addresses for signet are identical to testnet? Otherwise, we have a problem with bolt11 invoices and fallback addresses:

    0        assert b11['fallbacks'][0]['type'] == 'P2PKH'
    1>       assert b11['fallbacks'][0]['addr'] == 'mk2QpYatsKicvFVuTAQLBryyccRXMUaGHP'
    2E       AssertionError: assert 'sNkrc4kvorCt...cnzRg2jLt6saY' == 'mk2QpYatsKic...ryyccRXMUaGHP'
    3E         - mk2QpYatsKicvFVuTAQLBryyccRXMUaGHP
    4E         + sNkrc4kvorCtNKT7o34nycnzRg2jLt6saY
    

    But signet support in c-lightning was added while signet was still in flux, so maybe it doesn’t use different address versions any more?

  36. dr-orlovsky commented at 10:48 pm on September 22, 2020: none
    Yes, @rustyrussell, the addresses are identical: #18267 (review) and Signet just yesterday got merged into Bitcoin Core master
  37. nkohen commented at 8:04 pm on September 29, 2020: none
    I see that I’m too late to this party but for posterity I’d just like to say that I think that the decision made here was wrong because we have an hrp anyway and to decide to convey less information with these bytes makes no sense to me and complicates development (marginally) and helps no one.
  38. laanwj removed this from the milestone 22.0 on Jun 14, 2021
  39. michaelfolkson commented at 11:06 am on June 14, 2021: contributor
    I think this issue can be closed. There seem to be valid arguments for and against the status quo (StackExchange question on the current status here) but I can’t see it being changed now, over three years later since this issue was opened. Testnet and Signet addresses start tb and regtest addresses start bcrt. Energy is probably better spent ensuring community documentation and tutorials accurately reflect this.
  40. maflcko commented at 1:42 pm on June 14, 2021: member
    Now that third party software likely supports duplicate prefixes, i.e. not treating the prefix as a unique network id, it could actually make sense to switch regtest over. I think it was good to hold back changing the regtest prefix before signet came out. Not saying whether it should be changed, but now seems a better time to change than it was before signet came out.
  41. michaelfolkson commented at 5:50 pm on June 14, 2021: contributor
    @MarcoFalke: If the plan was for regtest, testnet and signet to all share the same prefix third party software would have had to have supported duplicate prefixes three years ago? I don’t get the rationale that it potentially makes more sense today than it did three years ago.
  42. bilthon commented at 5:51 pm on August 11, 2022: none

    I have to say that this singe issue can be a massive productivity drain for people trying to build testable infrastructure on top of bitcoin core.

    I understand some internal tests might break, but how much work are we talking about here? I’d be willing to contribute in fixing those in my spare time if it means that we get rid of the bcrt prefix forever.

  43. maflcko commented at 6:24 pm on August 11, 2022: member

    I don’t get the rationale that it potentially makes more sense today than it did three years ago.

    I think there was software out there (citation needed) that treated bech32 address prefixes as a unique identifier for the whole chain. Now that there are two chains with the same prefix (and affected software likely removed that incorrect assumption), there is one less reason to not do the change, thus it makes more sense.

  44. benthecarman commented at 6:27 pm on August 11, 2022: contributor

    I have to say that this singe issue can be a massive productivity drain for people trying to build testable infrastructure on top of bitcoin core. I understand some internal tests might break, but how much work are we talking about here? I’d be willing to contribute in fixing those in my spare time if it means that we get rid of the bcrt prefix forever.

    I think it is quite the oppposite, most (if not all) bitcoin libraries today support the bcrt prefix. There is no benefit to changing it to tb, you are only destroying information that otherwise would be in the address (if it is regtest vs testnet)

  45. bilthon commented at 11:28 pm on August 11, 2022: none

    Information that, as has been stated repeatedly before is not only irrelevant, but can easily be communicated out of band. And I don’t know what libraries you’re using but bitcoinjs-lib is giving me trouble with that.

    Anyways, I don’t want to over-extend an already extended thread, but keeping something that is clearly giving multiple devs more trouble than it is solving because in some hypothetical edge-case it might be useful doesn’t sound like a good reason.

  46. benthecarman commented at 11:32 pm on August 11, 2022: contributor

    Information that, as has been stated repeatedly before is not only irrelevant, but can easily be communicated out of band. And I don’t know what libraries you’re using but bitcoinjs-lib is giving me trouble with that.

    Anyways, I don’t want to over-extend an already extended thread, but keeping something that is clearly giving multiple devs more trouble than it is solving because in some hypothetical edge-case it might be useful doesn’t sound like a good reason.

    Looks like bitcoinjs-lib has support for bcrt

    https://github.com/bitcoinjs/bitcoinjs-lib/blob/5137976e7b420500aaf37af52919d301b9b94c9b/src/networks.js#L15

  47. msgilligan commented at 10:00 pm on February 21, 2023: none
    Add bitcoinj (https://github.com/bitcoinj/bitcoinj) to the list of implementations that use bcrt.
  48. maflcko commented at 3:25 pm on February 22, 2023: member
    Closing due to lack of direction and progress. Pull requests with improvements are always welcome.
  49. maflcko closed this on Feb 22, 2023

  50. bitcoin locked this on Feb 22, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me