RFC: Signet meta discussion #19787

Moving discussion for meta questions from the current open Signet PR here as I think there is a lot to discuss on the meta level that warrants its own Issue. It would be good to leave #18267 for review of the code. I will continue to add open questions here. @instagibbs comment: Under what circumstances would signet be rebooted? I presume any time there is suspected key theft/loss? Should nets just have a limited lifespan?

Bitcoin StackExchange on being able to experiment on Signet with multiple proposed soft forks whilst maintaining the ability of full nodes to validate the Signet chain.

Socratic Seminar on testnet, regtest and Signet. [edit: now linking to transcript]

Thanks!

To be clear I think the current 1-of-2 signet setup is fine, it's experimental, and getting stuff deployed to gain experience is probably the best thing. Going forward we need to set expectations so people don't spend energy on integration if it's not worth it from their end with respect to lifespan, etc.

@instagibbs I think the general idea is that we keep signet running indefinitely.

If at some point in time, one of the keys or its owner goes missing/is leaked/becomes otherwise invalid, we have two options:

1. We can rewrite the challenge to exclude the bad key, turning the 1-of-n into a 1-of-(n-1). ~(The solution does not commit to the challenge, so this is totally doable).~ (Edit: correction: new psbt style does commit to the challenge; perhaps a 'original challenge' and a 'updated challenge' as two separate settings could let us just arbitrarily go on forever. Would have to think about this.) It would by default result in a magic change, which may given the circumstances be a good thing, but since serialization also depends on this for seeking, it may not be ideal. (We may have to add some "signet starts at block X feature for this to be functional from IBD, as there may be blocks signed using the now invalidated key)
2. We can start a new signet.

The same rules apply if the owner turns evil and tries to... I don't know, disrupt the signet network users.

If some early-activated soft fork turns out to be buggy and causing too much of a headache code-wise to account for*, we may decide to reboot signet, or do a big reorg back to before the problem occurred. This I think will require some time/trial and error.

(* The current idea seems to be that we can probably get away with a lot of buggy soft forks by simply moving the goal post^W^Wactivation date as we run into issues, thus causing all the early buggy uses to transform into OP_TRUEs. @ajtowns was looking into this.)

The solution does not commit to the challenge

With the recent psbt change to BIP325, the solution does commit to the challenge (via the scriptPubKey of the two virtual txs), so both the old challenge as well as the new challenge needs to be provided in some way or another to validate the full history and future blocks. See also the slightly related discussion of different reorg policies here: https://github.com/bitcoin/bips/pull/947#discussion_r475115137

Going forward we need to set expectations so people don't spend energy on integration if it's not worth it from their end with respect to lifespan, etc.

Yeah longer term I think getting a proposed soft fork activated on Signet should be a prerequisite before being considered for activation on mainnet. One of the later stages after getting a BIP number, BIP review and open PR in Bitcoin Core repo.

(This is assuming one public Signet network run by Kalle, AJ that most people use with other smaller independent, permissionless Signets run by random other individuals.)

An approach like this makes sense to me to facilitate testing of a proposed soft fork (with possible multiple iterations) whilst maintaining ability of full nodes to validate the whole chain. Although I don't know how this OP_TRUE approach for each iteration of a proposed soft fork would work (unless it was a soft fork introducing a new opcode).

Obviously reboots and massive re-orgs are always an option but we should aim to avoid them if we possibly can.

@MarcoFalke Thanks, I forgot about the psbt update now committing to the challenge. I updated my response above.

I think we want to maintain the default signet in whatever way is optimal for as many people as possible to spend energy on integration, so that there's a super convenient testing infrastructure for it -- explorers, wallet support, debugging/attack tools, a reasonably complicated network of lightning nodes, etc. Running a custom signet is a great option to have, but it's more work if you also have to setup your own explorer and faucet, or can't test against other apps, or have to recompile debugging tools, etc.

Maybe that means we want to have the default signet only last 12 or 24 months, so you can sync the chain without downloading lots of old data; or eventually expire txs that got mined that now don't comply with new soft-fork rules. I know I find syncing testnet tedious, but otoh testnet has ~34 years worth of blocks, so it's probably not a problem for signet anytime soon.

I think the approach where the genesis block stays the same between the current default signet, custom signets, and any future rebooted default signet should minimise the work needed even if there is a chain restart though.

If block signing keys get compromised, that's definitely a hard-fork event; maybe you could preserve the existing chain even then by having some "assumevalid"-like behaviour though.

I think unintended reorgs can and should be avoided. There should be some sort of expectation for a stable default "append-only" "never-reorg" signet that explorers etc can use. Once a signet block on that chain has been mined, it can be assumed to be permanent. (Edit: This comment was about reorgs caused by consensus deployments, not about reorgs that happen in the network naturally)

Deploying a soft-fork (or even un-deploying a soft-fork, aka "hard fork") can simply be done by restarting the signet miner with new consensus rules. Picking appropriate activation times (as mentioned above) should be sufficient to avoid invalidation of existing blocks already in the chain.

I think any consensus deployment that activated on mainnet, can also be considered permanent on signet. Anything else should be considered experimental.

I am also wondering if the discussion here is better suited on the mailing list. When it comes to Bitcoin Core, we are mostly interested in implementing the BIP in code. The meta discussion might benefit from a larger audience.

@MarcoFalke Agree with the soft-fork comments; think I just about have a branch where all that should work out of the box now, which (unless it all falls apart) I'll post about on the list...

There should be some sort of expectation for a stable default "append-only" "never-reorg" signet that explorers etc can use.

That bothers me a bit -- you can't rely on a lack of reorgs on the main chain, so relying on it on a test chain might build bad habits? I think doing an slightly weaker expectation of marking blocks that are will be reorged (and maybe a different marking for when they contain intentional double-spends that will be invalidated in the reorg) might be the way to go? I think using version bits for those markers should work, so long as we can avoid needing them for activating soft forks.

I am also wondering if the discussion here is better suited on the mailing list

Agreed. I will post on the mailing list once I have the transcript of the Signet Socratic done. Feel free to post before then though.

I feel like having no reorgs kind of defeats the purpose as it no longer resembles the main chain. Having a guaranteed no 2+ block reorgs is fine, IMO.

Makes sense.

Closing for now per #19787 (comment)

I think unintended reorgs can and should be avoided. There should be some sort of expectation for a stable default "append-only" "never-reorg" signet that explorers etc can use. Once a signet block on that chain has been mined, it can be assumed to be permanent.

It is impossible to avoid reorgs in a multisig block signer setup. In any signing protocol there comes a point where one person has attained their threshold of information necessary to publish a signed block. For example in a k-of-n setup where one signer has possession of k-1 signatures from other signing nodes, giving them the choice to add their own signature and complete the block, or pretend to fail at the final step. They could wait for failure and allow the next proposed block to succeed, then publish their signature for the old block. Now there are two valid signed blocks for that height. Even if the chain continues to be built according to the block that didn't fail out of the signing protocol, some other node might see the withheld block first, then see a reorg to the other chain when subsequent blocks are published.

It is, AFAIK, impossible to avoid this scenario, though the details would on the signing protocol used. This is why, for example, Blockstream's Liquid product does not promise immediate finality, but rather has an at-most-1-block reorg policy. Transactions are final on the 2nd confirmation.

Regarding the original question about key rotation, this is a strong argument for using a (possibly MuSig) Schnorr aggregate key instead of a traditional k-of-n ECDSA setup. Besides saving space, it allows the signing functionary set to grow, shrink, sub delegate, rotate keys, etc. without changing the public parameters at all. People watching the chain need not even be aware that the signing set went from a 1-of-2 to a 2-of-3, for example, or that a new split was performed to remove a presumed compromised key.

bitcoin-dev mailing list post: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-August/018145.html