Testnet4 including PoW difficulty adjustment fix

fjahr commented at 10:55 pm on March 31, 2024: contributor

To supplement the ongoing conceptual discussion about a testnet reset I have drafted a move to v4 including a fix to the difficulty adjustment mechanism, which was part of the motivation that started the discussion.

Conceptual considerations:

The conceptual discussion about doing a testnet4 or softforking the fix into testnet3 is outside of the scope of this PR and I would ask reviewers to contribute their opinions on this on the ML instead. However, I am happy to adapt this PR to a softfork change on testnet3 if there is consensus for that instead.
The difficulty adjustment fix suggested here touches the CalculateNextWorkRequired function and uses the same logic used in GetNextWorkRequired to find the last previous block that was not mined with difficulty 1 under the exceptionf. An alternative fix briefly mentioned on the mailing list by Jameson Lopp would be to “restrict the special testnet minimum difficulty rule so that it can’t be triggered on the block right before a difficulty retarget”. That would also fix the issue but I find my suggestion here a bit more elegant.

DrahtBot commented at 10:55 pm on March 31, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	jsarenik, murchandamus
Concept NACK	kcalvinalvin
Concept ACK	pablomartin4btc
Stale ACK	wiz, russeree, Rob1Ham, Sjors, Emzy, jlopp, Retropex, craigraw, melvincarvalho
Ignored review	achow101

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#30203 (Enhance signet chain configuration in bitcoin.conf by BrandonOdiwuor)
#29686 (Update manpage descriptions by willcl-ark)
#17783 (common: Disallow calling IsArgSet() on ALLOW_LIST options by ryanofsky)
#17581 (refactor: Remove settings merge reverse precedence code by ryanofsky)
#17580 (refactor: Add ALLOW_LIST flags and enforce usage in CheckArgFlags by ryanofsky)
#17493 (util: Forbid ambiguous multiple assignments in config file by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label CI failed on Mar 31, 2024

murchandamus commented at 7:54 pm on April 1, 2024: contributor

Concept ACK

maflcko commented at 8:25 am on April 2, 2024: member

When resetting a test chain, it is also important to consider the script interpreter coverage of the current chain. Test chains are (usually) the first place to go to, to test new script primitives and protocols, as well as consensus deployments. The existing chain thus serves as a test for consensus implementations, apart from basic unit test vectors. It would be good to think how to preserve the test vectors in the chain. See also #11739 (comment) . Or if it is not needed, it would be good to say so. Maybe https://github.com/bitcoin-core/qa-assets/blob/main/unit_test_data/script_assets_test.json already covers a good portion?

Moreover, testnet is the only public chain where anyone can submit a nonstandard transaction from their laptop. Recall that policy is enforced on all networks equally (see commit e1dc15d69061e69351c72907444e8ded0ac7c88c), so getting a non-mempool transaction into a block is only possible for a miner, or by cooperating with a miner. So if the difficulty hack is removed completely, anyone wishing to submit a transaction would have to go purchase and set up mining hardware, or find a miner willing to accept the transaction. Not saying what is the best approach here, just saying that the effects should be considered and a change be done intentionally.

ajtowns commented at 4:08 pm on April 2, 2024: contributor

Probably we should support tracking both testnet3 and the new testnet4 for some time. Making the new code conditional on a different chain param that’s only set for testnet4 would probably be the easiest way of accomplishing that?

fjahr force-pushed on Apr 6, 2024

fjahr commented at 5:03 pm on April 6, 2024: contributor

Pushed some improvements and addressed some feedback. I am experimenting with some of the proposals from the mailing list and so I added Andres Poelstra’s suggested difficulty adjustment with 6h/1M from here: https://groups.google.com/g/bitcoindev/c/9bL00vRj7OU/m/kFPaQCzmBwAJ

Probably we should support tracking both testnet3 and the new testnet4 for some time. Making the new code conditional on a different chain param that’s only set for testnet4 would probably be the easiest way of accomplishing that?

Updated the code to introduce T4 (-testnet4) and keep T3 in place but give some deprecation warning when it’s used. I guess we would keep this in place for 1 or 2 releases and then remove T3 and then let -testnet run T4.

I am using the Genesis block hash to distinguish between the two testnets. There may be cleaner solutions but I think this is ok since it would be only temporary until T3 is removed.

Moreover, testnet is the only public chain where anyone can submit a nonstandard transaction from their laptop.

Is it really realistic that someone with just their CPU would be able to mine a block with their non-standard tx on the current testnet? If the bug isn’t active currently they would need to wait for it to become active and that could take weeks, right? And when it becomes active I would imagine the miner who found the first block in the difficulty=1 series just blasts the network and the CPU miner still has no chance to get a block in between.

We could revert #28354 for testnet4 if this is a feature that matters to users. Is it too much to ask that people use testmempoolaccept or run their own testnet node with -acceptnonstdtxn=0 when they want to test standardness of their tx? I would say no. And I don’t see another option to solve this on layer 1 if we assume nobody changes the defaults.

It would be good to think how to preserve the test vectors in the chain.

Interesting thought. I think once there is consensus to do T4 we will find a creative solution for this. Cool would be to convert this coverage to fuzzing coverage somehow but I am not sure if that’s realistic or worth the effort. Otherwise, we could write a program that looks at all the different scripts that exist on T3 and replays them on T4 or if we can compress them somehow like by filtering everything that doesn’t add coverage, then we turn it into a unit test that replays the interesting scripts.

fjahr force-pushed on Apr 6, 2024

murchandamus commented at 2:16 pm on April 10, 2024: contributor

Since some people consider the blockstorms an interesting feature of Testnet3, it might be interesting to only raise the difficulty of the delayed block exception to 100,000 instead of 1,000,000. This would allow the network to return to the organic difficulty in fewer difficulty periods and slow down the blockstorms but not remove the feature altogether. My understanding is that this would correspond roughly a tenth of one S9 mining on the network, so if no one had mined for a while, a single S9 could restart the network with ~60 s blocks, but wouldn’t churn out thousands of blocks per second.

Only allowing lower difficulty blocks after 6 hours could easily make testnet useless for extended periods, if someone put several ASICs on testnet for a while, it might prevent other users from getting confirmations for up to 6 hours. I could see an increase from the twenty minute rule to maybe an hour, but more seems counter to why the rule was introduced in the first place.

maflcko commented at 2:44 pm on April 10, 2024: member

Is it really realistic that someone with just their CPU would be able to mine a block

Yes, I am not sure what would be the problem. All you have to do is to set the time +20min and mine a block on your laptop. If you don’t want to try it yourself, you can come by to watch it on my laptop.

Since some people consider the blockstorms an interesting feature of Testnet3

After a quick chat with @murchandamus, an alternative fix would be to require the pre-retarget block to have the “correct” difficulty, so that all retarget periods are organic. The +20min hack would remain to allow a CPU to mine a few blocks, if needed, however, a block storm would be naturally limited by the +120h cut-off rule. This would limit the block storms to small block “gusts”, which seems good enough to make everyone happy?

Sjors commented at 9:18 am on April 15, 2024: member

I spun up seed.testnet4.bitcoin.sprovoost.nl and set it to use the magic bytes and port number.

When running with -debug=net I noticed it “Added hardcoded seed” a bunch of times for i2p and onion, which makes no sense. I’m guessing those are for testnet3.

If anyone wants to deploy a faucet, let me know and I’ll send some coins… unless someone reorgs me.

luke-jr commented at 9:40 pm on April 20, 2024: member

This seems too complicated for a testnet exception IMO. And it breaks the use case of someone testing being able to mine a block on-demand without actual mining hardware.

Shouldn’t it be enough to just fix the timewarp bug?

Sjors commented at 5:11 pm on April 22, 2024: member

it breaks the use case of someone testing being able to mine a block on-demand without actual mining hardware

I doubt many people do that. You can still set nProofOfWorkLimit to a lower value. We could add a code comment for that (or a setting). That way you can mine locally as fast as you want, without causing mayhem for others.

maflcko commented at 5:41 am on April 23, 2024: member

I doubt many people do that.

Two people raised the concern in this thread, so why would you doubt it?

fjahr commented at 2:48 pm on April 23, 2024: contributor

Yes, I am not sure what would be the problem. All you have to do is to set the time +20min and mine a block on your laptop. If you don’t want to try it yourself, you can come by to watch it on my laptop.

I missed that response, so if this is possible at any time with or without a block storm happening, I am not sure how the change here is making a difference? I will give it a try.

fjahr commented at 3:11 pm on April 23, 2024: contributor

I doubt many people do that.

Two people raised the concern in this thread, so why would you doubt it?

“Many” is very relative but I think we probably would not see a market for trading testnet coins against bitcoin if that is something everyone can do as easily as setting a bitcore core node for example.

Sjors commented at 5:01 pm on April 23, 2024: member

I am not sure how the change here is making a difference?

If it depends on the difficulty being 1 rather 1 million, that would make a difference. The two people who brought it up can definitely recompile, but maybe there’s a better solution - maybe just a startup flag to override the minimum difficulty?

maflcko commented at 5:07 pm on April 23, 2024: member

maybe just a startup flag to override the minimum difficulty?

I don’t think consensus rules of remote nodes can be affected by a local startup flag (or re-compilation).

If someone wanted to create a block locally only, they could use regtest.

fjahr force-pushed on May 2, 2024

fjahr renamed this:
~~[DO NOT MERGE] testnet4 including PoW difficulty adjustment fix~~
Testnet4 including PoW difficulty adjustment fix
on May 2, 2024

fjahr force-pushed on May 2, 2024

in src/pow.cpp:28 in 762c5b28d9 outdated

19@@ -20,10 +20,25 @@ unsigned int GetNextWorkRequired(const CBlockIndex* pindexLast, const CBlockHead
20     {
21         if (params.fPowAllowMinDifficultyBlocks)
22         {
23+            uint8_t target_spacing_multiplier{2};
24+
25+            // Testnet4
26+            if (params.hashGenesisBlock == uint256S("0x000000008d6faa98083fa55742aa82d4ed249bd1bfc3239c706e0a61ef9e3931")) {
27+                // Reset limit of Testnet4 is just above difficulty 1M.
28+                nProofOfWorkLimit = 0x1b0010c6U;

jlopp commented at 2:39 pm on May 2, 2024:

What’s the reasoning for raising the minimum difficulty to 1 million? This will effectively prevent a developer from being able to move the chain forward without an ASIC.

I’d think that closing the timewarp loophole + the difficulty adjustment loophole should be sufficient to prevent block storms without actually making it more difficult for a dev to mint a low cost block if the chain has stalled.

Sjors commented at 3:40 pm on May 2, 2024:

I’d think that closing the timewarp loophole + the difficulty adjustment loophole should be sufficient to prevent block storms without actually making it more difficult for a dev to mint a low cost block if the chain has stalled.

If that’s true then that’s fine. It seems multiple people like this ability, see @maflcko’s comments above. I’m still a bit confused about the intersection of bugs / features here.

fjahr commented at 3:48 pm on May 2, 2024:

It was suggested on the mailing list and it seemed to get some interest, so I added it here. But it has become clear in the discussion here that it seems to be pretty controversial so I will remove it here shortly.

Maybe there is a lower number that is not 1 that most people are happy with like @murchandamus suggested and I will be happy to add it back in that case but while that is being bikeshedded, I think it’s better to remove it.

Sjors commented at 8:50 am on May 3, 2024:

By the way, even if we do implement a higher minimum difficulty (i.e. lower maximum target), imo this should be done in chainparams.cpp by setting consensus.powLimit. If that makes it harder to grind a genesis block, perhaps that’s a good reason not to change it.

fjahr force-pushed on May 2, 2024

DrahtBot removed the label CI failed on May 2, 2024

jlopp commented at 4:41 pm on May 2, 2024: contributor

In keeping with genesis block tradition I’d propose the following coinbase string referencing an article published today regarding testnet: “CCN 02/May/2024 Bitcoin Testnet Could Need Reset”

0./generate-genesis -timestamp 1714658825 -pubkey 03752c999db32c08a6d62550a1ad83cf1731f35ac27096c27dfca5127c0c6dbd9b -psz "CCN 02/May/2024 Bitcoin Testnet Could Need Reset" -nonce 1
1Ctrl Hash:	0x000000004693a387d558bbf7e321f6915e82d1afac4db25ee40f44bb783e802b
2Target:		0x00000000ffff0000000000000000000000000000000000000000000000000000
3Blk Hash:	0xb9aae9d9ee85b72c1daa940ab4a40175e7d5ed845c0b345153e4b72b5d3cc9ff
4Mkl Hash:	0xab42b22e295fcd6ffd719a0d647661ab12c725a51725b725f6ac83dfe3f545fa
5Nonce:		119141657
6Timestamp:	1714658825
7Pubkey:		03752c999db32c08a6d62550a1ad83cf1731f35ac27096c27dfca5127c0c6dbd9b
8Psz:		'CCN 02/May/2024 Bitcoin Testnet Could Need Reset'

jlopp commented at 4:45 pm on May 2, 2024: contributor

Only allowing lower difficulty blocks after 6 hours could easily make testnet useless for extended periods, if someone put several ASICs on testnet for a while, it might prevent other users from getting confirmations for up to 6 hours. I could see an increase from the twenty minute rule to maybe an hour, but more seems counter to why the rule was introduced in the first place.

6 hours seems overkill to me. I’d think 3 hours would suffice, since it would mean that at worst someone would be able to generate a block 1 hour after the previous block if they also warped the time ahead by the max 2 hours.

murchandamus commented at 6:25 pm on May 2, 2024: contributor

6 hours seems overkill to me. I’d think 3 hours would suffice, since it would mean that at worst someone would be able to generate a block 1 hour after the previous block if they also warped the time ahead by the max 2 hours.

Three hours seems too long to me. That would mean that the first use of the low-difficulty loophole would require one hour of wait (by also setting your clock two hours to the future), but any subsequent block would be delayed by three hours. It would be trivial for anyone with a modicum of hashpower to disrupt liveness of testnet for any hashpowerless developers.

Considering @jlopp’s demonstration of the downsides of the low-difficulty loophole, and mulling it over a bit more, I would now prefer:

Retain the 20-minute difficulty exception This would permit developers to author a new block with at most reasonable delay. While nobody else is doing so, it would be possible to author a new block almost immediately, if someone is already making use of the exception, a wait of twenty minutes would be sufficient to allow another block.
Prevent use of the 20-minute difficulty exception on the last block in a difficulty period This would mitigate the blockstorms to only short block gusts: if no one else is making use of the exception, a user would be able to issue around six blocks almost instantaneously, but no more.
Do not increase the minimum difficulty Restricting authoring of new block to owners of ASIC equipment just seems unnecessary, if the difficulty reset to minimum difficulty is mitigated.

Obviously, the testnet would still be vulnerable to the timewarp bug, so perhaps testnet4 should include a variant of the proposed rule from the Great Consensus Cleanup that would require the first block of a difficulty period to be no more than two hours less than the previous difficulty period’s last block’s timestamp.

Alternatively, I would be satisfied if using the 20-minute difficulty exception would be changed such that nBits could remain at the proper value, and were available for any block in the difficulty period without incurring the difficulty resetting to the minimum. However, my understanding is that the difficulty check in block validation is context independent, and therefore a block would fail if the hash were higher than the nBits value in its own header.

fjahr force-pushed on May 2, 2024

fjahr commented at 11:46 pm on May 2, 2024: contributor

I had to rebase to fix a silent merge conflict and I think I managed to make the CI happy with the tests I added. In general, I think this can be moved out of draft status now since we are discussing the details of the changes in parameters but overall there seems to be no opposition to the change. I initially had removed Testnet3 as part of this already and keeping it in place makes this much less controversial/dangerous.

I have removed the adjustments on min difficulty and changed the exception period back to 20 min. I think keeping this as is seems to be the path that has the most support but if there is consensus on a mild adjustment of the exception period of course I can add this again. Given what was discussed maybe we can agree on 30 min or 1 h but if it’s unclear that this is really needed we should probably not change too many things at once.

I’d propose the following coinbase string referencing an article published today regarding testnet: “CCN 02/May/2024 Bitcoin Testnet Could Need Reset”

Happy to add this, I considered my genesis block just a placeholder anyway. The only reason we might use something different is if review here drags on for a few months and people would prefer something fresher by then. I will add it in my next push unless other reviewers disagree with the choice.

Prevent use of the 20-minute difficulty exception on the last block in a difficulty period @murchandamus do you explicitly prefer this over my proposed fix? I find it more elegant and intuitive if we allow the use on all blocks and prevent the adjustment from being affected by it.

perhaps testnet4 should include a variant of the proposed rule from the Great Consensus Cleanup that would require the first block of a difficulty period to be no more than two hours less than the previous difficulty period’s last block’s timestamp.

Interesting, I will try to see how easy it would be to do this.

Alternatively, I would be satisfied if using the 20-minute difficulty exception would be changed such that nBits could remain at the proper value, and were available for any block in the difficulty period without incurring the difficulty resetting to the minimum. However, my understanding is that the difficulty check in block validation is context independent, and therefore a block would fail if the hash were higher than the nBits value in its own header.

I am not sure I understand this: you are talking about the fix I have proposed here, right? If that is so, I am not sure I understand why the header would diverge from the block on the nBits though.

fjahr marked this as ready for review on May 2, 2024

fjahr force-pushed on May 3, 2024

fjahr commented at 1:07 am on May 3, 2024: contributor

perhaps testnet4 should include a variant of the proposed rule from the Great Consensus Cleanup that would require the first block of a difficulty period to be no more than two hours less than the previous difficulty period’s last block’s timestamp.

I have added the corresponding code from #15482 in a separate commit as a basis for discussion. Link to original code: https://github.com/bitcoin/bitcoin/pull/15482/commits/1f63030817c6a42d19c10bd16cb8fff700602081#diff-97c3a52bc5fad452d82670a7fd291800bae20c7bc35bb82686c2c0a4ea7b5b98R3251

ajtowns commented at 4:03 am on May 3, 2024: contributor

Prevent use of the 20-minute difficulty exception on the last block in a difficulty period

I think the worst case here is that someone with significant hashpower quickly mines a few retarget periods to get the difficulty to increase a lot, then stops mining, and no one else is around with enough hashpower who’s willing to mine a block.

With the rule above, that means we could get to the end of the period, but would then stall, as the last block would be expensive. With the rule in the PR we have, we’d instead probably get a full retarget period of min difficulty blocks, ~~and then drop to min difficulty and still have a block storm~~. [EDIT: with this patch, the 0’th block of each retarget period doesn’t have the min difficulty exception, and the difficulty adjustment will always work its way back to this block if it doesn’t find a higher than min difficulty block elsewhere in the retarget period, so testnet4 might stall on the 0’th block until an ASIC is pointed at it, but this won’t trigger a block storm]

Would it maybe be feasible to require min difficulty blocks to record the real difficulty in the nVersion field, and use that to calculate the expected difficulty for the next retarget period directly without having to search for a non-min-difficulty header? I think that wouldn’t even conflict with versionbits signalling (though min difficulty blocks would not be able to signal for activation, but that seems fine).

emsit commented at 4:53 am on May 3, 2024: none

I’d propose the following coinbase string referencing an article published today regarding testnet: “CCN 02/May/2024 Bitcoin Testnet Could Need Reset”

Happy to add this, I considered my genesis block just a placeholder anyway. The only reason we might use something different is if review here drags on for a few months and people would prefer something fresher by then. I will add it in my next push unless other reviewers disagree with the choice.

Is that a joke? A person who disrupted the use of testnet3 with their ‘Griefing Attack’ is now advertising in this manner? If testnet4 is not meant to be taken seriously, then I suggest:

“The best place to buy testnet4 coins is altquick.com”

Sjors commented at 7:43 am on May 3, 2024: member

The genesis block title by @jlopp is great. It refers to a failure of the previous thing. If people start to value testnet4 coins to the extend where its genesis block coinbase message is considered precious advertising space, then we need to reset it again.

Where does the public key come from?

require min difficulty blocks to record the real difficulty in the nVersion field, and use that to calculate the expected difficulty for the next retarget period directly without having to search for a non-min-difficulty header

I’d be curious to see what that patch look like.

a variant of the proposed rule from the Great Consensus Cleanup that would require the first block of a difficulty period to be no more than two hours less than the previous difficulty period’s last block’s timestamp

I suppose it’s fine to use the launch of testnet4 to test a fix we have in mind for mainnet later. But I see two potential (minor?) issues:

We lose the ability to cleanly test the great consensus cleanup soft fork in a trivial manner. Since part of the proposed rules is already active on testnet4.
What if someone comes up with a different but incompatible timewarp attack fix? I guess we can always just reset testnet again worst case, or apply the new rule retroactively and cause a deep reorg on testnet4.

I’m still trying to wrap my head around the block storm issue and the various fixes proposed above, will do some thinking.

in src/kernel/chainparams.cpp:342 in 4e8b29ae23 outdated

337+        assert(genesis.hashMerkleRoot == uint256S("0x4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b"));
338+
339+        vFixedSeeds.clear();
340+        vSeeds.clear();
341+        // nodes with support for servicebits filtering should be at the top
342+        // TODO: Add Testnet4 seeds

Sjors commented at 8:07 am on May 3, 2024:

4e8b29ae234a05181ff9fd64ab3a74ba997e2eac: you can add seed.testnet4.bitcoin.sprovoost.nl here. I’ll update it if the magic bytes change.

fjahr commented at 4:19 pm on May 3, 2024:

Added, sorry I forgot about that.

in src/chainparamsbase.cpp:48 in 4e8b29ae23 outdated

43@@ -43,6 +44,8 @@ std::unique_ptr<CBaseChainParams> CreateBaseChainParams(const ChainType chain)
44         return std::make_unique<CBaseChainParams>("", 8332, 8334);
45     case ChainType::TESTNET:
46         return std::make_unique<CBaseChainParams>("testnet3", 18332, 18334);
47+    case ChainType::TESTNET4:
48+        return std::make_unique<CBaseChainParams>("testnet4", 48332, 48334);

Sjors commented at 8:55 am on May 3, 2024:

4e8b29ae234a05181ff9fd64ab3a74ba997e2eac: in order to not run out of ports by testnet6, testnet5 should probably use ports 1833x again.

jlopp commented at 7:35 pm on June 30, 2024:

Ditto

Sjors commented at 9:56 am on May 3, 2024: member

Partial code review: the first commit 4e8b29ae234a05181ff9fd64ab3a74ba997e2eac looks good to me (of course it would partially change if we get a new genesis block).

I thought a bit more about block storms…

The only (relevant) thing that distinguishes testnet3 from mainnet is fPowAllowMinDifficultyBlocks. In our miner.cpp code we potentially lower pblock->nBits of the block we’re trying to mine once enough time has passed. This works because both the miner and validation rely on GetNextWorkRequired to tell us what the difficulty of a given block needs to be.

Because we “mess with” nBits itself, the code that handles difficulty adjustment (also GetNextWorkRequired, at the bottom) can be tricked. It takes the timestamp of the first block in the retarget period and passes it to CalculateNextWorkRequired along with the last block of retarget period (don’t worry about off-by-one here). The low difficulty blocks probably came in fast, so difficulty goes up by a factor 4. But that factor 4 is applied to the last nbits value, which is the minimum difficulty.

And so we have a situation where difficulty has to go all the way from 1 to something matching the actual hash power.

One way to fix that is, as discussed earlier, is to simply increase the minimum difficulty (maximum target) to something that requires a small ASIC. That would make block storms very expensive, but completely removes the ability to CPU mine. @fjahr’s earlier code (https://github.com/bitcoin/bitcoin/pull/29775#pullrequestreview-2035943330) did not change consensus.powLimit but instead changed GetNextWorkRequired to enforce this. As a consequence the initial blocks after genesis could have difficulty 1, but once the difficulty goes above a typical S9, it can no longer drop below that. That seems confusing (in addition to the above).

One can always find a friend with an S9 space heater to mine a block, if this is really the only way to prevent block storms, but another solution would be better.

So we want:

a way to CPU mine at very low difficulty, if nobody else mines a block within reasonable time
not (dramatically) reduce the difficulty (a gradual decrease does seem desirable, so normal mining can continue with whatever hash power is structurally available)

I think the key to that is to ensure difficulty adjustment is based on the last real-work nBits value.

The conceptually easiest thing to do imo is to stop lowering the nBits value, but instead simply ignore the nBits value for blocks that are 2 x 10 minutes in the future. That way the difficulty adjustment calculation is not affected anymore.

Unfortunately that allows anyone to override the most proof-of-work chain by producing blocks 20.01 minutes apart. That ability is capped by the 2-hour-future rule, but it still makes ASICs basically useless.

So then we get to @fjahr’s proposed solution in 598f563687280fa6832ac42b9dc97c3ae7f7581d. We keep the low nBits value in blocks, but when calculating the difficulty for the next retarget period, we look at the last “real” nBits value. @ajtowns’s suggestion to use nVersion could make sense if performance is a big issue, but otherwise I prefer the simplicity of @fjahr’s approach.

This still allows for ~2000 difficulty 1 blocks in a given retarget period, since anyone can just set their clock 20 minutes in the future at the start of each retarget period. It wouldn’t be a storm, but it would seem to make it fairly useless to point an ASIC at testnet.

These low difficulty blocks can be easily reorged by an ASIC, since they have very little total work. But in practice that requires them to invalidate the first block in such a series, because by default we produce block templates on our tip, even if it’s “brittle” tip.

This last problem could be dealt with by changing getblocktemplate to build an empty block on top of the last non-special-min-difficulty-rules-block. (by default, and while also keeping the bitcoin-cli -generate behaviour of building on the tip)

(I haven’t thought about how the time warp bug interacts with any of this)

ajtowns commented at 12:53 pm on May 3, 2024: contributor

I’d be curious to see what that patch look like.

Something along the lines of https://github.com/bitcoin/bitcoin/commit/970c0c31e99bebe2b60ade930c96600d1d82f7ca maybe? Essentially untested.

The main advantage of this is that block 0 in a retarget period could also be min difficulty, allowing you to keep mining min-difficulty blocks with 20 minutes between them, halving the difficulty every 4 weeks / 2016 blocks, until you get to something that the network can mine on top of. The disadvantage is it’s more intrusive.

Sjors commented at 1:35 pm on May 3, 2024: member

Here’s a commit that makes getblocktemplate reorg low difficulty testnet4 blocks: https://github.com/Sjors/bitcoin/commit/2125fc56163ceaddfadbc78ad0d0da5b1a99a8fa It adds some complexity, but not in the validation code. I wonder if it would be useful on testnet3.

Untested, other than the the template looks correct to me. I still need to figure out an easy way to setup a local stratum v1 pool (to CPU mine).

By the way, I noticed that src/bitcoin-cli -testnet4 -generate 1 1000000000 does not find a block if called a dozen times, even at difficulty 1. I’m not sure if that’s expected because the mining code is inefficient (single CPU thread) or if there’s a bug. I worked around it by temporarily setting the minimum difficulty as low as on retest.

fjahr force-pushed on May 3, 2024

fjahr commented at 4:18 pm on May 3, 2024: contributor

@fjahr’s earlier code (https://github.com/bitcoin/bitcoin/pull/29775#pullrequestreview-2035943330) did not change consensus.powLimit but instead changed GetNextWorkRequired to enforce this. As a consequence the initial blocks after genesis could have difficulty 1, but once the difficulty goes above a typical S9, it can no longer drop below that.

I did implement an exception to allow the difficulty to drop below the adjustment (1M in this case), see my comment on this in the old commit: https://github.com/bitcoin/bitcoin/commit/515e3a9b94675a42b8e9635a6ceb8ea26c30e4d7#diff-4667f00c3a075be2753aa6ebdeea4bdbb66ef6e0b3d6df313a430f2eb8669ffdR32 But yeah, if we change it using powLimit would make more sense, but it was a proof of concept and I would have needed to find someone to mine a new Genesis block for me just to keep this PR testable because I don’t have S9s lying around unfortunately :)

Something along the lines of 970c0c3 maybe? Essentially untested.

The main advantage of this is that block 0 in a retarget period could also be min difficulty, allowing you to keep mining min-difficulty blocks with 20 minutes between them, halving the difficulty every 4 weeks / 2016 blocks, until you get to something that the network can mine on top of. The disadvantage is it’s more intrusive.

Aside from it being intrusive in our code, the miners who are using their own custom software and want to test it in Testnet would need to implement it as well. But I am not sure if that is significant.

Maybe a more important question: What prevents someone malicious from writing an absurd number into the Version field? I guess to verify the number is actually correct worst case we would need to look further than our current difficulty adjustment period if it’s all 20min blocks so it might be easier if we just do that right away? I.e. we would allow the 20min rule on the 0th block but to get the real difficulty we look back as far as we need to find the last real difficulty block and then apply the intermediate adjustments on that number if there were any between the current block and the last real difficulty block. Still thinking this through further though…

EDIT: Either way this could become an issue for pruned nodes in an extreme case.

Here’s a commit that makes getblocktemplate reorg low difficulty testnet4 blocks: https://github.com/Sjors/bitcoin/commit/2125fc56163ceaddfadbc78ad0d0da5b1a99a8fa It adds some complexity, but not in the validation code. I wonder if it would be useful on testnet3.

I think that’s interesting. Maybe that is a helpful insight: if we can not find the perfect solution with a simple validation code adjustment, having miners reorg the min-difficulty blocks could make this attack a lot less feasible. I will think about it some more. However, I think those that use the 20min rule for mining their fancy non-standard transactions etc. might not like this because they would get reorged as well. Maybe we could just reorg the last block, not all min-diff blocks last seen. That would mean those that need this “feature” could have their fancy tx included in the blockchain if they mine one more min-diff block on top of it.

jlopp commented at 5:38 pm on May 3, 2024: contributor

Where does the public key come from? @Sjors I just grabbed a random public key from one of my testnet wallets; unclear if there’s any significance to the public key used for the genesis block.

By the way, I noticed that src/bitcoin-cli -testnet4 -generate 1 1000000000 does not find a block if called a dozen times, even at difficulty 1. I’m not sure if that’s expected because the mining code is inefficient (single CPU thread) or if there’s a bug. I worked around it by temporarily setting the minimum difficulty as low as on retest.

On a related note, my initial attempt at triggering low difficulty blocks involved me using a bash script to update my computer’s system time ahead by 20 minutes and then trying to generate a block. For whatever reason, Bitcoin Core did not seem to notice that the system time was far enough ahead that the difficulty only needed to be 1 and it kept generating high difficulty block templates. So I ended up having to modify my local node to always produce blocks with timestamps 20 minutes after the previous block.

Sjors commented at 6:07 pm on May 3, 2024: member

I don’t have S9s lying around unfortunately

I do, special discount for genesis blocks :-)

use the 20min rule for mining their fancy non-standard transactions etc

When a miner reorgs a block, do non-standard transactions end up in its mempool or are they forgotten? The latter is probably safest, but we could allow it on testnet for this use case.

We made acceptnonstdtxn=0 the default for testnet3 in #28354. We could flip it back on. Or we could make it easier to preferentially peer with no-standardness nodes (and miners) to get those in. I don’t think relying on low difficulty blocks is the right solution, at least I don’t think it’s a good enough reason to keep them around.

unclear if there’s any significance to the public key used for the genesis block.

There shouldn’t be, but it could lead to drama if people see it as a donation address. It would be better to use a provably unspendable public key, use one of the BIP32 test vectors or burn the coins with OP_RETURN. Though maybe not worth making a new block if it was ton of work.

For whatever reason, Bitcoin Core did not seem to notice that the system time was far enough ahead

Does your node still have network adjusted time? https://bitcoinops.org/en/newsletters/2024/02/07/#bitcoin-core-28956 Or is it something about median time past?

jlopp commented at 6:34 pm on May 3, 2024: contributor

Does your node still have network adjusted time?

That could explain it. IIRC I was originally using v25.

I can generate a genesis block in about 15 minutes so no biggie to try variations. Here’s an alternate genesis block with a zeroed out pubkey:

0./generate-genesis -timestamp 1714658825 -pubkey 000000000000000000000000000000000000000000000000000000000000000000 -psz "CCN 02/May/2024 Bitcoin Testnet Could Need Reset After 13 Years"
1Ctrl Hash:	0x000000005611bbf1419574641c1d880d2872645ec62eb2efd605fab98d0760d2
2Target:		0x00000000ffff0000000000000000000000000000000000000000000000000000
3Blk Hash:	0x372e42462a9898504ccd14c037bf98436289ba1ef27088433aad521fca09bf21
4Mkl Hash:	0xa2927b35a7444dd3e3ef24145a8c7054a4810c88d8cbed3896adfecae832b6fe
5Nonce:		836015000
6Timestamp:	1714658825
7Pubkey:		000000000000000000000000000000000000000000000000000000000000000000
8Coins:		5000000000
9Psz:		'CCN 02/May/2024 Bitcoin Testnet Could Need Reset After 13 Years'

fjahr commented at 1:52 am on May 4, 2024: contributor

We made acceptnonstdtxn=0 the default for testnet3 in #28354. We could flip it back on. Or we could make it easier to preferentially peer with no-standardness nodes (and miners) to get those in. I don’t think relying on low difficulty blocks is the right solution, at least I don’t think it’s a good enough reason to keep them around.

Yeah, I tend to agree, I thought about flipping it back on earlier as well: #29775 (comment) but we weren’t very far in the discussion yet. By now we are discussing pretty intrusive changes with the goal to prevent block storms but at the same time keep the 20min exploitable for this one good use case. These are obviously at odds. The motivation stated in #28354 was to align testnet behavior with the behavior of other chains but maybe compared to the other solutions this is the simplest solution to the problem. In terms of simplicity and from a maintenance perspective I think setting minPow to 1M and acceptnonstdtxn=1 as default is still a good solution that we should keep it in mind as an alternative to the other proposed solutions. There we also have two things that are at odds: developers that want to get their non-standard tx into a block easily and developers that want to test if their tx would get into a block on mainnet (the case that triggered #28354). But this seems to me like something that can be fixed with better documentation and/or some variant of preferential peering? At least both of these user groups are not trying to be malicious and can be assumed to have some level of technical expertise.

When a miner reorgs a block, do non-standard transactions end up in its mempool or are they forgotten? The latter is probably safest, but we could allow it on testnet for this use case.

I haven’t had time to look at the code but I would be surprised if the acceptnonstdtxn flag is ignored in this case. Making this a testnet specific exception here could be a solution that is a small patch but overall it’s not the simplest concept.

EDIT: Seems like acceptnonstdtxn is still enforced on the txs from reorged blocks. The flag sets m_require_standard on the mempool, this is then used in PreChecks(). That function is used in AcceptSingleTransaction() which is then used in MaybeUpdateMempoolForReorg() via AcceptToMemoryPool().

ajtowns commented at 4:27 am on May 4, 2024: contributor

and acceptnonstdtxn=1 as default is still a good solution

We’ve had a case where having this setting as default rendered $76M of bitcoin inaccessible for months, so that doesn’t seem like a good solution to me. If you’re testing something that will require miners with meaningful hashpower running non-default things on mainnet, finding a miner who’ll do the same thing on testnet first doesn’t seem that unreasonable to me. You can always test on regtest or a custom signet as well, after all.

As far as CCN 02/May/2024 Bitcoin Testnet Could Need Reset After 13 Years goes, the point of putting a news item in the genesis block was to prevent the possibility of pre-mining. Doing that with something referring to the project itself is exploitable: all you need to do is make a deal in advance with CCN that on May 2nd they’ll post an article with that title, and you can start mining in advance. For testnet that perhaps doesn’t matter, but if it doesn’t there’s no reason to change the commitment in the first place. Where it does matter is if people take our example and reuse it in an altcoin where premining would be a problem – but there’s an easy solution there: simply use the latest bitcoin block hash.

maflcko commented at 8:42 am on May 4, 2024: member

and acceptnonstdtxn=1 as default is still a good solution

Not sure. acceptnonstdtxn is limited in what it allows. There are many non-standard examples that are not even exposed as a setting.

fjahr commented at 4:53 pm on May 4, 2024: contributor

I will try to summarize a bit where I think we stand on the issues and ideas being discussed. Note, I am biased to keep things out that didn’t get strong support, if we don’t fix all the holes and things go wrong again, there is always Testnet5 and we will have more practice with it after we are done with Testnet4.

On preventing block storms while allowing developers to get their non standard txs into blocks
- Fixing the difficulty adjustment bug: Both solutions (disallowing min-difficulty on the last block and looking backward if the last block had min-difficulty) seem to have sufficient support to be included
- Increasing minimum difficulty (via minPow): 1M received rejections, 100k as @murchandamus suggested at some point didn’t receive feedback so I would consider it uninteresting to reviewers
- Increasing delay time to allow min-difficulty from 20min: 6h and 3h received rejections, my suggestions for 30min or 1h didn’t receive feedback so I consider there is no interest currently
- Adding a fix to the time-warp bug as added recently in the last commit: Didn’t get feedback since I added it, it sounded to me like @Sjors who suggested it hadn’t made up his mind about it either. I tend to remove it again because I like to keep things simple unless there is more support to add it.
- Re-enabling acceptnonstdtxn received rejections as an alternative solution to get non-standard txs into blocks outside of min-difficulty blocks
- @Sjors suggestion to nudge miners to reorg min-difficulty blocks didn’t see much engagement other than me thinking that it’s an interesting idea but it also has wider implications that would make Testnet behave very differently than Mainnet and I am not sure if that is worth it
- @ajtowns idea to persist the real difficulty in the version field saw some interest from Sjors and there is some unaddressed feedback from me here: #29775 (comment)
- @murchandamus also suggested earlier basically keeping the nBits intact in the header but ignoring it. It didn’t see much engagement but I think this is essentially a variant of AJ’s suggestion to persist the original difficulty in version. I think it would have the same downsides and I believe it comes down to looking back further than the current period may give us the same behavior with a much simpler code change. That is, if we are willing to ignore the problem this could cause pruned nodes in extreme cases (and I think the same danger is there if the value is persisted somehow but ignored since pruned nodes could just be lied to in that case).
Genesis Block
- @jlopp suggested block received rejections on the content of the message and “self-advertisement” though it wasn’t very clear to me what this was referring to. I think at least we need another variation with the latest block hash in the message.

Please let me know what I got wrong and if I forgot something. As it stands I tend to remove the time-warp fix again and just keep a fix for the difficulty adjustment bug, everything else doesn’t seem to have clear support yet.

emsit commented at 6:31 pm on May 4, 2024: none

Genesis Block

@jlopp suggested block received rejections on the content of the message and “self-advertisement” though it wasn’t very clear to me what this was referring to. I think at least we need another variation with the latest block hash in the message.

I came across the fact that Lopp was conducting a ‘Griefing Attack’ on testnet3 and wants to embed his message into the genesis block, which refers to this article:

https://www.ccn.com/news/crypto/bitcoin-testnet-could-need-reset-13-years-jameson-lopp-griefing/

I believe he is so biased against testnet3 that he decided to inconvenience its use for all testnet users. And frankly, I don’t understand how it would combat the trading that bothers him the most. I believe that someone who considers themselves a Bitcoin developer is not in a position to afford such behavior, let alone boast about it. I admit that I am also biased towards testnet3, as I have been operating https://coinfaucet.eu/ for 10 years and have distributed a total of 328,065.41 tBTC. So testnet is close to me, but I accept its successor. I’m just sorry that testnet4 probably won’t address the biggest problem that bothers you, trading. It should be noted that the ‘Griefing Attack’ is still an attack, and its aim was to make it difficult for people to use testnet3. There is no justification for it, regardless of how well-intentioned it may have been. Testnet3 has been here for 13 years and has functioned well despite the ‘Block Storms’ bug… Therefore, I propose that testnet4 be approached objectively and impartially, and that the genesis block not contain a message related to recent events surrounding testnet.

fjahr commented at 3:23 am on May 5, 2024: contributor

Thank you for giving more detail @emsit !

I admit that I am also biased towards testnet3, as I have been operating https://coinfaucet.eu/ for 10 years and have distributed a total of 328,065.41 tBTC. So testnet is close to me, but I accept its successor.

Thank you!

I’m just sorry that testnet4 probably won’t address the biggest problem that bothers you, trading.

I think the act of resetting is addressing it. If it’s sufficiently addressing it or if we need to do something more (like regularly scheduled resets etc.) we will only know once we have tried and I haven’t seen any other proposal to address this issue, nor do I think there is anything else we can do that would work unless we turn this into some variant of a signet.

It should be noted that the ‘Griefing Attack’ is still an attack, and its aim was to make it difficult for people to use testnet3. There is no justification for it, regardless of how well-intentioned it may have been. Testnet3 has been here for 13 years and has functioned well despite the ‘Block Storms’ bug…

I agree that it’s not great to grief the legitimate users of Testnet3 and we would probably have been able to get Testnet4 on the way without it.

I propose that testnet4 be approached objectively and impartially, and that the genesis block not contain a message related to recent events surrounding testnet.

While I may fully not agree with @jlopp ’s method, I very much agree with his goal which is to show that testnets in general should be completely meaningless and without value. This also means that I don’t care so much that he proposes to embed the article that interviews him, because I think we all agree (especially @jlopp ) that it’s embedded into something meaningless and without value. In the end, the headline of all of this is: we shouldn’t be passionate about any aspect of testnet at all, it should just serve its purpose. @jlopp has suggested a full Genesis block and mined it too, if he addresses the concern and adds a recent block hash to the message I will use it unless there is someone else here who also provides a full Genesis block that is mined with a different message that reviewers prefer over @jlopp s one. I simply don’t want to mine the Genesis block myself over and over again as the message gets bikeshedded.

ajtowns commented at 4:36 am on May 5, 2024: contributor

What prevents someone malicious from writing an absurd number into the Version field?

Same thing that prevents you from changing the nBits randomly – your block is invalid if you do. See https://github.com/bitcoin/bitcoin/commit/970c0c31e99bebe2b60ade930c96600d1d82f7ca#diff-4667f00c3a075be2753aa6ebdeea4bdbb66ef6e0b3d6df313a430f2eb8669ffdR47

I guess to verify the number is actually correct worst case we would need to look further than our current difficulty adjustment period if it’s all 20min blocks

The rules would be:

given a block A and its parent P, real_bits(A) = (A.time - P.time >= 20 minutes ? A.nVersion : A.nBits)
if A isn’t the first block in a retarget period, real_bits(A) = real_bits(P)
if A is the first block in a retarget period, real_bits(A) = x * real_bits(P) where x is the usual factor

That means you only ever need to look at the two preceding blocks (you need to know if they were 20 minutes apart, as to whether nVersion or nBits is what you look at) to calculate the current block’s target.

emsit commented at 5:11 am on May 5, 2024: none

@jlopp has suggested a full Genesis block and mined it too, if he addresses the concern and adds a recent block hash to the message I will use it unless there is someone else here who also provides a full Genesis block that is mined with a different message that reviewers prefer over @jlopp s one. I simply don’t want to mine the Genesis block myself over and over again as the message gets bikeshedded.

Does the genesis block need a message? It seems like a neutral stance to me.

jlopp commented at 12:51 pm on May 5, 2024: contributor

The genesis block message is just for fun, since almost nobody will ever see it. There were half a dozen articles written about testnet recently, however the CCN article was the only one with a relevant title that mentioned resetting testnet. I don’t think the “strength” of the message to prevent pre-mining really matters if we agree that the primary goal is for testnet tokens to not have value.

The goal of my griefing was to bring attention to this issue, and I dare say that it worked. We’re here to discuss looking forward to the future with testnet4; the future of testnet3 should be irrelevant. Will testnet4 manage to accrue value? There’s only one way to find out - though I seriously doubt that people are going to put money into a network that starts having a history of regularly being abandoned.

I also hope that we can standardize the process of resetting testnet so that it’s easier to do so in the future. As I mention in my essay, I believe the current state of testnet is a culmination of both technical issues (block storms) and cultural issues (apathy about putting in the work for a reset.)

I’m happy to continue generating as many genesis blocks as is required. Here’s a new one with a bitcoin block hash:

0./generate-genesis -psz "03/May/2024 000000000000000000001ebd58c244970b3aa9d783bb001011fbe8ea8e98e00e" -timestamp 1714777860 -pubkey 000000000000000000000000000000000000000000000000000000000000000000
1Ctrl Hash:	0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043
2Target:		0x00000000ffff0000000000000000000000000000000000000000000000000000
3Blk Hash:	0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043
4Mkl Hash:	0x7aa0a7ae1e223414cb807e40cd57e667b718e42aaf9306db9102fe28912b7b4e
5Nonce:		393743547
6Timestamp:	1714777860
7Pubkey:		000000000000000000000000000000000000000000000000000000000000000000
8Coins:		5000000000
9Psz:		'03/May/2024 000000000000000000001ebd58c244970b3aa9d783bb001011fbe8ea8e98e00e'

Or a longer one:

 0./generate-genesis -psz "03/May/2024 Bitcoin block 000000000000000000001ebd58c244970b3aa9d783bb001011fbe8ea8e98e00e" -timestamp 1714777860 -pubkey 000000000000000000000000000000000000000000000000000000000000000000
 1
 2Ctrl Hash:      0x00000000f3c23edbdaabb23bab7598874aed89a515ef182c121f7469c9b8b05e
 3Target:         0x00000000ffff0000000000000000000000000000000000000000000000000000
 4Blk Hash:       0x00000000f3c23edbdaabb23bab7598874aed89a515ef182c121f7469c9b8b05e
 5Mkl Hash:       0x44c6a56ff59c8016031e3be56c4c9d60fa95086a7803a454f7b507084d0c6e12
 6Nonce:          3185950820
 7Timestamp:      1714777866
 8Pubkey:         000000000000000000000000000000000000000000000000000000000000000000
 9Coins:          5000000000
10Psz:            '03/May/2024 Bitcoin block 000000000000000000001ebd58c244970b3aa9d783bb001011fbe8ea8e98e00e'

emsit commented at 5:50 pm on May 5, 2024: none

With the tool you’re using, it’s possible to generate an empty block (with a gap), and after removing the condition in the code, even with null (I haven’t studied further issues if psz was empty, the block was successfully generated.) Or would it be possible to write only the testnet4 release date in Psz?

0./generate-genesis -timestamp $(date +%s) -pubkey 000000000000000000000000000000000000000000000000000000000000000000 -psz " "
1Ctrl Hash:	0x00000000320b39907d50bfd4e60d1b767ffbf403ec20b5c4fb0b5726ab802525
2Target:		0x00000000ffff0000000000000000000000000000000000000000000000000000
3Blk Hash:	0x00000000320b39907d50bfd4e60d1b767ffbf403ec20b5c4fb0b5726ab802525
4Mkl Hash:	0xe52d218265343e7bfd1f2cca47e39708a201b4b272e027c70d7e654dcdcc60ba
5Nonce:		3232400823
6Timestamp:	1714922250
7Pubkey:		000000000000000000000000000000000000000000000000000000000000000000
8Coins:		5000000000
9Psz:		' '

0./generate-genesis -timestamp $(date +%s) -pubkey 000000000000000000000000000000000000000000000000000000000000000000 -psz ""
1Ctrl Hash:	0x000000004b8dcf71e8160eaac5d964dc7940cd30b741df09d444d5224c5910c3
2Target:		0x00000000ffff0000000000000000000000000000000000000000000000000000
3Blk Hash:	0x000000004b8dcf71e8160eaac5d964dc7940cd30b741df09d444d5224c5910c3
4Mkl Hash:	0xb3f9a8c19683a9bb0475e37b13a77ed3051965c642da2aef093c7fe3eb08e3bc
5Nonce:		3958254383
6Timestamp:	1714928680
7Pubkey:		000000000000000000000000000000000000000000000000000000000000000000
8Coins:		5000000000
9Psz:		''

fjahr force-pushed on May 6, 2024

fjahr commented at 2:53 am on May 6, 2024: contributor

I’m happy to continue generating as many genesis blocks as is required. Here’s a new one with a bitcoin block hash

I have included this one in the latest push, thank you!

Does the genesis block need a message? It seems like a neutral stance to me.

It could be empty but as @ajtowns has expressed here, to prevent a long pre-mine a current block hash should be included and that’s not the first time I have heard this opinion. So I think just having the block hash and date in the message is as neutral as it gets.

ajtowns commented at 5:17 am on May 6, 2024: contributor

nudge miners to reorg min-difficulty blocks

Maybe something to consider would be to have the min-difficulty interval vary. eg, after a real difficulty block, min-difficulty applies if the delta is 1h10m; but it reduces to 20m for consecutive min difficulty blocks. So you could mine perhaps 3 min-difficulty blocks immediately, but would then have to wait 10m before you can do another one, and if a real difficulty block is mined in the meantime, you’re back to a long-ish wait before you can try again. Not really sure there’s anything worth fixing here though – if someone decides to mine min difficulty blocks whenever the tip’s timestamp is less than 1h40m in the future, is that really a big deal?

Sjors commented at 11:22 am on May 6, 2024: member

I’m fine with either fixing the timewarp or not, but in the latter case we might needs testnet5 soon - if someone decides to exploit it.

I think at least we need another variation with the latest block hash in the message.

I don’t think that’s necessary. However, you could use a recent block hash as a provably unspendable public key (instead of all zeros).

fjahr force-pushed on May 6, 2024

fjahr commented at 1:54 pm on May 6, 2024: contributor

Added @wiz as the second seed node 🚀

fjahr commented at 1:56 pm on May 6, 2024: contributor

I don’t think that’s necessary. However, you could use a recent block hash as a provably unspendable public key (instead of all zeros).

Well, we now have a genesis block with the hash in the message and the all zeros public key. I don’t think that makes a difference anymore?

jlopp commented at 2:17 pm on May 6, 2024: contributor

I consider closing the block storm loophole to be rather important - it’s the primary reason why TBTC became scarce and accrued value far faster than ought to have happened. Given that the patch is rather trivial, it would be a huge missed opportunity not to fix it at this time.

fjahr commented at 2:25 pm on May 6, 2024: contributor

I consider closing the block storm loophole to be rather important - it’s the primary reason why TBTC became scarce and accrued value far faster than ought to have happened. Given that the patch is rather trivial, it would be a huge missed opportunity not to fix it at this time.

I agree. Could you say precisely which (combination) of the ideas given here so far do you consider a fix? Currently this PR includes the fix for block storms as we know them now in the second commit with the downside that a miner can still run up the hash rate and leave (described here in more detail) and the fix for the time warp attack in the third commit. Do you think this is sufficient or if not, what else would you like to be included?

jlopp commented at 2:41 pm on May 6, 2024: contributor

I like all of the currently proposed changes.

So the question is this: if a high hashrate miner deliberately stalls testnet and we limp along for an entire difficulty adjustment period with minimum difficulty blocks, what should happen?

I think, in that case, perhaps we SHOULD reset the actual difficulty to the minimum difficulty (or to a sane ASIC difficulty like 1,000,000) and allow the network to find the real hashrate again.

Thus, it would still technically be possible for a block storm to occur, but only if someone expended a decent amount of opportunity cost mining testnet rather than mainnet.

murchandamus commented at 5:00 pm on May 6, 2024: contributor

Prevent use of the 20-minute difficulty exception on the last block in a difficulty period

@murchandamus do you explicitly prefer this over my proposed fix? I find it more elegant and intuitive if we allow the use on all blocks and prevent the adjustment from being affected by it.

Alternatively, I would be satisfied if using the 20-minute difficulty exception would be changed such that nBits could remain at the proper value, and were available for any block in the difficulty period without incurring the difficulty resetting to the minimum. However, my understanding is that the difficulty check in block validation is context independent, and therefore a block would fail if the hash were higher than the nBits value in its own header.

I am not sure I understand this: you are talking about the fix I have proposed here, right? If that is so, I am not sure I understand why the header would diverge from the block on the nBits though.

Sorry, ISTM that I had misunderstood how you implemented that the difficulty adjustment was unaffected by the 20-minute exception. I am fine with any solution that prevents loss of liveliness and mitigates extended blockstorms: i.e. I’m fine with block gusts of <20 blocks or waiting for ~20 minutes to be able to mine a block, but would consider it a failure if there are no new blocks on testnet for extended periods of time while someone is waiting for confirmation.

So the question is this: if a high hashrate miner deliberately stalls testnet and we limp along for an entire difficulty adjustment period with minimum difficulty blocks, what should happen?

It would also make sense to me that if a majority of the blocks in a difficulty period were found with the exception, the difficulty adjust downwards towards some minimum. At the very least, it should halve the real difficulty, because the blocks took twice as long as expected, even if they all were mined at the minimum difficulty.

TheBlueMatt commented at 7:46 pm on May 6, 2024: contributor

I put this elsewhere but figured I should copy it here:

One thing that was done at the start of testnet3 was to include a reasonable amount of test coverage of unique scripts. This was, at a minimum, the Bitcoin Core script tests that were available but probably included scripts from other projects as well. As a result, just syncing the first few thousand blocks of testnet3 is a reasonable test case for anything validating scripts.

I’d strongly encourage the same be done for testnet4, but to improve coverage substantially compared to testnet3 we should consider getting some test-cases out of the Bitcoin Core script fuzzer(s) and embedding that in the chain.

It was also pointed out to me that testnet3 includes a timewarp exploit early in its history, so might want to pre-mine the same if we don’t try to fix that as a part of testnet4

In practice today that probably means pre-mining a handful of blocks to add the test cases and then checkpointing the end of the pre-mine.

in src/chainparamsbase.cpp:21 in 0fae96f589 outdated

16@@ -17,7 +17,8 @@ void SetupChainParamsBaseOptions(ArgsManager& argsman)
17     argsman.AddArg("-regtest", "Enter regression test mode, which uses a special chain in which blocks can be solved instantly. "
18                  "This is intended for regression testing tools and app development. Equivalent to -chain=regtest.", ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::CHAINPARAMS);
19     argsman.AddArg("-testactivationheight=name@height.", "Set the activation height of 'name' (segwit, bip34, dersig, cltv, csv). (regtest-only)", ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::DEBUG_TEST);
20-    argsman.AddArg("-testnet", "Use the test chain. Equivalent to -chain=test.", ArgsManager::ALLOW_ANY, OptionsCategory::CHAINPARAMS);
21+    argsman.AddArg("-testnet", "Use the testnet3 chain. Equivalent to -chain=test. Support for testnet3 is deprecated and will be removed with the next release. Consider moving to testnet4 now by using -testnet4.", ArgsManager::ALLOW_ANY, OptionsCategory::CHAINPARAMS);
22+    argsman.AddArg("-testnet4", "Use the testnet3 chain. Equivalent to -chain=testnet4.", ArgsManager::ALLOW_ANY, OptionsCategory::CHAINPARAMS);

craigraw commented at 10:19 am on May 7, 2024:

Typo here - should be Use the testnet4 chain.

fjahr force-pushed on May 8, 2024

Sjors commented at 7:55 am on May 8, 2024: member

[…] use a recent block hash as a provably unspendable public key (instead of all zeros).

Well, we now have a genesis block with the hash in the message and the all zeros public key. I don’t think that makes a difference anymore?

I find it more elegant, but only if we need another genesis block.

In practice today that probably means pre-mining a handful of blocks to add the test cases and then checkpointing the end of the pre-mine.

That makes sense if we want the test cases as early as possible in the chain. In that case someone should provide such a script in the next few months, then we can make a new genesis block, run the script, add the checkpoint and merge this PR.

I do hope we can rid of the checkpoint code entirely (see #25725), but maybe by then testnet4 will have enough work on it that a reorg is unlikely.

I’m fine with the difficulty fix in e172a96c0781de2bbd69312905d2c16cc1745c2f. Although these minimum difficulty blocks are always at risk of getting wiped out using https://github.com/Sjors/bitcoin/commit/2125fc56163ceaddfadbc78ad0d0da5b1a99a8fa, I agree that doesn’t need to be the default - given the use case of including non-standard transactions.

It would also make sense to me that if a majority of the blocks in a difficulty period were found with the exception, the difficulty adjust downwards towards some minimum. At the very least, it should halve the real difficulty, because the blocks took twice as long as expected, even if they all were mined at the minimum difficulty.

In the current implementation it’s far quicker to jack up the difficulty by 1000x than it takes to drop it. I didn’t do the math on the way up, the way down takes 40 weeks (4 week retarget period, each cutting difficulty in half).

The testnet4 specific code could increase nActualTimespan by n minutes for each minimum difficulty block. For n = 20 that speeds up the way down to 20 weeks (4 week retarget period, each cutting difficulty by 4). For n = 60 the network (almost) recovers in 12 weeks (4 week retarget period, each cutting difficulty by 8).

This can be done by only touching code inside the if (testnet4) branch.

emsit commented at 8:20 am on May 8, 2024: none

Is there a plan to replace the path ‘/testnet3’ with ‘/testnet’? Due to backward compatibility, that might not be a good idea.

https://github.com/fjahr/bitcoin/commit/37e202adc1c734740104c6a5e1d2bd021102764e#diff-583f605f372805d0bc87ebfc893f835b3f66746f560611e682bd917de17586d2 https://github.com/fjahr/bitcoin/commit/37e202adc1c734740104c6a5e1d2bd021102764e#diff-723b9eeeb819047ef03f827ee8a3ccfef48ce7ef6a340a068a68b551993fe70b

wiz approved

wiz commented at 3:33 pm on May 8, 2024: contributor

Tested ACK @ 73c0cdc89bb3 from https://github.com/fjahr/bitcoin/commit/73c0cdc89bb3 Tested ACK @ cb895f48a743 from https://github.com/fjahr/bitcoin/tree/2024-04-testnet-4-fix-v27

Running cb895f48a743 on https://mempool.space/testnet4 using CPU mining, looks good to me

I noticed @Sjors seed at seed.testnet4.bitcoin.sprovoost.nl does not currently return any seeds, but I suppose that might simply be because his node doesn’t know about any yet. I’ve added a bunch of my nodes at seed.testnet4.wiz.biz for now, until a proper P2P network gets setup.

ajtowns commented at 4:49 pm on May 8, 2024: contributor

In the current implementation it’s far quicker to jack up the difficulty by 1000x than it takes to drop it. I didn’t do the math on the way up, the way down takes 40 weeks (4 week retarget period, each cutting difficulty in half).

If your target difficulty takes you 10 minutes to mine a block on average, and the current difficulty is 0.1% of that, it’d take 10k blocks over about 5 days to increase the difficulty 1000x, or about the same work as you’d need to mine ~690 blocks at the target difficulty. If you were able to quickly cycle the difficulty back down again, that would make block storms pretty easy still?

Sjors referenced this in commit 37e202adc1 on May 9, 2024

Sjors commented at 7:43 am on May 9, 2024: member

@emsit wrote:

Is there a plan to replace the path ‘/testnet3’ with ‘/testnet’?

That’s a mistake in the documentation, the behavior isn’t changed. @wiz wrote:

because his node doesn’t know about any yet.

Or not anymore. The seed requires some amount of recent uptime. I had one running as well, but it was on an earlier genesis block. Fixed, there goes my premine :-)

My node found your peers and synced to the tip.

If you were able to quickly cycle the difficulty back down again, that would make block storms pretty easy still?

So that suggests we shouldn’t make downward adjustment too fast. Perhaps just add 20 minutes to nActualTimespan so it drops by usual maximum 4x?

craigraw commented at 11:21 am on May 9, 2024: none

Implemented support in Sparrow in https://github.com/sparrowwallet/sparrow/commit/d420c71673de5823ba1f6c4a0d4411ebd747c7ba, currently following the approach here to use testnet3 when run with -n testnet and testnet4 with -n testnet4. Wallets are loading successfully using either the mempool.space Electrum public server or directly with Bitcoin Core RPC.

Sjors commented at 11:22 am on May 9, 2024: member

@craigraw that might be premature, if we decide to do another genesis block.

craigraw commented at 11:28 am on May 9, 2024: none

@Sjors It shouldn’t matter - if the network is reset, the existing wallet history will simply disappear when the wallet is reloaded. I will certainly warn of the possibility of a future network reset on release.

wiz commented at 6:54 pm on May 9, 2024: contributor

that might be premature, if we decide to do another genesis block.

From reading this thread it seems like there is rough consensus for the testnet4 genesis block now, and the discussion has mostly shifted to tweaking the difficulty adjustment algorithm and what fancy test transactions to mine into blocks - so I imagine the blockchain might get re-org’d and genesis block probably doesn’t change, but as @craigraw said even if it does that’s totally fine too.

As for being “premature”, to be honest many people are complaining that testnet3 has become unusable for their development workflows and CI integrations etc. so I am happy to help launch a beta testnet4 network and make the APIs available on mempool.space for those who need it, with a warning note saying the network might be reset at anytime.

In any case, I hope you guys can get this PR merged soon - as others said if we make a new testnet every year or two that sounds great.

Sjors commented at 7:08 am on May 10, 2024: member

@wiz the v28 release isn’t going to happen sooner, so I assume you’d like this merged so people can more easily test on the master branch? That’s probably fine - worst case people need to delete their ~/.bitcoin/testnet4 directory when the release comes out.

jlopp commented at 2:18 pm on May 12, 2024: contributor

TACK 06c2c713c52b60231efc3e00d2c5eb0bf9e345f9

DrahtBot requested review from wiz on May 12, 2024

DrahtBot requested review from murchandamus on May 12, 2024

wtogami commented at 5:36 am on May 13, 2024: contributor

So if the difficulty hack is removed completely, anyone wishing to submit a transaction would have to go purchase and set up mining hardware, or find a miner willing to accept the transaction.

With zero changes to Core, signet could serve the same need to get non standard transactions into test blocks. Signet could have a website where you submit nonstandard transactions for the miner to include. This would directly parallel how out-of-band transactions are added by miners on mainnet.

russeree commented at 6:47 am on May 13, 2024: contributor

tACK - 06c2c713c52b60231efc3e00d2c5eb0bf9e345f9

Have been mining and stress testing (reorgs) and functions as expected. Nuanced testing and full review have not been completed.

craigraw commented at 6:55 am on May 13, 2024: none

tACK https://github.com/bitcoin/bitcoin/commit/06c2c713c52b60231efc3e00d2c5eb0bf9e345f9

Emzy commented at 7:05 am on May 13, 2024: contributor

tACK 06c2c71

I just tested mining with CPUs and a S9 ASIC.

DrahtBot added the label Needs rebase on May 13, 2024

in src/kernel/chainparams.cpp:349 in 06c2c713c5 outdated

329+        nDefaultPort = 48333;
330+        nPruneAfterHeight = 1000;
331+        m_assumed_blockchain_size = 0;
332+        m_assumed_chain_state_size = 0;
333+
334+        const char* testnet4_genesis_msg = "03/May/2024 000000000000000000001ebd58c244970b3aa9d783bb001011fbe8ea8e98e00e";

Roasbeef commented at 6:42 pm on May 13, 2024:

Wouldn’t it be better to defer the creation of the genesis block until a much farther date, so commit to changing during the rc phase, until the final release. We’d then include a similar mainnet block height hash or w/e as well. This way people aren’t mining the chain for months before it’s included in a real release.

fjahr commented at 11:51 pm on May 13, 2024:

Wouldn’t it be better to defer the creation of the genesis block until a much farther date, so commit to changing during the rc phase, until the final release. We’d then include a similar mainnet block height hash or w/e as well.

We could still reset it before the release if reviewers really think it’s necessary (I said so early in the discussion here as well) but we couldn’t have tested that the PR works without a Genesis block. And think it’s good we discussed what’s to be included in the message early and didn’t bikeshed that part last minute before the release.

This way people aren’t mining the chain for months before it’s included in a real release.

It’s not clear to me why that is a problem though.

Sjors commented at 7:14 am on May 14, 2024:

It could be a problem if we want test vectors early in the chain. But that can be fixed by a reset once those test vectors are available.

russeree commented at 7:43 am on May 14, 2024:

I don’t think the fact the chain has been mined on has any large impact and may even reduce the value proposition later on since there would be ’large’ swaths of coins that the early devs have.

Disclaimer: I mined 800K+ testnet4 coins with 700Th/s. I also don’t mind burning those coins with OP_RETURN if desired by the community.

Sjors commented at 8:57 am on May 14, 2024:

@russeree the use case for these test vectors is for regression testing, so you’d spin up a testnet4 node, sync a few thousand blocks and delete it again. Having such a small number of blocks might allow you to store the actual blocks on the CI server - so it doesn’t need to make network requests.

Roasbeef commented at 0:59 am on July 3, 2024:

It’s not clear to me why that is a problem though.

It gives those involved in the creation of the new parameters months of mining before the “public”, which will skew the distribution of new testnet coins. If those that mined the chain for months before public release aren’t also running faucets to easily disseminate the coins, then we end up in the same position where testnet coins are hard to obtain by developers.

Rob1Ham commented at 8:34 pm on May 13, 2024: none

tACK https://github.com/bitcoin/bitcoin/commit/06c2c713c52b60231efc3e00d2c5eb0bf9e345f9

in src/validation.cpp:4188 in 06c2c713c5 outdated

4027@@ -4028,6 +4028,16 @@ static bool ContextualCheckBlockHeader(const CBlockHeader& block, BlockValidatio
4028     if (block.GetBlockTime() <= pindexPrev->GetMedianTimePast())
4029         return state.Invalid(BlockValidationResult::BLOCK_INVALID_HEADER, "time-too-old", "block's timestamp is too early");
4030 
4031+    // Testnet4 only: Check timestamp against prev for difficulty-adjustment
4032+    // blocks to prevent timewarp attacks (see https://github.com/bitcoin/bitcoin/pull/15482).
4033+    if (consensusParams.hashGenesisBlock == uint256S("0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043")) {

Sjors commented at 7:24 am on May 14, 2024:

06c2c713c52b60231efc3e00d2c5eb0bf9e345f9: we have to remember to update this hardcoded value in three different places if we reset testnet4 (before the release). I verified that it currently matches the actual genesis block (getblockhash 0).

It’s not dangerous to mainnet if we forget, it would only break testnet4. But I would prefer to introduce an enum class Network so we can do consensusParams.network == Network::Testnet4. It can wait for a followup.

If you’re worried about growing struct Params you can remove fPowNoRetargeting and replace its use with consensusParams.network == Network::Regtest`. Basically anything that’s only used by a single test network might as well use this more direct approach.

fjahr commented at 11:03 pm on May 28, 2024:

Sounds good, but keeping this for a follow-up.

in src/validation.cpp:4191 in 06c2c713c5 outdated

4027@@ -4028,6 +4028,16 @@ static bool ContextualCheckBlockHeader(const CBlockHeader& block, BlockValidatio
4028     if (block.GetBlockTime() <= pindexPrev->GetMedianTimePast())
4029         return state.Invalid(BlockValidationResult::BLOCK_INVALID_HEADER, "time-too-old", "block's timestamp is too early");
4030 
4031+    // Testnet4 only: Check timestamp against prev for difficulty-adjustment
4032+    // blocks to prevent timewarp attacks (see https://github.com/bitcoin/bitcoin/pull/15482).
4033+    if (consensusParams.hashGenesisBlock == uint256S("0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043")) {
4034+        if (pindexPrev->nHeight % consensusParams.DifficultyAdjustmentInterval() == consensusParams.DifficultyAdjustmentInterval() - 1) {

Sjors commented at 7:47 am on May 14, 2024:

06c2c713c52b60231efc3e00d2c5eb0bf9e345f9: Suggested comment:

0// For the first block of each difficulty adjustment interval,
1// except the genesis block.

fjahr commented at 11:03 pm on May 28, 2024:

taken with minor edit

in src/validation.cpp:4185 in 06c2c713c5 outdated

4027@@ -4028,6 +4028,16 @@ static bool ContextualCheckBlockHeader(const CBlockHeader& block, BlockValidatio
4028     if (block.GetBlockTime() <= pindexPrev->GetMedianTimePast())
4029         return state.Invalid(BlockValidationResult::BLOCK_INVALID_HEADER, "time-too-old", "block's timestamp is too early");
4030 
4031+    // Testnet4 only: Check timestamp against prev for difficulty-adjustment

Sjors commented at 8:13 am on May 14, 2024:

06c2c713c52b60231efc3e00d2c5eb0bf9e345f9: I initially thought that this should be moved to ConnectBlock, somewhere in the “Sanity checks”, maybe right before bool fScriptChecks = true;. But it’s fine here.

It doesn’t need to be here, because we don’t use the system clock for this check. However it’s more readable to have it right next to the time-too-old check. So is that safe?

ContextualCheckBlockHeader is only called when we receive a header. It’s called by AcceptBlockHeader (and by the miner), which in turn is called by ProcessNewBlockHeaders (header message or compact block) and AcceptBlock (block message, to store on disk). Notably it’s not called during a reorg or when doing a reindex. However in both these cases the headers will have been previously checked for this issue.

fjahr commented at 11:02 pm on May 28, 2024:

I will keep it here for now unless other reviewers think it should be moved.

in src/pow.cpp:66 in e172a96c07 outdated

60@@ -61,7 +61,19 @@ unsigned int CalculateNextWorkRequired(const CBlockIndex* pindexLast, int64_t nF
61     // Retarget
62     const arith_uint256 bnPowLimit = UintToArith256(params.powLimit);
63     arith_uint256 bnNew;
64-    bnNew.SetCompact(pindexLast->nBits);
65+
66+    // Special difficulty rule for Testnet4
67+    if (params.fPowAllowMinDifficultyBlocks && params.hashGenesisBlock == uint256S("0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043")) {

Sjors commented at 8:34 am on May 14, 2024:

e172a96c0781de2bbd69312905d2c16cc1745c2f: params.fPowAllowMinDifficultyBlocks && is redundant. Though you could an assert.

fjahr commented at 11:02 pm on May 28, 2024:

right, I removed the redundant check, also because I already rely only on hashGenesisBlock in the timewarp code as well.

in src/pow.cpp:69 in e172a96c07 outdated

65+
66+    // Special difficulty rule for Testnet4
67+    if (params.fPowAllowMinDifficultyBlocks && params.hashGenesisBlock == uint256S("0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043")) {
68+        // Use the last non-special-min-difficulty-rules-block
69+        const CBlockIndex* pindex = pindexLast;
70+        const unsigned int pow_min{bnPowLimit.GetCompact()};

Sjors commented at 8:35 am on May 14, 2024:

e172a96c0781de2bbd69312905d2c16cc1745c2f: const uint32_t? (matching return type of GetCompact())

fjahr commented at 11:02 pm on May 28, 2024:

done

in src/pow.cpp:79 in e172a96c07 outdated

66+    // Special difficulty rule for Testnet4
67+    if (params.fPowAllowMinDifficultyBlocks && params.hashGenesisBlock == uint256S("0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043")) {
68+        // Use the last non-special-min-difficulty-rules-block
69+        const CBlockIndex* pindex = pindexLast;
70+        const unsigned int pow_min{bnPowLimit.GetCompact()};
71+        while (pindex->pprev && pindex->nHeight % params.DifficultyAdjustmentInterval() != 0 && pindex->nBits == pow_min)

Sjors commented at 8:51 am on May 14, 2024:

e172a96c0781de2bbd69312905d2c16cc1745c2f: this had me confused again, so maybe a comment is helpful.

If miners go away in the middle of difficulty adjustment period A, this rule will find the last “real” nBits value and apply it to the first block of period B. If nothing changes, then for period C this seeks back to the first block of period B, finds the real value and applies it.

But then how do the first blocks of B and C ever mined?

Well, remember that GetNextWorkRequired ignores the nBit value if enough time went by.

A test would be nice, but unfortunately it seems the only way to do that is to introduce a new regtest with the same rule, but at much lower difficulty.

jsarenik commented at 5:21 am on May 15, 2024:

Could one possibly test with -noconnect, i.e. without connecting to the network and getting any other than genesis testnet4 block?

fjahr commented at 11:02 pm on May 28, 2024:

Well, remember that GetNextWorkRequired ignores the nBit value if enough time went by.

That last part isn’t true. The first block of the difficulty period does not allow the usage of the 20-min rule. That code is wrapped in the following if statement so GetNextWorkRequired always passes off to CalculateNextWorkRequired for the first block in a new difficulty period.

0// Only change once per difficulty adjustment interval
1if ((pindexLast->nHeight+1) % params.DifficultyAdjustmentInterval() != 0) {
2    ...
3}

I managed to confuse myself with this again today as well, so I added an adapted version of your comment here. The confusing part for me is that this is not a change of behavior which is why I also failed to emphasize this in the BIP so far. The first block of the difficulty period also can not use the 20-min rule in Testnet 3. But there it can get difficulty 1 when the last block of the previous period was mined with the 20-min rule, which then leads to a block storm.

For this to work the way you describe we would need to add the 20-min also in CalculateNextWorkRequired but that just gives us back the good ol’ block storms. Maybe we could add something crazy there like 24 hours, that would mean we reset to 1 to get unstuck if a block isn’t found for a day without anyone pointing their asic to it. But I am not sure we really need this.

murchandamus commented at 7:23 pm on July 29, 2024:

As mentioned on the BIP PR, it would be easier to simply read the difficulty from the first block in each difficulty period since it must have the actual difficulty.

fjahr commented at 9:02 pm on July 30, 2024:

Done, I am now using the first block

Sjors approved

Sjors commented at 8:54 am on May 14, 2024: member

tACK 06c2c713c52b60231efc3e00d2c5eb0bf9e345f9

2024-05-30: retracting ACK because I misunderstood the anti block-storm measure, see #29775 (comment).

There’s a bunch of things here and in earlier comments that warrant followup, but not worth losing ACKs.

We currently have 24729 blocks and the difficulty is 16,777,216. That makes sense: we’ve had 12 retarget periods, and if each time the difficulty maximally increased, you get exactly 4^12.

In order to get test vectors in early, we’d have to reset with a new genesis block and immediately publish the transactions.

It would be nice to set the genesis block difficulty to whatever the network difficulty is when we reset it. But that’s pure cosmetics.

I reviewed the timewarp commit, because ideally we get this right the first time. Then a mainnet soft fork only needs to change the if (consensusParams.hashGenesisBlock... check with DeploymentActiveAfter.

jsarenik commented at 5:18 am on May 15, 2024: none

@Sjors wrote:

If anyone wants to deploy a faucet, let me know and I’ll send some coins… unless someone reorgs me.

Yes, I’d be happy to set up a Testnet4 faucet similar to Alt Signet Faucet. Please send me some testnet coins to tb1p4tp4l6glyr2gs94neqcpr5gha7344nfyznfkc8szkreflscsdkgqsdent4 on Testnet4.

emsit commented at 5:48 am on May 15, 2024: none

If someone has coins they don’t need, they can also donate them to the donate address of the faucet: https://coinfaucet.eu/en/btc-testnet4/

tb1qn9rvr53m7qvrpysx48svuxsgahs88xfsskx367

emsit commented at 6:09 am on May 15, 2024: none

@Sjors wrote:

If anyone wants to deploy a faucet, let me know and I’ll send some coins… unless someone reorgs me.

Yes, I’d be happy to set up a Testnet4 faucet similar to Alt Signet Faucet. Please send me some testnet coins to tb1p4tp4l6glyr2gs94neqcpr5gha7344nfyznfkc8szkreflscsdkgqsdent4 on Testnet4.

Podelil som sa s tebou, svet je maly :smiley:

jsarenik commented at 7:00 am on May 15, 2024: none

I shared with you, the world is small 😃 ![]

Thank you! One new testnet4 faucet running at https://testnet4.anyone.eu.org/

wiz commented at 9:46 am on May 17, 2024: contributor

testnet4 faucet up at https://mempool.space/testnet4/faucet feel free to donate coins to tb1q0dzcgv7scppjxsnwlzpkt02vlmc5rtr40wyjgr

kcalvinalvin commented at 3:23 pm on May 23, 2024: contributor

Maybe I’m getting a wrong vibe for the whole PR but what happened to actually having a written out specification. Especially like issues open like this. The best thing out there now is just the mailing list and this PR, which I don’t even know which parts of the mailing list this PR is supporting without reading into the code.

Testnet is where devs actually test things and this is ridiculous that it’s being pushed out like this. Yes for mainnet people mostly run Core but for testnet people are more open to alt-implementations and so just pushing it out like this is irresponsible because there needs to be cross compatibility between implementations.

I’ll be NACK-ing all testnet4 support on btcd and will not be supporting testnet4 utreexod until there’s a written out specification for it.

sipa commented at 6:28 pm on May 23, 2024: member

There are several BIPs that contain specifications relating to testnet, so perhaps a BIP is the right place to define testnet4? The BIP process predates testnet3, but only by a few months, so I don’t think we should see the lack of a testnet3 BIP as an argument against this.

kcalvinalvin commented at 7:15 pm on May 23, 2024: contributor

I don’t think we should see the lack of a testnet3 BIP as an argument against this.

Not even a BIP but some document that specifies testnet4 besides just a PR that still might get changed.

I think in 2024 we can agree that there’s more than just Bitcoin Core and asking other implementations to “read the Bitcoin Core codebase” is a ridiculous ask.

fjahr commented at 10:29 pm on May 27, 2024: contributor

Here is a BIP PR for Testnet 4: https://github.com/bitcoin/bips/pull/1601

I think the written specification needs to be a BIP to be considered meaningful in the long run. If I just put it in a gist or something like that it depends on me alone to make changes should they become necessary for example. I would rather have it be managed by the community if the written specification is what people turn to.

The PR still might get changed obviously but I will update the BIP PR accordingly.

fjahr force-pushed on May 28, 2024

fjahr commented at 11:13 pm on May 28, 2024: contributor

I’m pouring one out for all the tACKs we’ve lost but the rebase was necessary for a possible merge.

I have addressed the comments from @Sjors and I think those were all that are in scope for this PR here. Mostly it’s adding comments and two small code simplifications.

0git range-diff master 06c2c713c52b60231efc3e00d2c5eb0bf9e345f9 86fea43762859478868bcca66e7ab56e8728e58f

I think the chain replay idea from @TheBlueMatt is probably best tracked in a separate issue. Potentially there are already projects out there that can provide the necessary functionality, I am not aware of anything like that though.

DrahtBot removed the label Needs rebase on May 28, 2024

fjahr commented at 11:36 pm on May 28, 2024: contributor

The CI failure doesn’t seem related, somehow the test-each-commit job was instantly cancelled.

fjahr commented at 1:08 pm on May 29, 2024: contributor

In the current implementation it’s far quicker to jack up the difficulty by 1000x than it takes to drop it. I didn’t do the math on the way up, the way down takes 40 weeks (4 week retarget period, each cutting difficulty in half).

The testnet4 specific code could increase nActualTimespan by n minutes for each minimum difficulty block. For n = 20 that speeds up the way down to 20 weeks (4 week retarget period, each cutting difficulty by 4). For n = 60 the network (almost) recovers in 12 weeks (4 week retarget period, each cutting difficulty by 8).

I forgot to address this comment from @Sjors earlier: I think it’s an interesting idea but I am not sure about the adverse effects this could have. The base case for the network should be that we have a fluctuating but somewhat stable hashrate and a few people will use the 20-min rule to get their non-standard txs in or just to get some coins. How many these will be of the 2016 blocks, I don’t know. Let’s say it’s 100 20-min blocks and they are always mined instantly (no time wasted; I am ignoring real 20-min blocks of which we will certainly also see a few). Then in this state when difficulty should be adjusted up (because the 100 blocks came fast) but instead the difficulty would be adjusted down because of the adjustment.

The worst case I think is that someone tries to get as many 20-min blocks as possible constantly and with that grinds down the difficulty so that we will have a much faster block time than 10 minutes. I didn’t do the exact math on it to see where we end up in an equilibrium in that case but I think this would pretty annoying.

I think even if there is no attack we would end up with a faster block time on average. Maybe we don’t want to assume any stability as the base case but I am still not sure the upside outweighs the downside on this. I will keep thinking about it more if I can think of more edge cases where this might lead to different outcomes than we want.

murchandamus commented at 3:57 pm on May 29, 2024: contributor

I missed that before, but the improved text on the BIP made me realize. It looks like an attacker could jack up the difficulty by mining a few difficulty periods with an ASIC and then stop after the last block in a difficulty period. The network would then be on a difficulty some 4n higher than before, and stuck looking to mine a first block at full difficulty.

I previously understood that for the difficulty it just goes back to the latest block that has a non-1 difficulty, and didn’t realize that the first block would need to be mined at full difficulty. Is there a way to prevent the reset to minimum difficulty while allowing the 20-minute exception for every block? Would that require something like @ajtowns’s storing the actual difficulty in the version or similar?

jsarenik commented at 4:03 pm on May 29, 2024: none

Tested ACK 86fea43762

DrahtBot requested review from Sjors on May 29, 2024

DrahtBot requested review from craigraw on May 29, 2024

DrahtBot requested review from jlopp on May 29, 2024

DrahtBot requested review from russeree on May 29, 2024

fjahr commented at 4:15 pm on May 29, 2024: contributor

I missed that before, but the improved text on the BIP made me realize. It looks like an attacker could jack up the difficulty by mining a few difficulty periods with an ASIC and then stop after the last block in a difficulty period. The network would then be on a difficulty some 4n higher than before, and stuck looking to mine a first block at full difficulty.

I previously understood that for the difficulty it just goes back to the latest block that has a non-1 difficulty, and didn’t realize that the first block would need to be mined at full difficulty. Is there a way to prevent the reset to minimum difficulty while allowing the 20-minute exception for every block? Would that require something like @ajtowns’s storing the actual difficulty in the version or similar?

Yes, this has also been described here by AJ. @Sjors suggestion to change nActualTimespan was also aimed at this problem but I have a few doubts outlined above. I think @ajtowns suggestion to use the version field is the only complete solution for this problem so far.

murchandamus commented at 5:42 pm on May 29, 2024: contributor

Yes, this has also been described here by AJ.

I must have read that before it was edited and missed the edit.

@Sjors suggestion to change nActualTimespan was also aimed at this problem but I have a few doubts outlined above. I think @ajtowns suggestion to use the version field is the only complete solution for this problem so far.

Thanks for keeping the overview! :) I guess if someone were to troll testnet in that manner, it might have a social solution, where we could probably convince some ASIC-directing individual to point some hashpower at Testnet for one block per difficulty period until the difficulty recovers to reasonable heights. Otherwise, it might be a good time to reset again. ;)

ajtowns commented at 0:10 am on May 30, 2024: contributor

I missed that before, but the improved text on the BIP made me realize. It looks like an attacker could jack up the difficulty by mining a few difficulty periods with an ASIC and then stop after the last block in a difficulty period. The network would then be on a difficulty some 4n higher than before, and stuck looking to mine a first block at full difficulty.

I don’t think that’s much of a concern – all you’d need to do is invalidateblock the last block of the period, mine a new one with a much later timestamp, and then mine another block in the new period, that no longer has a 4x increased difficulty. At that point your new chain has more work than the old chain, and you continue from there with normal difficulty.

An attacker with 50x more hashpower than everyone else combined could conceivably rush 2015 blocks in ~6 hours, leaving the chain stalled for two weeks, and potentially repeat that attack as often as they liked, but that’s probably a fair amount of hashpower to dedicate to griefing testnet, at which point switching to signet or spinning up testnet5 would presumably make sense.

TheBlueMatt commented at 5:50 pm on May 30, 2024: contributor

In practice today that probably means pre-mining a handful of blocks to add the test cases and then checkpointing the end of the pre-mine.

That makes sense if we want the test cases as early as possible in the chain. In that case someone should provide such a script in the next few months, then we can make a new genesis block, run the script, add the checkpoint and merge this PR.

I do hope we can rid of the checkpoint code entirely (see #25725), but maybe by then testnet4 will have enough work on it that a reorg is unlikely.

It also seems to me that its not crazy to have a testnet-specific validity rule that isn’t “the checkpoint code” :).

Sjors commented at 6:15 pm on May 30, 2024: member

@Sjors suggestion to change nActualTimespan was also aimed at this problem

No it wasn’t, it seems I was confused myself and thought there was no problem: #29775 (review)

murchandamus commented at 6:46 pm on May 30, 2024: contributor

I missed that before, but the improved text on the BIP made me realize. It looks like an attacker could jack up the difficulty by mining a few difficulty periods with an ASIC and then stop after the last block in a difficulty period. The network would then be on a difficulty some 4n higher than before, and stuck looking to mine a first block at full difficulty.

I don’t think that’s much of a concern – all you’d need to do is invalidateblock the last block of the period, mine a new one with a much later timestamp, and then mine another block in the new period, that no longer has a 4x increased difficulty. At that point your new chain has more work than the old chain, and you continue from there with normal difficulty.

I think that could reduce the difficulty by up to a factor of 16 (if you are willing to wait up to eight weeks), but I don’t see how someone needing to manually intervene and most likely still needing an ASIC mitigates the potential liveness issue here.

An attacker with 50x more hashpower than everyone else combined could conceivably rush 2015 blocks in ~6 hours, leaving the chain stalled for two weeks, and potentially repeat that attack as often as they liked, but that’s probably a fair amount of hashpower to dedicate to griefing testnet, at which point switching to signet or spinning up testnet5 would presumably make sense.

I’m not concerned with an attacker that has 50× more hashpower. If we were in a situation where Testnet has no ASICs mining it and someone points a millionth of the mainnet hashpower at Testnet (today about 650 TH/s), they could easily mine 10 difficulty periods in minutes and put the first block of the next difficulty period well out of the range of non-ASICs. If someone points more hashrate at it, e.g. in the range of a thousandth of mainnet, it could easily shoot up the difficulty even out of the range of a small number of S9s, which nominally have 14 TH/s.

As far as I recall, that’s exactly the problem we had with Testnet 1 and 2 that lead to the 20-minute exception being introduced in the first place: mining pools occasionally test their setups on Testnet and cause the difficulty to shoot up like crazy.

boring877 commented at 6:57 pm on May 31, 2024: none

Testnet 3 will always be the only testnet. its the market who make the rules !

fjahr commented at 9:31 pm on May 31, 2024: contributor

If we were in a situation where Testnet has no ASICs mining it and someone points a millionth of the mainnet hashpower at Testnet (today about 650 TH/s), they could easily mine 10 difficulty periods in minutes and put the first block of the next difficulty period well out of the range of non-ASICs.

The hashpower on Testnet3 seems to have been fluctuating around ~500 TH/s over the past 2 years: https://mempool.space/testnet/graphs/mining/hashrate-difficulty If that data is correct, I am not so concerned about 650 TH/s but I am not sure if the 20-min exception blocks mess with those statistics.

Why do you think this issue has never happened on Testnet3? Someone can run up the difficulty there today like you describe and leave on the last block of a difficulty adjustment period, the chain would stall the same as with the code here.

I wrote a little script to check how many adjustment blocks took longer than 20min and what the most extreme cases were. It doesn’t look too bad honestly, over these 13 years we have had a handful that took more than an hour, two over 5h, but I would have expected to find worse. The delta is just comparing the timestamp to the previous block, so granted, there could be some shenanigans going on and that’s not the real delta but even if the prev block actually had a timestamp 2 hours in the future for example, it still doesn’t seem too terrible to me to have these few outliers over 13 years.

 0Height: 16128, Time Difference: 1201 seconds
 1Height: 20160, Time Difference: 1338 seconds
 2Height: 26208, Time Difference: 1613 seconds
 3Height: 32256, Time Difference: 2430 seconds
 4Height: 38304, Time Difference: 1843 seconds
 5Height: 92736, Time Difference: 1742 seconds
 6Height: 108864, Time Difference: 1379 seconds
 7Height: 127008, Time Difference: 19999 seconds
 8Height: 205632, Time Difference: 3426 seconds
 9Height: 225792, Time Difference: 2892 seconds
10Height: 243936, Time Difference: 2509 seconds
11Height: 270144, Time Difference: 8494 seconds
12Height: 276192, Time Difference: 1410 seconds
13Height: 278208, Time Difference: 5749 seconds
14Height: 308448, Time Difference: 2898 seconds
15Height: 314496, Time Difference: 1884 seconds
16Height: 316512, Time Difference: 8331 seconds
17Height: 318528, Time Difference: 3992 seconds
18Height: 322560, Time Difference: 5553 seconds
19Height: 328608, Time Difference: 5121 seconds
20Height: 374976, Time Difference: 1428 seconds
21Height: 530208, Time Difference: 3827 seconds
22Height: 546336, Time Difference: 2645 seconds
23Height: 921312, Time Difference: 1830 seconds
24Height: 925344, Time Difference: 1664 seconds
25Height: 1034208, Time Difference: 1205 seconds
26Height: 1116864, Time Difference: 3558 seconds
27Height: 1179360, Time Difference: 2390 seconds
28Height: 1288224, Time Difference: 1469 seconds
29Height: 1296288, Time Difference: 2198 seconds
30Height: 1354752, Time Difference: 2686 seconds
31Height: 1518048, Time Difference: 2423 seconds
32Height: 1572480, Time Difference: 1607 seconds
33Height: 1576512, Time Difference: 3231 seconds
34Height: 1578528, Time Difference: 4781 seconds
35Height: 1580544, Time Difference: 1757 seconds
36Height: 1665216, Time Difference: 4310 seconds
37Height: 1721664, Time Difference: 2089 seconds
38Height: 1830528, Time Difference: 1461 seconds
39Height: 1901088, Time Difference: 1672 seconds
40Height: 1903104, Time Difference: 2170 seconds
41Height: 1971648, Time Difference: 2227 seconds
42Height: 1975680, Time Difference: 1683 seconds
43Height: 2034144, Time Difference: 1615 seconds
44Height: 2096640, Time Difference: 1849 seconds
45Height: 2098656, Time Difference: 1940 seconds
46Height: 2102688, Time Difference: 1495 seconds
47Height: 2134944, Time Difference: 1494 seconds
48Height: 2163168, Time Difference: 1359 seconds
49Height: 2225664, Time Difference: 3482 seconds
50Height: 2245824, Time Difference: 1852 seconds
51Height: 2314368, Time Difference: 2212 seconds
52Height: 2342592, Time Difference: 1461 seconds
53Height: 2348640, Time Difference: 1685 seconds
54Height: 2419200, Time Difference: 2266 seconds
55Height: 2421216, Time Difference: 1885 seconds
56Height: 2425248, Time Difference: 1477 seconds
57Height: 2431296, Time Difference: 18461 seconds
58Height: 2530080, Time Difference: 1603 seconds
59Height: 2534112, Time Difference: 1884 seconds
60Height: 2538144, Time Difference: 1213 seconds
61Height: 2578464, Time Difference: 3759 seconds

ajtowns commented at 3:02 pm on June 11, 2024: contributor

I think that could reduce the difficulty by up to a factor of 16 (if you are willing to wait up to eight weeks), but I don’t see how someone needing to manually intervene and most likely still needing an ASIC mitigates the potential liveness issue here.

If someone’s attacking testnet with 4x more hashpower than the rest of the network (ie, 80% hash rate), can’t they just do a 50% attack and mine empty blocks? If you’re trying to run a chain with minority hashpower you need to do something like signet as far as I can see.

they could easily mine 10 difficulty periods in minutes and put the first block of the next difficulty period well out of the range of non-ASICs.

10 difficulty periods with a 4x difficulty increase each is the equivalent of 704,642,400 blocks in the time the rest of the network would mine 1 block, so that’s eight 9’s (99.999999%) of hashpower. If the rest of the network ran invalidateblock and tried mining until they had a more work chain, they’d be waiting for 704 million blocks…

murchandamus commented at 5:10 pm on June 12, 2024: contributor

Why do you think this issue has never happened on Testnet3? Someone can run up the difficulty there today like you describe and leave on the last block of a difficulty adjustment period, the chain would stall the same as with the code here.

Do blocks mined with the exception count towards the total work according to difficulty-1 or the actual difficulty? If it’s the latter, in testnet3 you could just invalidate the last block in the previous difficulty period with a difficulty-1 block. If it’s the former, I agree, the absence of this attack may indicate that I’m worrying too much.

Either way, it seems like my concern has been heard, but is not shared broadly, so please feel free to consider it addressed.

lozanopo approved

fjahr commented at 9:39 pm on June 19, 2024: contributor

Do blocks mined with the exception count towards the total work according to difficulty-1 or the actual difficulty? If it’s the latter, in testnet3 you could just invalidate the last block in the previous difficulty period with a difficulty-1 block. If it’s the former, I agree, the absence of this attack may indicate that I’m worrying too much.

(Side-note: I first read difficulty-1 as “difficulty minus 1” and got a bit confused but I think you mean the same as “difficulty=1”)

I needed to check this myself: Total work is calculated here and here (the formula is identical). This calls GetBlockProof() which uses the blocks nBits, not the actual target difficulty of the current period. So total work is only increased by difficulty-1.

DrahtBot added the label Needs rebase on Jun 24, 2024

fjahr force-pushed on Jun 24, 2024

fjahr commented at 10:01 pm on June 24, 2024: contributor

Rebased to fix merge conflict

FYI: This PR is scheduled as the topic for the PR Review Club on July 3rd: https://bitcoincore.reviews/29775 You can see a preview of my suggested questions here: https://github.com/bitcoin-core-review-club/website/pull/765 Consider yourself invited!

DrahtBot removed the label Needs rebase on Jun 25, 2024

jsarenik commented at 4:03 am on June 26, 2024: none

Tested ACK aba504759caa

in src/kernel/chainparams.cpp:337 in e40f06acfd outdated

317+        consensus.vDeployments[Consensus::DEPLOYMENT_TAPROOT].bit = 2;
318+        consensus.vDeployments[Consensus::DEPLOYMENT_TAPROOT].nStartTime = Consensus::BIP9Deployment::ALWAYS_ACTIVE;
319+        consensus.vDeployments[Consensus::DEPLOYMENT_TAPROOT].nTimeout = Consensus::BIP9Deployment::NO_TIMEOUT;
320+        consensus.vDeployments[Consensus::DEPLOYMENT_TAPROOT].min_activation_height = 0; // No activation delay
321+
322+        consensus.nMinimumChainWork = uint256{};

Sjors commented at 7:12 am on June 26, 2024:

e40f06acfd090797f14abdc1768d7b96ae4eeed8: one advantage of having already had some mining is that we can set nMinimumChainWork shortly before the release (if only so people can test the headers pre-sync behavior).

in src/pow.cpp:61 in 71ce039c20 outdated

60@@ -61,7 +61,28 @@ unsigned int CalculateNextWorkRequired(const CBlockIndex* pindexLast, int64_t nF
61     // Retarget

Sjors commented at 7:24 am on June 26, 2024:

71ce039c20f78d6454c3f930d1510f7d3e3b8652: two more followup suggestions:

Document CalculateNextWorkRequired (including that this is only called for retarget blocks) and GetNextWorkRequired (called for every block)
Add Assume((pindexLast->nHeight + 1) % params.DifficultyAdjustmentInterval() == 0); at the top

in src/pow.cpp:80 in 71ce039c20 outdated

76+        // nBits are preserved because the first blocks of periods B and C
77+        // do not allow for the usage of the min-difficulty exception.
78+        const CBlockIndex* pindex = pindexLast;
79+        const uint32_t pow_min{bnPowLimit.GetCompact()};
80+        while (pindex->pprev && pindex->nHeight % params.DifficultyAdjustmentInterval() != 0 && pindex->nBits == pow_min)
81+            pindex = pindex->pprev;

Sjors commented at 7:32 am on June 26, 2024:

I just noticed that this wile code is duplicated from GetNextWorkRequired. A helper function might be nice here. The original is inside an if (params.fPowAllowMinDifficultyBlocks), so refactoring there isn’t too scary. It makes it more clear that we consider “last non-special-min-difficulty-rules-block” in two places.

Alternatively your remark in the rationale section of the BIP might get rid of this dpulicate code altogether:

For the block storm fix an alternative fix could have been to prevent the last block in a difficulty period from applying the existing difficulty exception. Both solutions were deemed acceptable and there was no clear preference among reviewers.

Sjors commented at 10:07 am on June 26, 2024:

Also, let’s add something like this below // Special difficulty rule for testnet::

0// Note that the first block of an interval can't be min-difficulty.

fjahr commented at 9:05 pm on July 30, 2024:

This was made obsolete by directly using the first block of the difficulty period. I also shortened the existing comment because of that.

Sjors approved

Sjors commented at 10:03 am on June 26, 2024: member

ACK aba504759caa13338aa574a7839d078b989fbf6d

in test/functional/wallet_crosschain.py:28 in aba504759c outdated

23@@ -24,6 +24,12 @@ def setup_network(self):
24         self.nodes[1].chain = 'testnet3'
25         self.nodes[1].extra_args = ['-maxconnections=0', '-prune=550'] # disable testnet sync
26         self.nodes[1].replace_in_config([('regtest=', 'testnet='), ('[regtest]', '[test]')])
27+
28+        # Switch node 1 to testnet4 before starting it.

hodlinator commented at 11:17 am on June 28, 2024:

*node 2

fjahr commented at 9:03 pm on July 30, 2024:

done

in src/init.cpp:913 in aba504759c outdated

907@@ -906,6 +908,11 @@ bool AppInitParameterInteraction(const ArgsManager& args)
908         return InitError(errors);
909     }
910 
911+    // Testnet3 deprecation warning
912+    if (chain == ChainType::TESTNET) {
913+        LogPrintf("Warning: Support for testnet3 is deprecated and will be removed in an upcoming release. Consider switching to testnet4.\n");

hodlinator commented at 11:45 am on June 28, 2024:

LogPrintf is deprecated since f7ce5ac08c669ac763e275bb7c82dcfb2b1b6c33 / August 2023. Should use LogInfo.

fjahr commented at 9:02 pm on July 30, 2024:

done

Emzy commented at 8:52 pm on June 28, 2024: contributor

Tested ACK aba5047

jlopp commented at 7:38 pm on June 30, 2024: contributor

ACK https://github.com/bitcoin/bitcoin/commit/aba504759caa13338aa574a7839d078b989fbf6d

murchandamus commented at 7:39 pm on July 1, 2024: contributor

Do blocks mined with the exception count towards the total work according to difficulty-1 or the actual difficulty? If it’s the latter, in testnet3 you could just invalidate the last block in the previous difficulty period with a difficulty-1 block. If it’s the former, I agree, the absence of this attack may indicate that I’m worrying too much.

(Side-note: I first read difficulty-1 as “difficulty minus 1” and got a bit confused but I think you mean the same as “difficulty=1”)

I needed to check this myself: Total work is calculated here and here (the formula is identical). This calls GetBlockProof() which uses the blocks nBits, not the actual target difficulty of the current period. So total work is only increased by difficulty-1.

Yeah I meant that as “difficulty one”, i.e. minimum difficulty.

in src/validation.cpp:4191 in aba504759c outdated

4182@@ -4183,6 +4183,18 @@ static bool ContextualCheckBlockHeader(const CBlockHeader& block, BlockValidatio
4183     if (block.GetBlockTime() <= pindexPrev->GetMedianTimePast())
4184         return state.Invalid(BlockValidationResult::BLOCK_INVALID_HEADER, "time-too-old", "block's timestamp is too early");
4185 
4186+    // Testnet4 only: Check timestamp against prev for difficulty-adjustment
4187+    // blocks to prevent timewarp attacks (see https://github.com/bitcoin/bitcoin/pull/15482).
4188+    if (consensusParams.hashGenesisBlock == uint256S("0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043")) {
4189+        // Check timestamp for the first block of each difficulty adjustment
4190+        // interval, except the genesis block.
4191+        if (pindexPrev->nHeight % consensusParams.DifficultyAdjustmentInterval() == consensusParams.DifficultyAdjustmentInterval() - 1) {

stickies-v commented at 6:06 pm on July 3, 2024:

Is there a reason this can’t be simplified to:

0        if (nHeight % consensusParams.DifficultyAdjustmentInterval() == 0) {

Sjors commented at 6:43 am on July 4, 2024:

This is copied from GetNextWorkRequired which doesn’t have the nHeight helper. The simplification makes sense, or we can add a helper function: #29775 (review)

fjahr commented at 9:32 pm on July 30, 2024:

I used the simplification here, thanks!

pablomartin4btc commented at 9:59 pm on July 3, 2024: member

Concept ACK

melvincarvalho commented at 4:35 am on July 9, 2024: none

I’ve been running this for about a month, observing the blocks closely. For the most part I have found it to work really well. Much easier to work with than testnet3. There is a strange occurrence that happens, not sure I would call it an “attack” but someone is shifting timestamps by exactly 20 minutes 1 second. Sometimes several times in a row. Generally it is up to 6 blocks at most. I am not sure what the exact trigger is, but it doesnt happen too often.

Here is an example of the longest that I have seen, which is 10, but the timestamps change half way through.

https://mempool.space/testnet4/block/000000004e280b94d1109ec8d94aa29d2b5171c85dbaab40b71c856616949a5a

T4 is going to be great for developers to work with, and I think it should be used more, for example with layer 2 solutions such as client validation, and colored coins. Thanks much for putting this PR together. Not sure if I’m allowed to ACK, but I support :)

Sjors commented at 6:33 am on July 9, 2024: member

someone is shifting timestamps by exactly 20 minutes 1 second

This lets them produce blocks at difficulty 1, i.e. with a CPU instead of an ASIC.

jsarenik commented at 6:52 am on July 9, 2024: none

someone is shifting timestamps by exactly 20 minutes 1 second

This lets them produce blocks at difficulty 1, i.e. with a CPU instead of an ASIC. @Sjors, how can I do that, too? What tools could be used to produce a block header with a particular shifted timestamp so that in the next step one could use bitcoin-util grind to perform proof of work on hex header string (i.e. CPU-mine such a block)?

What conditions need to be met to cause a blockchain reorg this way?

UPDATE: Reading bitcoin.it and developer.bitcoin.org already

Sjors commented at 9:05 am on July 9, 2024: member

@jsarenik if those resources don’t help, this would be a good Bitcoin Stack Exchange question.

garlonicon commented at 5:59 pm on July 9, 2024: none

Well, I already shared a diff somewhere: https://bitcointalk.org/index.php?topic=5496494.msg64205870#msg64205870

I simply don’t know a better way to compete with ASICs on testnets. Because testnet3 and testnet4 are the only networks I know of, where CPUs and ASICs can co-exist.

For example: if after 20 minutes, any block would be accepted, but at the same time, the real difficulty would be placed in the block header, then that kind of trick would be impossible, because ASICs would trivially provide more work, somewhere between the minimal, and the real difficulty. However, because you can mine at difficulty one, or at the real difficulty, and you cannot work on both cases at the same time, then the end result is that some ASIC can work hard, and get a block, but you can spend much less work on your CPU, and also get a similar reward.

not sure I would call it an “attack”

If it is “an attack”, then it can be easily patched: any ASIC-mined block can easily reorg some CPU-mined block. Also, ASIC miners can apply similar strategies as well, and then CPU miners will have no chance again (a good question is: should devs be able to CPU-mine those coins or not).

Another “solution” is to treat CPU-mined blocks as “weak blocks”, which could be intended to “always be reorged”. But then, it would push testnet even closer to the mainnet, if the real difficulty will be always preserved, and if chains of easier blocks will always vanish. Or: the whole testnet can consist of only weak blocks, then it would have no chance to gain any value, if everything will be always reorged, and if archival testing blocks would be available only as a stale blocks.

A lot of things to think about, but as long as testnet4 is not officially deployed, it can be adjusted if needed.

Edit: You can inspect closer tb1qvy67actlsslmvwpjzk9hnx6jf04g5y26sz5crx on testnet3 and testnet4. This is the address I use for testing. Some time ago, I heard that it is harder and harder for developers to get some new test coins. You can check, how many blocks were generated on them (only with CPU! No ASIC used!), and answer a question “is it enough for development?”.

melvincarvalho commented at 6:15 pm on July 9, 2024: none

Great analysis, thanks

I have seen two instances where the diff drops to 1. The first is when the mining dries up for 20 minutes, and then someone will mine a block, just to keep the chain going (normally wiz :)).

The other is when people are deliberately propagating blocks with a future timestamp, up to about 6 blocks. Coincidentally 6 * 20 = 2 hours. I do not know what triggers this.

The 20 minute rule when used properly is quite nice as it keeps the chain flowing. The future spoofed timestamp blocks dont happen too often, so are not too disruptive.

Regarding reorgs, it would make sense to just build on the 20 minute block (as a kind of etiquette) with an asic and not reorg. The spoofed timestamps might be more fair game for a reorg. But this is not yet happening.

in src/util/chaintype.cpp:19 in aba504759c outdated

14@@ -15,6 +15,8 @@ std::string ChainTypeToString(ChainType chain)
15         return "main";
16     case ChainType::TESTNET:
17         return "test";
18+    case ChainType::TESTNET4:
19+        return "testnet4";

Kixunil commented at 9:36 am on July 26, 2024:

Can this be test4? The inconsistency looks ridiculous. Although signet has a similar problem already.

fjahr commented at 8:04 pm on July 30, 2024:

I’m not so sure. I think anywhere we print this string testnet4 it is nicer because it’s closer to what users actually say. I also interpret the choice that was made for signet to be what we want to do going forward, since it was the latest addition on the list. So I tent to leave this as is unless there are other arguments than the inconsistency.

Kixunil commented at 5:24 am on July 31, 2024:

Well, I guess test should’ve been named testnet3 from the start. Maybe this is an improvement but I guess it will also confuse people who are used to the old naming or think they can just change test to test4 in their configs. Maybe this should be explicitly called out in release post or something.

fjahr commented at 5:57 pm on July 31, 2024:

Ok, I will keep in mind making it explicit in the release notes/docs.

murchandamus commented at 4:56 pm on August 5, 2024:

Is there a separate PR with Release Notes already or was the plan to do that in a separate PR?

Retropex commented at 5:30 pm on July 29, 2024: none

tACK aba504759caa13338aa574a7839d078b989fbf6d

DrahtBot requested review from pablomartin4btc on Jul 29, 2024

fjahr force-pushed on Jul 30, 2024

fjahr commented at 8:14 pm on August 2, 2024: contributor

The BIP has been merged. The other outstanding matters have been discussed in the latest IRC meeting and it appears that there is nothing left that is considered blocking. This would be a great time for everyone to test the latest version of the PR here once again so that we can merge it before the feature freeze on August 12th. Thanks everyone!

jsarenik commented at 7:20 am on August 4, 2024: none

Tested ACK d4dedbc1029a

Running on anyone.eu.org also for mainnet and signet (and two custom signets) ever since testing this branch and everything works well. Thank you @fjahr !

DrahtBot requested review from Sjors on Aug 4, 2024

craigraw commented at 7:25 am on August 5, 2024: none

tACK d4dedbc102

in test/functional/p2p_dos_header_tree.py:23 in 6001b2b4ec outdated

19@@ -20,6 +20,7 @@
20 class RejectLowDifficultyHeadersTest(BitcoinTestFramework):
21     def set_test_params(self):
22         self.setup_clean_chain = True
23+        # TODO: Update to testnet4 when a checkpoint is available

murchandamus commented at 4:58 pm on August 5, 2024:

I don’t think we need checkpoints after v24.0 introduced Header Pre-Syncing

fjahr commented at 11:48 pm on August 5, 2024:

Removed the TODO, this test file will need to change anyway when Testnet3 is ready to be removed.

in test/functional/wallet_crosschain.py:79 in 6001b2b4ec outdated

68             assert_raises_rpc_error(-4, 'Wallet files should not be reused across chains.', self.nodes[1].loadwallet, node0_wallet)
69+            assert_raises_rpc_error(-4, 'Wallet files should not be reused across chains.', self.nodes[2].loadwallet, node0_wallet)
70             assert_raises_rpc_error(-4, 'Wallet files should not be reused across chains.', self.nodes[0].restorewallet, 'w', node1_wallet_backup)
71+            assert_raises_rpc_error(-4, 'Wallet files should not be reused across chains.', self.nodes[0].restorewallet, 'w', node2_wallet_backup)
72             assert_raises_rpc_error(-4, 'Wallet files should not be reused across chains.', self.nodes[1].restorewallet, 'w', node0_wallet_backup)
73+            assert_raises_rpc_error(-4, 'Wallet files should not be reused across chains.', self.nodes[2].restorewallet, 'w', node0_wallet_backup)

murchandamus commented at 5:01 pm on August 5, 2024:

Assuming that Testnet3 and Testnet4 backups should not be compatible, perhaps it would be good to also test that the node1_wallet_backup cannot be loaded with nodes[2] and vice versa.

fjahr commented at 11:48 pm on August 5, 2024:

done, this also uncovered a naming conflict in a file path above!

in src/pow.cpp:66 in 7368d771d2 outdated

60@@ -61,7 +61,19 @@ unsigned int CalculateNextWorkRequired(const CBlockIndex* pindexLast, int64_t nF
61     // Retarget
62     const arith_uint256 bnPowLimit = UintToArith256(params.powLimit);
63     arith_uint256 bnNew;
64-    bnNew.SetCompact(pindexLast->nBits);
65+
66+    // Special difficulty rule for Testnet4
67+    if (params.hashGenesisBlock == uint256S("0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043")) {

murchandamus commented at 5:06 pm on August 5, 2024:

I think I have seen someone mention something along the lines before, but it would be nice to have a global constant for the testnet 4 Genesis Block Hash instead of it being hard-coded like this.

sipa commented at 8:38 pm on August 5, 2024:

Even with a constant, this seems like the wrong approach. The PoW code shouldn’t be aware of what networks exist at all, much less be aware of the testnet4 genesis block.

Add a property to the chainparams, similar to but separate from fPowAllowMinDifficultyBlocks to control whether a particular chain has this rule active.

fjahr commented at 11:48 pm on August 5, 2024:

I had planned to do this as a follow-up but I added a new property enforce_BIP94 to handle this now.

in src/validation.cpp:4191 in d4dedbc102 outdated

4187+    // blocks to prevent timewarp attacks (see https://github.com/bitcoin/bitcoin/pull/15482).
4188+    if (consensusParams.hashGenesisBlock == uint256S("0x00000000da84f2bafbbc53dee25a72ae507ff4914b867c565be350b0da8bf043")) {
4189+        // Check timestamp for the first block of each difficulty adjustment
4190+        // interval, except the genesis block.
4191+        if (nHeight % consensusParams.DifficultyAdjustmentInterval() == 0) {
4192+            if (block.GetBlockTime() < pindexPrev->GetBlockTime() - 60 * 60 * 2) {

murchandamus commented at 5:25 pm on August 5, 2024:

I was a bit surprised that you allowed two hours earlier, because I thought the original proposal was “allow[ing] nTime to go backwards by 600 seconds” (see Great Consensus Cleanup: Discussion), but actually 2h makes more sense to me, because that allows a miner to use the actual time even if the author of the last block in the prior difficulty period postdated it 2h into the future.

achow101 commented at 8:33 pm on August 5, 2024:

The BIP says 600 seconds. Regardless of which value is ultimately chosen, the BIP and this PR should be consistent with each other.

fjahr commented at 9:13 pm on August 5, 2024:

I have opened a PR to align the BIP to the PR here: https://github.com/bitcoin/bips/pull/1658

murchandamus commented at 5:25 pm on August 5, 2024: contributor

tACK d4dedbc1029a59d9df0ceaefe5b6645d39d7eaa1

I reviewed the code and successfully synced a node with Testnet4 from this branch.

melvincarvalho commented at 6:38 pm on August 5, 2024: none

tACK d4dedbc

achow101 commented at 8:03 pm on August 5, 2024: member

@kcalvinalvin Does the existence of the BIP change your NACK?

in src/kernel/chainparams.cpp:361 in d4dedbc102 outdated

356+        base58Prefixes[EXT_PUBLIC_KEY] = {0x04, 0x35, 0x87, 0xCF};
357+        base58Prefixes[EXT_SECRET_KEY] = {0x04, 0x35, 0x83, 0x94};
358+
359+        bech32_hrp = "tb";
360+
361+        vFixedSeeds = std::vector<uint8_t>(std::begin(chainparams_seed_test), std::end(chainparams_seed_test));

achow101 commented at 8:20 pm on August 5, 2024:

In 6001b2b4ec7b38fb95a37f3476b015aa5348acc3 “testnet: Introduce Testnet4”

This is going to reuse the testnet3 fixed seeds. As this is a completely different network, I don’t think that’s a good idea.

The fixed seeds generation scripts probably should be updated too.

fjahr commented at 11:47 pm on August 5, 2024:

I changed the script and added a small list of nodes, just as a placeholder for testing since this will be changed once again with the release.

achow101 commented at 8:34 pm on August 5, 2024: member

There also appears to be a silent merge conflict with master:

0kernel/chainparams.cpp: In constructor ‘CTestNet4Params::CTestNet4Params()’:
1kernel/chainparams.cpp:392:9: error: ‘ChainTxData’ has no non-static data member named ‘nTxCount’
2  392 |         };
3      |         ^

murchandamus commented at 8:38 pm on August 5, 2024: contributor

@kcalvinalvin Does the existence of the BIP change your N-A-C-K? @achow101: You are now incorrectly being listed as having n-A-C-Ked this PR. :grin:

achow101 added this to the milestone 28.0 on Aug 5, 2024

testnet: Introduce Testnet4 74a04f9e7a

testnet: Add Testnet4 difficulty adjustment rules fix 0100907ca1

testnet: Add timewarp attack prevention for Testnet4 6bfa26048d

fjahr force-pushed on Aug 5, 2024

fjahr commented at 11:45 pm on August 5, 2024: contributor

Is there a separate PR with Release Notes already or was the plan to do that in a separate PR?

I will do a separate PR as soon as this is merged. I want to make sure first that this actually gets merged in time and the wording in release notes tends to get bikeshedded so we can still do that after the feature freeze and it doesn’t need to hold up the review here.

There also appears to be a silent merge conflict with master

Rebased and fixed

jsarenik commented at 10:57 am on August 6, 2024: none

tACK 6bfa26048dba

DrahtBot requested review from murchandamus on Aug 6, 2024

in src/chainparamsbase.cpp:20 in 74a04f9e7a outdated

16@@ -17,7 +17,8 @@ void SetupChainParamsBaseOptions(ArgsManager& argsman)
17     argsman.AddArg("-regtest", "Enter regression test mode, which uses a special chain in which blocks can be solved instantly. "
18                  "This is intended for regression testing tools and app development. Equivalent to -chain=regtest.", ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::CHAINPARAMS);
19     argsman.AddArg("-testactivationheight=name@height.", "Set the activation height of 'name' (segwit, bip34, dersig, cltv, csv). (regtest-only)", ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::DEBUG_TEST);
20-    argsman.AddArg("-testnet", "Use the test chain. Equivalent to -chain=test.", ArgsManager::ALLOW_ANY, OptionsCategory::CHAINPARAMS);
21+    argsman.AddArg("-testnet", "Use the testnet3 chain. Equivalent to -chain=test. Support for testnet3 is deprecated and will be removed with the next release. Consider moving to testnet4 now by using -testnet4.", ArgsManager::ALLOW_ANY, OptionsCategory::CHAINPARAMS);

murchandamus commented at 7:32 pm on August 6, 2024:

Are we sure that we will drop support next release? Otherwise it might be better to be a bit less specific:

0    argsman.AddArg("-testnet", "Use the testnet3 chain. Equivalent to -chain=test. Support for testnet3 is deprecated and will be removed in a future release. Consider moving to testnet4 now by using -testnet4.", ArgsManager::ALLOW_ANY, OptionsCategory::CHAINPARAMS);

fjahr commented at 8:21 pm on August 6, 2024:

I think it’s better to be precise and we could still shift it if that becomes absolutely necessary for a good reason. It seems better to have people prepare for it to be dropped in the next release unless there is good reason brought forward for shifting. This is better than the alternative that we are vague and then people claim last minute we can not do it in the next release already because we were not clear enough. I don’t want to give an invite to the “Testnet3 is the real testnet” crowd :)

murchandamus commented at 1:15 pm on August 7, 2024:

I was thinking of business to business companies using Testnet3 as a development environment. They may have dozens of enterprise clients that have built Testing infrastructure on Testnet3 and serve transaction data by interfacing with Bitcoin Core. Even if the service provider is willing to upgrade, it can be a pain to get the customers to update (speaking from experience). I’m fine with deprecating Testnet3, but I would suggest that the option should only be removed from Bitcoin Core with v29.

I also noticed that the two mentions of future deprecation are not completely consistent, but that could be addressed in a follow-up or isn’t that big a deal anyhow.

src/chainparamsbase.cpp:

0   argsman.AddArg("-testnet", "Use the testnet3 chain. Equivalent to -chain=test. Support for testnet3 is deprecated and will be removed with the next release. Consider moving to testnet4 now by using -testnet4.", ArgsManager::ALLOW_ANY, OptionsCategory::CHAINPARAMS);

src/init.cpp:

0    // Testnet3 deprecation warning
1    if (chain == ChainType::TESTNET) {
2        LogInfo("Warning: Support for testnet3 is deprecated and will be removed in an upcoming release. Consider switching to testnet4.\n");
3    }

fjahr commented at 1:27 pm on August 7, 2024:

Ok, unless I have to re-touch here I will make it consistent in the follow-up where I add the release notes.

I’m fine with deprecating Testnet3, but I would suggest that the option should only be removed from Bitcoin Core with v29.

That’s my intention, since we are adding Testnet4 for v28 the next version I am referring to is v29. Or were you concerned with removing it from master right after v28 is tagged? I would be fine to aim for a removal with v30 instead giving ~1 year of deprecation period. But I really think it serves everyone best if we make are clear on the intended timeline right away to manage expectations and invite people to speak up as early as possible if they have an issue with it. I prefer a faster timeline but indeed I don’t have a good overview of such b2b Testnet3 infrastructure and how much work it is to move it.

sipa commented at 1:33 pm on August 7, 2024:

I would suggest:

In v28: add testnet4, and a warning about testnet3 being “removed in a future version” (this PR).
In v29: actually deprecate testnet3, but still allow it with an explicit -deprecatedrpc=testnet3 (so someone can still use it, but must deliberately enable the option if they really need it). This is generally the approach we take with removing features from RPC. This is a whole network and not just an RPC command, but close enough.
In v30: testnet3 is gone, assuming no riots and pitchforks after v29.

fjahr commented at 1:38 pm on August 7, 2024:

@sipa Timeline sounds good to me, but why not be precise in the v28 warning and say something like “deprecated in the next version and removed in the version after”? I don’t see any downside in being clear on our intentions.

sipa commented at 1:57 pm on August 7, 2024:

@fjahr I guess because I feel it’s hard to make concrete promises about something that far out. If we know we will remove testnet3 in v30, we should just deprecate it in v28 already (including a -deprecatedrpc requirement), rather than saying it will be deprecated. If not, the best we can do is announce an intent that it will be removed in the future, without concrete timing.

murchandamus commented at 1:59 pm on August 7, 2024:

Ok, unless I have to re-touch here I will make it consistent in the follow-up where I add the release notes.

I’m fine with deprecating Testnet3, but I would suggest that the option should only be removed from Bitcoin Core with v29.

That’s my intention, since we are adding Testnet4 for v28 the next version I am referring to is v29. Or were you concerned with removing it from master right after v28 is tagged? I would be fine to aim for a removal with v30 instead giving ~1 year of deprecation period. But I really think it serves everyone best if we make are clear on the intended timeline right away to manage expectations and invite people to speak up as early as possible if they have an issue with it. I prefer a faster timeline but indeed I don’t have a good overview of such b2b Testnet3 infrastructure and how much work it is to move it.

Err, I meant in two versions, so not in 29, but in 30. :facepalm:

fjahr commented at 2:45 pm on August 7, 2024:

I guess because I feel it’s hard to make concrete promises about something that far out.

Ok, I don’t see it as a promise but as a declared intention. Users don’t gain anything from us removing a feature, they could all just stop using it without it being removed so we don’t need to promise them the removal. We rather do that for ourselves to relieve ourselves of the maintenance burden. We intended to do that for a specific version unless someone comes forward and makes a case that convinces us to carry the burden a bit longer.

If we know we will remove testnet3 in v30, we should just deprecate it in v28 already (including a -deprecatedrpc requirement), rather than saying it will be deprecated. If not, the best we can do is announce an intent that it will be removed in the future, without concrete timing.

Not deprecating right away has the (small) convenience upside for users that they don’t have to set the -deprecatedrpc option for another 6 months. That’s a bit softer off-ramp. I think it’s reasonable to go through the steps/timeline that you outlined and if we largely agree that’s a good path I think we can tell users about it too.

Anyway, I would suggest we continue this in the follow-up. I will pick this up and make a suggestion for an alternative wording which we can still discuss after feature freeze.

instagibbs commented at 4:08 pm on August 7, 2024:

If it’s being considered for testnet3, I’d recommend a fairly long deprecation cycle, at least 2 releases.

boring877 commented at 6:01 pm on August 7, 2024:

why not just keep it ? why act like communist and remove things you desire ~

alot of data and infrastructure already on testnet3, why force people your own desires ~

boring877 commented at 6:04 pm on August 7, 2024:

@fjahr I guess because I feel it’s hard to make concrete promises about something that far out. If we know we will remove testnet3 in v30, we should just deprecate it in v28 already (including a -deprecatedrpc requirement), rather than saying it will be deprecated. If not, the best we can do is announce an intent that it will be removed in the future, without concrete timing.

what you mean testnet3 gone ? you seriously wants to delete it~ you can use testnet3 as reference for old scripts thats been used …

stop actting like you make decisions ~

murchandamus commented at 6:07 pm on August 7, 2024:

what you mean testnet3 gone ? you seriously wants to delete it~ you can use testnet3 as reference for old scripts thats been used …

No worries, you can continue to use Testnet 3 as long as you desire. It’s just not going to be supported by Bitcoin Core starting with some upcoming version.

fjahr commented at 6:12 pm on August 7, 2024:

@fjahr I guess because I feel it’s hard to make concrete promises about something that far out. If we know we will remove testnet3 in v30, we should just deprecate it in v28 already (including a -deprecatedrpc requirement), rather than saying it will be deprecated. If not, the best we can do is announce an intent that it will be removed in the future, without concrete timing.

what you mean testnet3 gone ? you seriously wants to delete it~ you can use testnet3 as reference for old scripts thats been used …

stop actting like you make decisions ~

I don’t want to give an invite to the “Testnet3 is the real testnet” crowd :)

Q.E.D.

pinheadmz commented at 6:50 pm on August 7, 2024:

@boring877 Your comments are off topic and inflammatory, resulting in a 24 ban from the organization to cool off. I’m not going to delete your comments since this PR is closed anyway, and contributors offered informational replies. To appeal a moderation decision, open an issue in bitcoin-core/meta

melvincarvalho commented at 8:13 pm on August 7, 2024:

Testnet3 is always available from the right branch and can be forked by any interested community.

Consensus is that Testnet4 will be better for Bitcoin testing.

There is still value in Testnet3, especially for observing Bitcoin beyond 2.8 million blocks, understanding reorgs, and exploring reorg protection strategies.

If there’s a community for Testnet3, fork it, create a website, and get folks to run nodes.

Bitcoin devs have decided that Testnet3 is getting harder to use, and Testnet4 should be easier. If you want to keep the Testnet3 experiment going, please do. I think it’s worthwhile, but the approach above is a reasonable path, IMHO.

murchandamus approved

murchandamus commented at 7:47 pm on August 6, 2024: contributor

tACK 6bfa26048dbafb91e9ca63ea8d3960271e798098

Reviewed code changes and resynced the node from scratch

achow101 commented at 4:49 pm on August 7, 2024: member

ACK 6bfa26048dbafb91e9ca63ea8d3960271e798098

achow101 merged this on Aug 7, 2024

achow101 closed this on Aug 7, 2024

Sjors commented at 9:57 am on August 8, 2024: member

Post-merge light re-utACK

Happy to see enforce_BIP94 replace the hardcoded genesis block.

Followup suggestion: let’s make 60 * 60 * 2 a constant and add a static_assert that it’s equal to MAX_FUTURE_BLOCK_TIME. They are not the same thing, since time-timewarp-attack is (testnet4) consensus whereas time-too-new is not. But if we ever change one without changing the other, it can cause a chain split.

I didn’t test the new seeds generation script.

zawy12 commented at 3:41 pm on August 8, 2024: none

There’s usually a “timewarp” attack if

monotonic timestamps are not enforced,
timespan limits are used (1/4 & 4x in BTC), and
miner has >50% hashrate.

It doesn’t depend on the 2015/2016 “hole” in bitcoin’s measurement of nActualtimespan.

The difficulty fix seems to still have a hole if nActualtimespan can be negative. Granted, the attack below requires 16 weeks. The fix is an improvement because the 3 conditions above usually require only 2.5 difficulty adjustment windows to get an infinite number of blocks (even without the 2015 “hole”).

MTP and this fix are indirect ways of enforcing monotonic timestamps. Monotonicity comes from Leslie Lamport’s 1978 paper that applies to all concepts of “distributed consensus” without regard to the algorithm. He discusses the physical impossibility of travelling to the past (but not the future). In our case, this means a meaningful (non-negative) measurement of work. Height monotonicity and even proof of proposed block ordering via the monotonicity of hash references aren’t enough because they aren’t used to measure work like timestamps.

Condition 2 is a problem because it subverts the math of the work measurement.

Lamport also proves timestamp accuracy must be much smaller than the length of consensus rounds (a block). Bitcoin consensus allowing much greater error is what enables selfish mining (here’s my suggested fix). Selfish miners have to assign a timestamp before they know when they need to release the block. Honest miners can exploit this lack of knowledge by enforcing better synchronization (accurate timestamps). The synchronization is relaxed if PoW indicates a partition has occurred

To summarize this attack: attacker does a private mine to assign future timestamps at the 4x timespan limit two times in a row (two 2016 periods) while keeping the MTP held back. The timestamp at the end of the 2nd period and beginning of 3rd period is 8 * 2016 * 600 seconds into the future (16 weeks). The 3rd period thereby has 1/16 of the original difficulty (16x the target). Attacker then assigns a past timestamp at end of 3rd period (which is allowed by the MTP being held back) which increases difficulty back to 1/4 of the original difficulty (this causes nActualtimespan to be negative). This allows the 4th period to have the first timestamp in the past and its last timestamp at the same 8 * 2016 * 600 future time as before, bringing difficulty back down to 1/16. He keeps doing this pattern in the last 2 periods until his current time equals the 8 * 2016 * 600 future timestamp and then he releases the resulting 41,500 blocks when he should have gotten only 8,064 blocks.

correction: in “actual time” column I have +1/2 at the end of the equation in one place and +1/8 in two places that should be +T/2 and +T/8 which can be deduced by the context.

Zcash/Digishield (& maybe Grin copied them close enough) is the only difficulty algorithm I know of that isn’t subject to this attack despite the fulfilling the 3 conditions but it’s because their difficulty window is set to the MTP which is how monotonicity is indirectly enforced in the consensus. Setting MTP = 1 instead of 11 could be a BIP to enforce monotonicity. I’m just kidding. I don’t want to get yelled at.

murchandamus commented at 6:14 pm on August 8, 2024: contributor

@zawy12: I think this gets a bit off-topic here, perhaps it would be better to post about your timewarp scenario to the mailing list or Delving Bitcoin.

zawy12 commented at 8:04 pm on August 8, 2024: none

@murchandamus I was only interested in letting those calling it a fix to be aware that it’s not a fix.

cculianu referenced this in commit 8211395f67 on Aug 8, 2024

fjahr commented at 8:22 pm on August 8, 2024: contributor

Followup suggestion: let’s make 60 * 60 * 2 a constant and add a static_assert that it’s equal to MAX_FUTURE_BLOCK_TIME. They are not the same thing, since time-timewarp-attack is (testnet4) consensus whereas time-too-new is not. But if we ever change one without changing the other, it can cause a chain split.

Well, sounds like we could also just use MAX_FUTURE_BLOCK_TIME there then. I have addressed this now in #30604.

murchandamus commented at 9:56 pm on August 8, 2024: contributor

@fjahr: Having thought a bit more about the attack scenario @zawy12 describes, it might be useful to additionally require that the first block of a difficulty period N has a lower timestamp than the last block of the difficulty period N.

I’ll respond to Zawy on Delving, if the topic is posted there, or post there myself in the next few days to explain my thinking.

zawy12 commented at 11:13 pm on August 8, 2024: none

As @murchandamus just said, preventing nActualtimespan from going negative could stop my attack above.

An alternate fix to restricting nActualtimespan is for no block to be more than 2 hours and 80 minutes before its parent block. The 80 minutes is to provide protection from a dual Sybil attack on the 40 minute allowable error in peer time. Allowing & relying on peer time at all is another fundamental consensus error.

I prefer monotonicity on every block & removal of timespan limits but I know from the past that those are a “no go” with everyone, as is accurate timestamp enforcement. Restricting timestamps to less than 2 hr 80 minutes for every block is a weak form of monotonicity.

Me, Greg Maxwell, Brian Cohen, Jacob Eliosoff, and Johnson Lau discussed the current type of fix in email 6 years ago after a August 2018 mailing list initiation. GM opened discussion but then fell silent (“It’s not too important, and can be fixed quickly and easily if it happens”). J Lau suggested 1 day max in the past for the 1st block of each period and provided reasoning that it was a good method. Bram concurred and proposed 3 hours to reduce the max 7% manipulation Lau had calculated. I proposed doing it for every block even though at that time I had not thought of the attack above, but felt something like it was possible. Bram said “maybe every block is a good idea” by his own reasoning. The others didn’t object or acknowledge. Recounting this is to show we’re not suddenly making things up.

achow101 referenced this in commit 389cf32aca on Aug 9, 2024

glozow referenced this in commit 2f7d9aec4d on Aug 15, 2024

hodlinator referenced this in commit 737a1951a3 on Aug 27, 2024

hodlinator referenced this in commit 49f9b645ea on Aug 27, 2024

fanquake referenced this in commit f4a10911c7 on Aug 28, 2024

melvincarvalho commented at 8:15 am on September 22, 2024: none

Sorry if this is the wrong spot for feedback. Post-merge, Testnet4 seems solid overall. One issue: difficulty appears to be rising exponentially, likely due to someone spoofing timestamps and pushing in 6 blocks every 20 minutes. I could be off here, but if difficulty keeps spiking, mining might become impractical, making reorgs easier. A possible fix: setting up a pool that builds on genuine 20-minute blocks (not spoofed ones), and having some miners point their hashpower to it.

garlonicon commented at 6:20 pm on September 22, 2024: none

difficulty appears to be rising exponentially

Not necessarily. There are many CPU-mined blocks, but the chainwork is not rising by that much. And CPU miners are not only producing more blocks than needed: they are also pushing timestamps forward. Which means, that if it will be too hard to mine, then:

More people will be mining with CPU difficulty (which is good, because then, everything is left to lottery-based network propagation).
The difficulty will stop rising, because if you have 20 minutes per block (instead of scheduled 10 minutes), then it will start decreasing.
Only one ASIC block per difficulty retarget is needed. And testnet4 consensus is set up, to reach specifically this case, so nobody should be surprised. The rules were simply set, to specifically create that kind of outcome.

For example: what you can currently see, is just a lot of CPU blocks, mined by https://mempool.space/testnet4/address/tb1q3u8f5899ymkatx69h0n3sw0qpalgwdmrcj80dm

And also note, that ASIC miners can always mine a single block, and revert a lot of CPU-mined blocks in this way. And: if a lot of fees will be there, spread over many blocks, then sooner or later, reorging hundreds of CPU-mined blocks, and collecting the highest fees, will be profitable enough, to trigger such reorg.

And then, CPU-mined blocks will be, what they always should: just a weak blocks, to test things, to propagate transactions faster, and to vanish, and be reorged, when someone will honestly start mining with the network difficulty. Then, maybe we will get closer to the desired outcome, where all testnet blocks are temporary, and quickly becoming stale. Because in practice, all of that mining power, should simply be contributed towards mainnet, and should allow temporarily getting some coins, which would then be reorged, to lose any value, that someone could assign to them; and to crash all kind of trading, that could happen on test coins.

if difficulty keeps spiking, mining might become impractical

Testnet4 guarantees, that ASICs are needed. If no ASIC is there, then all CPU miners will be stuck, after mining 2016 blocks. Which means, that it is guaranteed to mine a single ASIC block per two weeks, no matter what. And if CPU miners will be blocked by raised difficulty, then it will take more than two weeks, to mine a single ASIC block, and then, the difficulty will start falling.

setting up a pool

Note that if you use OP_SIZE on DER signatures, then you can send coins to Proof of Work, without using OP_CAT. In general, if you want to ensure a given reward, for a given amount of Proof of Work, then this is the way to go. By setting up a pool, you will just jump into that race, and the difficulty will skyrocket, because then, the network will become even more similar to the mainnet.

You don’t need pools on testnets. Mining pools are based on shares. And a share is just a regular block, with lowered difficulty. And in testnet4, you already have that, because you can switch into CPU difficulty, at any time, and then just improve your network propagation.

fuleru commented at 1:04 am on October 17, 2024: none

It’s a bit scary. Currently, testnet4 block generation is controlled. https://mempool.space/testnet4/address/tb1q2dsc94zq40nwnz27w5rxljwllutnwjtlxk44fz

fjahr commented at 4:43 pm on October 19, 2024: contributor

Re: the recent discussion here, see #31117

darosior commented at 4:09 pm on November 26, 2024: member

One comment on the timewarp fix. It’s OK in this specific case to have the check in ContextualCheckBlockHeader() since this is a new network and a previous version could not conceivably have let in an invalid header. However for a patch to an existing network, we should consider something akin to what Matt did in #15482 for the 64 bytes txs check. See also this comment in ConnectBlock(): https://github.com/bitcoin/bitcoin/blob/f7144b24be09e7db2173a0b15d73f99a08b98118/src/validation.cpp#L2437-L2449

Detroitin commented at 1:01 pm on December 2, 2024: none

huh! well I guess every block I hit that is 194.47 million Dif. hopefully helps.. I did see the pools “luck” go up LOL. im knowingly fighting a loosing battle. but i hate seeing a bully..

Testnet4 including PoW difficulty adjustment fix #29775

Code Coverage

Reviews

Conflicts