Testnet4 consensus failure due to timewarp related "softfork"

Sjors commented at 12:25 pm on September 2, 2024: member

#30647 was effectively a soft-fork on testnet4 with no activation plan.

It reduced the timewarp mitigation allowance from 7200 to 600 seconds.

It’s enforced by folks running v28.0rc1 and recent master, but not by miners.

This causes new nodes to be stuck at block 42,335: https://mempool.space/testnet4/block/00000000542792e54a720567ba66157d48cdae7bfd01c1b678d0f07a2ed56e99

As observed on BitcoinTalk: https://bitcointalk.org/index.php?topic=5499150.msg64488234#msg64488234 and pointed out by @garlonicon in #30614 (comment)

The simplest solution would for a majority of testnet4 hash power to upgrade to the release candidate and call:

bitcoin-cli -testnet4 invalidateblock 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242
bitcoin-cli -testnet4 reconsiderblock 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242

This would cause a 16+ hour reorg for any older testnet4 node that’s not stuck. Users who upgraded their node after this block was created, and accepted by their node under the old rules, can either wait for this reorg to happen, or also use the above commands.

Sjors commented at 12:34 pm on September 2, 2024: member

Note that the block that upgraded nodes will get stuck on - 42,335 - has difficulty 1. It was probably mined by setting the machine clock 20 minutes in the future. The original testnet4 rules allowed the next block to use the real time. But under the new rules it needs to be 10 minutes in the future (which #30681 takes care of).

Sjors commented at 12:45 pm on September 2, 2024: member

The invalidateblock -> reconsiderblock block approach doesn’t work, because the timewarp attack is only checked in ContextualCheckBlockHeader. According to the code documentation -reindex-chainstate won’t work either.

bitcoind -reindex does the trick, and only takes a minute on this tiny chain fortunately.

Sjors commented at 12:50 pm on September 2, 2024: member

cc @mempool

maflcko added the label Tests on Sep 2, 2024

1440000bytes commented at 8:51 pm on September 2, 2024: none

cc @mempool

I don’t think this notified anyone

inyellowbus commented at 7:27 am on September 3, 2024: none

bitcoind -reindex does the trick, and only takes a minute on this tiny chain fortunately.

Reindex doesn’t work for me. After starting the node, everything still stacks on block 42335. Maybe I need to do something else before starting the reindex? Or register a connection to some explicit node?

Sjors commented at 7:38 am on September 3, 2024: member

everything still stacks on block 42335.

This is expected. You’ll have to wait for a miner to do the same. Welcome to the testnet4 UASF :-)

achow101 commented at 3:45 pm on September 3, 2024: member

Anyone have an asic to reorg that?

Sjors commented at 5:07 pm on September 3, 2024: member

It’s too hot to turn on my S9 :-)

russeree commented at 11:29 pm on September 3, 2024: contributor

Anyone have an asic to reorg that?

MARA hash on the way! Luxor also said they are willing to help as well. Shout out to Nick to even turning me onto this problem.

hashbender commented at 11:58 pm on September 3, 2024: none

You have my hashpower 🫡

1440000bytes commented at 0:38 am on September 4, 2024: none

You have my hashpower 🫡

This will help

wiz commented at 1:07 am on September 4, 2024: contributor

ok will do

hashbender commented at 1:12 am on September 4, 2024: none

Here’s my last chain tip in case you want to verify the correct chain. This is from Bitcoin Core version v28.0rc1

0hashbender@macbook-pro-124: bitcoin-cli -testnet4 getchaintips
1[
2  {
3    "height": 42375,
4    "hash": "00000000000000306de15dd9ee1e6af0461e5bdef12d6dc6be212ca7f5296fa1",
5    "branchlen": 0,
6    "status": "active"
7  }
8]

russeree commented at 1:25 am on September 4, 2024: contributor

Here’s my last chain tip in case you want to verify the correct chain. This is from Bitcoin Core version v28.0rc1

0hashbender@macbook-pro-124: bitcoin-cli -testnet4 getchaintips
1[
2  {
3    "height": 42375,
4    "hash": "00000000000000306de15dd9ee1e6af0461e5bdef12d6dc6be212ca7f5296fa1",
5    "branchlen": 0,
6    "status": "active"
7  }
8]

Got 500TH/s pointed toward T4 UASF

garlonicon commented at 1:14 pm on September 4, 2024: none

I think there is a small risk, that we will see a huge double-spending attempt in testnet4, because after block 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242, there are transactions in the old chain, which are not picked by nodes, mining the new chain. Which means, that suddenly, when the deep reorg will revert hundreds of blocks, all of those transactions will suddenly become unconfirmed.

So, if anyone has any coins, which were received after block number 42335, then they can be easily double-spent on the new chain, by anyone, just because all of those transactions will vanish. Which means, that as long, as the chain is not yet reorged, people can submit their double spending transactions to the new chain, get them confirmed, and successfully reverse a payment, which had hundreds of confirmations, when a huge chain reorg will kick in.

Also, there is some incentive to download transactions from old node, and pass it secretly to the new node (without broadcasting to the network), because in this way, a miner can get bigger coinbase amount, than other miners.

remcoros commented at 6:09 pm on September 4, 2024: none

I think there is a small risk, that we will see a huge double-spending attempt in testnet4, because after block 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242, there are transactions in the old chain, which are not picked by nodes, mining the new chain. Which means, that suddenly, when the deep reorg will revert hundreds of blocks, all of those transactions will suddenly become unconfirmed.

So, if anyone has any coins, which were received after block number 42335, then they can be easily double-spent on the new chain, by anyone, just because all of those transactions will vanish. Which means, that as long, as the chain is not yet reorged, people can submit their double spending transactions to the new chain, get them confirmed, and successfully reverse a payment, which had hundreds of confirmations, when a huge chain reorg will kick in.

Also, there is some incentive to download transactions from old node, and pass it secretly to the new node (without broadcasting to the network), because in this way, a miner can get bigger coinbase amount, than other miners.

I don’t think there’s an incentive problem. Coins have no value and it’s easy enough to cpu mine some future blocks if you need tn4 coins.

While it might be annoying for users/developers to have their transactions rolled back, that’s a small price to pay for living on the edge and running a node with unmerged code, while there were still discussions going on about a consensus rule change. Bitcoin Core 28.0-RC1 is the first ‘official’ release with testnet4 support and that includes the time-warp fix.

russeree commented at 6:43 pm on September 4, 2024: contributor

I think there is a small risk, that we will see a huge double-spending attempt in testnet4, because after block 00000000000000263393ce5f648afd53676f13d360cc9f264b89351623bf1242, there are transactions in the old chain, which are not picked by nodes, mining the new chain. Which means, that suddenly, when the deep reorg will revert hundreds of blocks, all of those transactions will suddenly become unconfirmed. So, if anyone has any coins, which were received after block number 42335, then they can be easily double-spent on the new chain, by anyone, just because all of those transactions will vanish. Which means, that as long, as the chain is not yet reorged, people can submit their double spending transactions to the new chain, get them confirmed, and successfully reverse a payment, which had hundreds of confirmations, when a huge chain reorg will kick in. Also, there is some incentive to download transactions from old node, and pass it secretly to the new node (without broadcasting to the network), because in this way, a miner can get bigger coinbase amount, than other miners.

I don’t think there’s an incentive problem. Coins have no value and it’s easy enough to cpu mine some future blocks if you need tn4 coins.

While it might be annoying for users/developers to have their transactions rolled back, that’s a small price to pay for living on the edge and running a node with unmerged code, while there were still discussions going on about a consensus rule change. Bitcoin Core 28.0-RC1 is the first ‘official’ release with testnet4 support and that includes the time-warp fix.

My question is why did we merge code into Core that forked T4 without contacting any of the larger swaths of hashrate and or having the HR to outrun the previously canonical chain?

achow101 commented at 7:10 pm on September 4, 2024: member

My question is why did we merge code into Core that forked T4 without contacting any of the larger swaths of hashrate and or having the HR to outrun the previously canonical chain?

There was no indication at the time that anyone would mine blocks that were outside of MAX_TIMEWARP of 600. And generally, the assumption is that if someone is running code from an unmerged PR, they should be following it and be aware that it might majorly change before it is merged.

Also, it’s testnet :shrug:. We can just make another one.

benthecarman commented at 9:54 pm on September 4, 2024: contributor

I am on 28.0rc1 and seem to be on a different chain. Block is 00000000000044a7284978db5161b2f4ca012b3fb981a9405a4ce2473f3e4a5f for height 42545, unsure how to get to the main chain

garlonicon commented at 3:05 am on September 5, 2024: none

unsure how to get to the main chain

You are on the main chain. But the deep reorg didn’t kick in yet, so if you use for example mempool.space to see, if you are on the main chain or not, then that page is still showing the old chain. I guess it can take a few days to get there, but we are getting closer and closer, to trigger that reorg.

russeree commented at 5:14 am on September 5, 2024: contributor

My question is why did we merge code into Core that forked T4 without contacting any of the larger swaths of hashrate and or having the HR to outrun the previously canonical chain?

There was no indication at the time that anyone would mine blocks that were outside of MAX_TIMEWARP of 600. And generally, the assumption is that if someone is running code from an unmerged PR, they should be following it and be aware that it might majorly change before it is merged.

Also, it’s testnet 🤷. We can just make another one.

“the assumption is that if someone is running code from an unmerged PR, they should be following it and be aware that it might majorly change before it is merged.”

I thought about this earlier and agree with this statement. There is no difference other than subtlety of timestamp validation and just using a new genesis block. Either way I would have been forked.

garlonicon commented at 6:21 am on September 5, 2024: none

Also, we can now clearly see, why weak blocks are needed: mempool.space shows more than 500 transactions, waiting for being confirmed, but only a few of them are really mined, and the rest is waiting patiently, because nobody rebroadcasted them.

And there is more: it is possible to easily double-spend them, because it will take some time, before they will be shared in P2P way.

It is also something to consider, in case when mempools are congested: if you can see a miner, sending blocks with a low health, it doesn’t always mean censorship. It can also mean, that it was just recently started, and then, you can pass a low fee transaction, directly to this node, and get it confirmed, even though there will be a lot of higher paying transactions. Because if they are not re-broadcasted, then they are “invisible” for newly started nodes.

Sjors commented at 6:48 am on September 5, 2024: member

@garlonicon I think you’re going a bit off topic here, a weak blocks proposals seems more suitable for the mailinglist.

achow101 commented at 2:23 pm on September 5, 2024: member

My nodes’ logs indicate this was resolved at 2024-09-05 02:06:40 UTC when the new chain reorg’d the timewarped one.

achow101 closed this on Sep 5, 2024

bitcoin deleted a comment on Sep 19, 2024

garlonicon commented at 8:48 pm on October 18, 2024: none

I guess some ASIC miners still didn’t upgrade their nodes:

 0$ getchaintips
 1  ...
 2  {
 3    "height": 50831,
 4    "hash": "000000000060a808a1067bed2feb0b22261383bb0f9b6e3f0dee2436eb8e5048",
 5    "branchlen": 432,
 6    "status": "headers-only"
 7  },
 8  {
 9    "height": 50826,
10    "hash": "000000000035411cdce9394ab799205c7dd0ce99fd23b23f9481e591635fd0f2",
11    "branchlen": 427,
12    "status": "headers-only"
13  },
14  {
15    "height": 50817,
16    "hash": "0000000073a34762fc9f48419a7fba80c17783398848d8627b620cef7cd61bb4",
17    "branchlen": 1,
18    "status": "valid-headers"
19  },
20  {
21    "height": 50811,
22    "hash": "00000000000105aeeb41726a8cca80aa0515cce83fbbb6042d90129173bdd2ca",
23    "branchlen": 412,
24    "status": "headers-only"
25  },
26  ...

Over 400 different blocks. So, it seems that some forks are still there. This time at block number 50400.

murchandamus commented at 4:38 pm on October 21, 2024: contributor

@wiz, @russeree: Are you still mining on Testnet4?

wiz commented at 11:28 pm on October 21, 2024: contributor

I figured the community had started mining it by now - will take a look

russeree commented at 6:45 am on October 22, 2024: contributor

@wiz, @russeree: Are you still mining on Testnet4?

Currently no, I assumed honest HR had started mining on T4 now so I pointed my miners back at Mainnet. Should I reconsider?

murchandamus commented at 2:57 pm on October 22, 2024: contributor

I have the impression that the person that was selling testnet3 coins may be the one that has monopolized mining testnet4 now

ozankaymak commented at 3:00 pm on December 13, 2024: none

Currently due to this timestamp rule, the forks happening are out of control. A big reorg just happened. I think what is happening is someone keeps mining and forking their own chain and mining again. You can see it here.

Sjors commented at 4:56 am on December 14, 2024: member

Screenshot for archival purposes.

Today I only see small 1 block reorgs, but a while ago there was as slightly bigger one:

My guess is that isn’t related to the timewarp fix, instead people are trying to “exploit” the testnet (3 and 4) rule that you can mine a difficulty 1 block if you put the timestamp 20 minutes in the future (of the last block). You can only do that for 6 blocks, because the 6th block will be 120 minutes in the future. Because not all node clocks are accurate, the 6th block may end up rejected by some nodes and not by others. And then someone else wins.

cc @0xB10C curious about your take on this.

Sjors commented at 4:58 am on December 14, 2024: member

Another possibility is that someone is using an ASIC to mine difficulty 1 blocks and their pool software runs into race conditions. Some of these blocks have identical timestamps.

If someone wants to study this in more details, Delving Bitcoin or the mailinglist are probably a better places, to make sure your findings are not buried in a closed GitHub issue.

Testnet4 consensus failure due to timewarp related “softfork” #30786