Cannot generate blocks after invalidateblock on regtest chain longer than ~64 blocks #15905

Bitcoin Core version: v0.17.99.0-d6e700e40 running on OSX

I'm testing a client against reorgs from a Bitcoin Core node in regtest. To create a reorg I generate some blocks, then use invalidateblock and generatetoaddress again to create a fork. The pattern works fine as long as the regtest blockchain is not too long.

I'm not sure the exact threshold where it stops working, but it's somewhere between 50 - 64 blocks.

The error is "AcceptBlockHeader [hash] is marked invalid" but that invalid header should be long gone -- I'm generating a brand new block now...?

I guess bitcoind is probably re-generating the exact same block the second time, but not sure why the error only comes up on longer chains.

To reproduce:

Short chains, behaves as expected:

bitcoind -regtest

bitcoin-cli generatetoaddress 50 mfWxJ45yp2SFn7UciZyNpvDKrzbhyfKrY8

bitcoin-cli invalidateblock <one of the hashes returned from previous command>

bitcoin-cli generatetoaddress 1 mfWxJ45yp2SFn7UciZyNpvDKrzbhyfKrY8

Result:

2019-04-26T22:56:11Z InvalidChainFound: invalid block=37337d435262823883a7864536c08b06b07fe61205f02a404ec050e51a794191  height=49  log2_work=6.6438562  date=2019-04-26T22:56:13Z
2019-04-26T22:56:11Z InvalidChainFound:  current best=725a7df044bc3b1bbb0e103749470d345aea0cdbfc2629c31818cedd45b20b46  height=48  log2_work=6.6147098  date=2019-04-26T22:56:13Z
2019-04-26T22:56:15Z CreateNewBlock(): block weight: 900 txs: 0 fees: 0 sigops 400
2019-04-26T22:56:15Z UpdateTip: new best=6f87e7f8d16404d8387d194e585d0b0e3b285a4cee9df76e4967783421b092c2 height=49 version=0x20000000 log2_work=6.6438562 tx=50 date='2019-04-26T22:56:15Z' progress=1.000000 cache=0.0MiB(50txo)

Slightly longer chains, unable to proceed:

bitcoind -regtest

bitcoin-cli generatetoaddress 64 mfWxJ45yp2SFn7UciZyNpvDKrzbhyfKrY8

bitcoin-cli invalidateblock <one of the hashes returned from previous command>

bitcoin-cli generatetoaddress 1 mfWxJ45yp2SFn7UciZyNpvDKrzbhyfKrY8

Result:

2019-04-26T22:56:42Z InvalidChainFound: invalid block=6d73a1240616f89d2b382534edd543d0aa91321d850bcc0210547cfb174f1c92  height=62  log2_work=6.9772799  date=2019-04-26T22:56:45Z
2019-04-26T22:56:42Z InvalidChainFound:  current best=5317e42ded19040fc71f33a3db275abdd393c7cad1044d7f482c6c6f8b8925ff  height=61  log2_work=6.9541963  date=2019-04-26T22:56:44Z
2019-04-26T22:56:44Z CreateNewBlock(): block weight: 900 txs: 0 fees: 0 sigops 400
2019-04-26T22:56:44Z ERROR: AcceptBlockHeader: block 6d73a1240616f89d2b382534edd543d0aa91321d850bcc0210547cfb174f1c92 is marked invalid
2019-04-26T22:56:44Z ERROR: ProcessNewBlock: AcceptBlock FAILED (duplicate (code 0))

Hello @pinheadmz,

how did you resolve this issue? We're stumbled upon it as well.

@dexX7 which version of Bitcoin Core did you see the issue with? Don't hesitate to provide any other info or steps needed to reproduce what you saw.

I followed the above steps just now on current master with 50, 64, and 100 blocks in regtest, and was unable to reproduce the error -- it worked as expected.

I was looking at a similar block propagation issue after invalidateblock lately in #19397 with @jnewbery (who just posted an v interesting analysis that I plan to look more into to learn what is going on). Don't hesitate to have a look or review if interested.

@jonatack this happens after running invalidateblock and generating a replacement block at in same second and with the same coinbase address, the resulting block has the same hash as the invalidated block which then generates the "AcceptBlock FAILED (duplicate (code 0))" error reported above.

For our project (@dexX7 ) we run some bespoke tests that generate blocks, invalidates the chain then generates replacement blocks. This issue only arose after moving from the generate RPC call to generatetoaddress using the same address each time.

Due to needing to generate replacement blocks in the same second it does not show up in manual tests, if you script it on regtest then this is more likely to show up. After our script failed I did manually recreate the problem by hard coding the block time in CreateNewBlock and skipping its call to UpdateTime.

You can see the same blockhash being generated here with block time fixed and the same coinbase address being used. Note that the client being run is based on Bitcoin 0.18.

user@disco:/mnt/development/github/omnicore-original/src$ ./omnicore-cli -regtest generatetoaddress 1  moneyqMan7uh8FqdCA2BV5yZ8qVrc9ikLP
[
  "1fbfceb118b6303dc3329d912524331fc49d7e92a87996151058ad73f48ccf22"
]
user@disco:/mnt/development/github/omnicore-original/src$ ./omnicore-cli -regtest stop
Omni Core server stopping
user@disco:/mnt/development/github/omnicore-original/src$ rm -Rf ~/.bitcoin
user@disco:/mnt/development/github/omnicore-original/src$ ./omnicored -regtest -daemon
Omni Core server starting
user@disco:/mnt/development/github/omnicore-original/src$ ./omnicore-cli -regtest generatetoaddress 1 moneyqMan7uh8FqdCA2BV5yZ8qVrc9ikLP
[
  "1fbfceb118b6303dc3329d912524331fc49d7e92a87996151058ad73f48ccf22"
]

Running another manual test with the same set up and using invalidateblock.

user@disco:/mnt/development/github/omnicore-original/src$ ./omnicore-cli -regtest generatetoaddress 1 moneyqMan7uh8FqdCA2BV5yZ8qVrc9ikLP
[
  "1fbfceb118b6303dc3329d912524331fc49d7e92a87996151058ad73f48ccf22"
]
user@disco:/mnt/development/github/omnicore-original/src$ ./omnicore-cli -regtest invalidateblock 1fbfceb118b6303dc3329d912524331fc49d7e92a87996151058ad73f48ccf22
user@disco:/mnt/development/github/omnicore-original/src$ ./omnicore-cli -regtest generatetoaddress 1 moneyqMan7uh8FqdCA2BV5yZ8qVrc9ikLP
error code: -32603
error message:
ProcessNewBlock, block not accepted

Solutions for this would be to make your script wait a second after running invalidateblock or using a different coinbase address.

Solutions for this would be to make your script wait a second after running invalidateblock or using a different coinbase address.

Yeah the issue for me was bitcoind was re-creating a new block with the same hash as the previously invalidated block. Changing any part of the block would fix it - extranonce, nonce, timestamp, coinbase address etc.

I noticed we didn't have this problem in bcoin, and I think it's because that library adds a random 4-byte nonce to every coinbase scriptSig (after the BIP34 height commitment). On mainnet I think this is usually taken care of by extraNonce, but we could prevent this issue on regtest in bitcoind by adding a similar mechanism.

I've also come across this issue for v0.19.1.0-g58ba7c314d552cea8cb024960a8504577aee586f.

Same exact process:

• Run invalidateblock on regtest
• Generate a new block to the same address.

The issue seems to be the exact same, the block that is being generated has the exact same fields as the invalidated one, so the former conflicts with the latter.