miner: Reorg Testnet4 minimum difficulty blocks

fjahr commented at 4:40 pm on October 19, 2024: contributor

Note: This is just a proof of concept put here for discussion/feedback/brainstorming/improvement. I am not sure if we want to merge this into core at all since it can be dealt with externally as well. But this was the simplest way for me to make it work and it’s probably also simpler for many miners to run this branch rather than another external software.

Context

Testnet4 fixes block storms but it kept the minimum-difficulty exception in place. The rationale was that this would be the only way developers could still get their non-standard transactions into the blockchain. However, probably due to the public discussion of this feature, this has attracted the attention of people whose motivation might be A) grab as many testnet coins as possible or B) to annoy developers. There are currently one or more actors who just max out the min-difficulty exception as much as possible and as quickly as possible, see https://mempool.space/testnet4. There are always several min-difficulty blocks appearing instantly after a normal difficulty mined block.

There are several implications to this:

Developers can not get their non-standard transactions into the blockchain using the min-exception blocks, unless they put in significant effort to outrace the spammers
The actual difficulty is artificially raised
Less coins are going to faucets (though this is a very minor issue)

Solution

The solution to this problem was already briefly discussed in the original pull request for Testnet4: The min-difficulty blocks can all be easily re-orged by a single block with the actual difficulty. The code here implements this functionality within the getblocktemplate RPC. All blocks with min-difficulty (including forks) are temporarily marked invalid while the template is created. The blocks are only temporarily marked invalid since another miner might find an actual difficulty block on top of the min-difficulty blocks and we don’t want to create a fork in the chain in that case.

We do want to keep the transactions from the blocks and include them in the block template. That should work with this solution via the orphan pool but I haven’t done extensive testing on this yet. In reality it seems like there are several transactions in these min-difficulty blocks with a minimum block height and these can not be included in the re-org block (except the ones that are in the first one). However, given that these transactions appear in almost every block it might just be self-transfers created by the spammer.

Mining Pool

The follow-up to this would be to run a public (solo) pool with this enabled so that we can get some more hash rate behind this without requiring any effort on the side of those with hash rate and demotivate the spammers.

Acknowledgement

I worked on this at Bitcoin++ in Berlin a week ago together with Emzy and Sjors. Together we were able to demo a pool running a previous version of this PR and connected a miner to it. That pool isn’t public yet though.

Demo

Here are some log outputs showing how first the orphan blocks are invalidated and then the actual chain is rolled back. Then, after the template was constructed, the blocks are reconsidered.

 02024-10-19T13:14:29Z InvalidChainFound: invalid block=00000000004967e9a55121ced9a1b5df2bbf794edef6eaeca56985180dca178a  height=50944  log2_work=71.373305  date=2024-10-19T13:58:13Z
 12024-10-19T13:14:29Z InvalidChainFound:  current best=000000000097f5f8174415729790d2d83b90af466564ddf416247a050681caee  height=50947  log2_work=71.373305  date=2024-10-19T14:58:16Z
 22024-10-19T13:14:29Z InvalidChainFound: invalid block=00000000002663fc47b4f47626296cc2d3118320581c4ff0eaa6317812bffdb2  height=50945  log2_work=71.373305  date=2024-10-19T14:18:14Z
 32024-10-19T13:14:29Z InvalidChainFound:  current best=000000000097f5f8174415729790d2d83b90af466564ddf416247a050681caee  height=50947  log2_work=71.373305  date=2024-10-19T14:58:16Z
 42024-10-19T13:14:29Z InvalidChainFound: invalid block=000000000062e7e0743d4e97085a1696c7352096ea53e58daad468fc8995f397  height=50946  log2_work=71.373305  date=2024-10-19T14:38:15Z
 52024-10-19T13:14:29Z InvalidChainFound:  current best=000000000097f5f8174415729790d2d83b90af466564ddf416247a050681caee  height=50947  log2_work=71.373305  date=2024-10-19T14:58:16Z
 62024-10-19T13:14:29Z UpdateTip: new best=0000000000638927100cc5578d750a9fcb763020fdab55bf27ba39ba26a57000 height=50946 version=0x2840e000 log2_work=71.373305 tx=862937 date='2024-10-19T14:38:15Z' progress=1.000000 cache=0.6MiB(2838txo)
 72024-10-19T13:14:29Z InvalidChainFound: invalid block=000000000097f5f8174415729790d2d83b90af466564ddf416247a050681caee  height=50947  log2_work=71.373305  date=2024-10-19T14:58:16Z
 82024-10-19T13:14:29Z InvalidChainFound:  current best=0000000000638927100cc5578d750a9fcb763020fdab55bf27ba39ba26a57000  height=50946  log2_work=71.373305  date=2024-10-19T14:38:15Z
 92024-10-19T13:14:29Z UpdateTip: new best=0000000000d35f958ad057b68c5822990b8b91325f6e549a2386b578e231a8b0 height=50945 version=0x27b82000 log2_work=71.373305 tx=862933 date='2024-10-19T14:18:14Z' progress=1.000000 cache=0.6MiB(2834txo)
102024-10-19T13:14:29Z InvalidChainFound: invalid block=0000000000638927100cc5578d750a9fcb763020fdab55bf27ba39ba26a57000  height=50946  log2_work=71.373305  date=2024-10-19T14:38:15Z
112024-10-19T13:14:29Z InvalidChainFound:  current best=0000000000d35f958ad057b68c5822990b8b91325f6e549a2386b578e231a8b0  height=50945  log2_work=71.373305  date=2024-10-19T14:18:14Z
122024-10-19T13:14:29Z UpdateTip: new best=0000000000c0704c802da78900f465ea3eaeba55719b76c88bc9f2a43bacb58c height=50944 version=0x22de4000 log2_work=71.373305 tx=862930 date='2024-10-19T13:58:13Z' progress=1.000000 cache=0.6MiB(2832txo)
132024-10-19T13:14:29Z InvalidChainFound: invalid block=0000000000d35f958ad057b68c5822990b8b91325f6e549a2386b578e231a8b0  height=50945  log2_work=71.373305  date=2024-10-19T14:18:14Z
142024-10-19T13:14:29Z InvalidChainFound:  current best=0000000000c0704c802da78900f465ea3eaeba55719b76c88bc9f2a43bacb58c  height=50944  log2_work=71.373305  date=2024-10-19T13:58:13Z
152024-10-19T13:14:29Z UpdateTip: new best=0000000000f6f9d3588bbc15a2bd5da43c4470301b4361104a9781e530830bfb height=50943 version=0x224c4000 log2_work=71.373305 tx=862929 date='2024-10-19T13:38:12Z' progress=1.000000 cache=0.6MiB(2832txo)
162024-10-19T13:14:29Z InvalidChainFound: invalid block=0000000000c0704c802da78900f465ea3eaeba55719b76c88bc9f2a43bacb58c  height=50944  log2_work=71.373305  date=2024-10-19T13:58:13Z
172024-10-19T13:14:29Z InvalidChainFound:  current best=0000000000f6f9d3588bbc15a2bd5da43c4470301b4361104a9781e530830bfb  height=50943  log2_work=71.373305  date=2024-10-19T13:38:12Z
182024-10-19T13:14:29Z UpdateTip: new best=0000000000000005a8ae7897cfd4a0a9f1f45c89bf9f4dc090d7074b2b7a4586 height=50942 version=0x20000000 log2_work=71.373305 tx=862928 date='2024-10-19T13:18:11Z' progress=1.000000 cache=0.6MiB(2832txo)
192024-10-19T13:14:29Z InvalidChainFound: invalid block=0000000000f6f9d3588bbc15a2bd5da43c4470301b4361104a9781e530830bfb  height=50943  log2_work=71.373305  date=2024-10-19T13:38:12Z
202024-10-19T13:14:29Z InvalidChainFound:  current best=0000000000000005a8ae7897cfd4a0a9f1f45c89bf9f4dc090d7074b2b7a4586  height=50942  log2_work=71.373305  date=2024-10-19T13:18:11Z
212024-10-19T13:14:29Z CreateNewBlock(): block weight: 14069 txs: 18 fees: 35293 sigops 412
222024-10-19T13:14:29Z UpdateTip: new best=0000000000f6f9d3588bbc15a2bd5da43c4470301b4361104a9781e530830bfb height=50943 version=0x224c4000 log2_work=71.373305 tx=862929 date='2024-10-19T13:38:12Z' progress=1.000000 cache=0.6MiB(2832txo)
232024-10-19T13:14:29Z UpdateTip: new best=0000000000c0704c802da78900f465ea3eaeba55719b76c88bc9f2a43bacb58c height=50944 version=0x22de4000 log2_work=71.373305 tx=862930 date='2024-10-19T13:58:13Z' progress=1.000000 cache=0.6MiB(2832txo)
242024-10-19T13:14:29Z UpdateTip: new best=0000000000d35f958ad057b68c5822990b8b91325f6e549a2386b578e231a8b0 height=50945 version=0x27b82000 log2_work=71.373305 tx=862933 date='2024-10-19T14:18:14Z' progress=1.000000 cache=0.6MiB(2834txo)
252024-10-19T13:14:29Z UpdateTip: new best=0000000000638927100cc5578d750a9fcb763020fdab55bf27ba39ba26a57000 height=50946 version=0x2840e000 log2_work=71.373305 tx=862937 date='2024-10-19T14:38:15Z' progress=1.000000 cache=0.6MiB(2838txo)
262024-10-19T13:14:29Z UpdateTip: new best=000000000097f5f8174415729790d2d83b90af466564ddf416247a050681caee height=50947 version=0x25f34000 log2_work=71.373305 tx=862947 date='2024-10-19T14:58:16Z' progress=1.000000 cache=0.6MiB(2853txo)

DrahtBot commented at 4:40 pm on October 19, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process. A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#27433 (getblocktemplate improvements for segwit and sigops by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Mining on Oct 19, 2024

fjahr marked this as a draft on Oct 19, 2024

DrahtBot commented at 6:23 pm on October 19, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/31773418335

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

DrahtBot added the label CI failed on Oct 19, 2024

fjahr commented at 12:47 pm on October 21, 2024: contributor

CI is broken because BIP94 is also active on regtest and this change breaks some functional tests that are not related to the issue. I don’t think I need to work on fixing those tests unless I get an indication that people think this is actually interesting to merge.

We are getting some report that the getblocktemplate sometimes hangs on a small-ish debian 12 server. I have not been able to reproduce this locally or on a small-ish ubuntu server. Would be great if anyone is interested in testing this on their setup or has an idea what might cause this by looking at the code. So far, I have not been able to find the problem.

miner: Reorg Testnet4 minimum difficulty blocks 786487133e

fjahr force-pushed on Oct 21, 2024

murchandamus commented at 5:30 pm on October 21, 2024: contributor

In addition to the immediate minimum difficulty blocks after each full difficulty block, someone appears to have started bloating the UTXO set. They have created about 1.5 M UTXOs today so far: https://mempool.space/testnet4/address/tb1q8z0jcrdmfuhyafz06ukhyvfnx4vhcn5azv6jkr

garlonicon commented at 12:43 pm on October 22, 2024: none

The rationale was that this would be the only way developers could still get their non-standard transactions into the blockchain.

Well, the solution to that is quite simple: if non-standard transactions would be normally propagated, then developers could simply publish them, and have them confirmed. Unless you want to get an additional spam protection, by requiring every developer to provide 32 leading zero bits in Proof of Work. But in that case, just doing OP_SIZE on a signature will give you the same outcome.

grab as many testnet coins as possible

Note that testnet4 is already traded at 20 satoshis per 1 tBTC4. And testnet5 will also be, if it will be released in a similar way: https://altquick.com/exchange/market/BitcoinTestnet4

There are always several min-difficulty blocks appearing instantly after a normal difficulty mined block.

When coins get non-zero value, then people are simply trying to get the maximum number of blocks, and sell them. And the original algorithm from Bitcoin Core is far from optimal. If you pick future timestamps, then you can for example instantly mine as many CPU blocks as possible, and then put some ASIC block on top of that. Which means, that staying 2 hours in the future is profitable. And also: mining ASIC blocks 2 hours in the future is also profitable, because then, you block potential CPU-mined blocks from being broadcasted, for the next 20 minutes (meanwhile, you prepare such block, and then release it, when the network time will get there).

The actual difficulty is artificially raised

It is always the case, if you count CPU-mined blocks into difficulty adjustments. Because if you have 2016 blocks, where 1008 blocks are CPU-mined, and 1008 blocks are ASIC-mined, then in practice, the real difficulty should be calculated after the next 1008 ASIC blocks. But instead, it is re-adjusted now, so the more CPU-mined blocks are there, the less ASIC-mined blocks remains (until falling above 10 minutes per block on average, when the difficulty then starts falling).

The min-difficulty blocks can all be easily re-orged by a single block with the actual difficulty.

If it will be harder to mine testnet4, then the value per coin will increase. Which means more trading. Unless people will stay in testnet3, where the chain is faster, because of blockstorms.

We do want to keep the transactions from the blocks and include them in the block template.

All of them? Including also those with zero fees, or other non-matching mempool policy?

Sjors commented at 1:34 pm on October 24, 2024: member

cc @Emzy

1440000bytes commented at 1:53 pm on October 24, 2024: none

Is it possible for core to remove testnet to avoid maintenance burden?

Regtest and Signet could work fine for testing. Maybe testnet could be supported by some fork if there is enough demand.

plebhash commented at 2:46 pm on October 24, 2024: none

Regtest and Signet could work fine for testing.

testnet4 is being used for testing in Stratum V2 development, where a network with realistic mining dynamics is important

neither regtest nor signet allow for this

this is not a statement with regards to core maintainance though, I’m not opinionated with regards to that

just pointing out that there is demand for this kind of network

garlonicon commented at 8:36 am on October 25, 2024: none

where a network with realistic mining dynamics is important

What about using “getblocktemplate” on mainnet, and mining blocks, which will quickly become stale? If you want to have worthless coins, then they should be stale, in other cases, they will be traded.

Another possibility is to use “Pay to Proof of Work” addresses on signet/regtest. Then, moving a coin will require grinding a DER signature, to get it below N bytes, so it will require some mining power.

just pointing out that there is demand for this kind of network

Now, testnet4 coins are actively traded, as well as testnet3 coins. Which means, that it is time for testnet5. But: it should have different rules, because if blocks would have any persistence, then they will be traded again. That’s why all of them should be stale, so they could be shared between interested parties, while also being worthless at the same time.

Davidson-Souza commented at 6:06 pm on October 25, 2024: none

In 7864871. If I fire a getblocktemplate before sync is done, I get a segfault.

 0==1025895== Thread 7 b-httpworker.3:
 1==1025895== Invalid read of size 4
 2==1025895==    at 0x36A95F: node::(anonymous namespace)::MinerImpl::rollbackTestnet4() (interfaces.cpp:997)
 3==1025895==    by 0x42F759: getblocktemplate()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)#1}::operator()(RPCHelpMan const&, JSONRPCRequest const&) const [clone .isra.0] (mining.cpp:666)
 4==1025895==    by 0x433821: __invoke_impl<UniValue, getblocktemplate()::<lambda(const RPCHelpMan&, const JSONRPCRequest&)>&, const RPCHelpMan&, const JSONRPCRequest&> (invoke.h:61)
 5==1025895==    by 0x433821: __invoke_r<UniValue, getblocktemplate()::<lambda(const RPCHelpMan&, const JSONRPCRequest&)>&, const RPCHelpMan&, const JSONRPCRequest&> (invoke.h:116)
 6==1025895==    by 0x433821: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), getblocktemplate()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)#1}>::_M_invoke(std::_Any_data const&, RPCHelpMan const&, JSONRPCRequest const&) (std_function.h:291)
 7==1025895==    by 0x8B74B1: operator() (std_function.h:591)
 8==1025895==    by 0x8B74B1: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (util.cpp:679)
 9==1025895==    by 0x3FC5E7: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const [clone .isra.0] (server.h:101)
10==1025895==    by 0x3FC92F: __invoke_impl<bool, CRPCCommand::CRPCCommand(std::string, RpcMethodFnType)::<lambda(const JSONRPCRequest&, UniValue&, bool)>&, const JSONRPCRequest&, UniValue&, bool> (invoke.h:61)
11==1025895==    by 0x3FC92F: __invoke_r<bool, CRPCCommand::CRPCCommand(std::string, RpcMethodFnType)::<lambda(const JSONRPCRequest&, UniValue&, bool)>&, const JSONRPCRequest&, UniValue&, bool> (invoke.h:114)
12==1025895==    by 0x3FC92F: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (std_function.h:290)
13==1025895==    by 0x4B77FF: operator() (std_function.h:591)
14==1025895==    by 0x4B77FF: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (server.cpp:512)
15==1025895==    by 0x4B8CF7: ExecuteCommands (server.cpp:477)
16==1025895==    by 0x4B8CF7: CRPCTable::execute(JSONRPCRequest const&) const (server.cpp:497)
17==1025895==    by 0x4B8DE6: JSONRPCExec(JSONRPCRequest const&, bool) (server.cpp:353)
18==1025895==    by 0x59B80B: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (httprpc.cpp:218)
19==1025895==    by 0x5A95E9: operator() (std_function.h:591)
20==1025895==    by 0x5A95E9: operator() (httpserver.cpp:62)
21==1025895==    by 0x5A95E9: WorkQueue<HTTPClosure>::Run() (httpserver.cpp:117)
22==1025895==    by 0x5A5594: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (httpserver.cpp:415)
23==1025895==  Address 0x84 is not stack'd, malloc'd or (recently) free'd
24==1025895== 
25==1025895== 
26==1025895== Process terminating with default action of signal 11 (SIGSEGV): dumping core
27==1025895==  Access not within mapped region at address 0x84
28==1025895==    at 0x36A95F: node::(anonymous namespace)::MinerImpl::rollbackTestnet4() (interfaces.cpp:997)
29==1025895==    by 0x42F759: getblocktemplate()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)#1}::operator()(RPCHelpMan const&, JSONRPCRequest const&) const [clone .isra.0] (mining.cpp:666)
30==1025895==    by 0x433821: __invoke_impl<UniValue, getblocktemplate()::<lambda(const RPCHelpMan&, const JSONRPCRequest&)>&, const RPCHelpMan&, const JSONRPCRequest&> (invoke.h:61)
31==1025895==    by 0x433821: __invoke_r<UniValue, getblocktemplate()::<lambda(const RPCHelpMan&, const JSONRPCRequest&)>&, const RPCHelpMan&, const JSONRPCRequest&> (invoke.h:116)
32==1025895==    by 0x433821: std::_Function_handler<UniValue (RPCHelpMan const&, JSONRPCRequest const&), getblocktemplate()::{lambda(RPCHelpMan const&, JSONRPCRequest const&)#1}>::_M_invoke(std::_Any_data const&, RPCHelpMan const&, JSONRPCRequest const&) (std_function.h:291)
33==1025895==    by 0x8B74B1: operator() (std_function.h:591)
34==1025895==    by 0x8B74B1: RPCHelpMan::HandleRequest(JSONRPCRequest const&) const (util.cpp:679)
35==1025895==    by 0x3FC5E7: CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const [clone .isra.0] (server.h:101)
36==1025895==    by 0x3FC92F: __invoke_impl<bool, CRPCCommand::CRPCCommand(std::string, RpcMethodFnType)::<lambda(const JSONRPCRequest&, UniValue&, bool)>&, const JSONRPCRequest&, UniValue&, bool> (invoke.h:61)
37==1025895==    by 0x3FC92F: __invoke_r<bool, CRPCCommand::CRPCCommand(std::string, RpcMethodFnType)::<lambda(const JSONRPCRequest&, UniValue&, bool)>&, const JSONRPCRequest&, UniValue&, bool> (invoke.h:114)
38==1025895==    by 0x3FC92F: std::_Function_handler<bool (JSONRPCRequest const&, UniValue&, bool), CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}>::_M_invoke(std::_Any_data const&, JSONRPCRequest const&, UniValue&, bool&&) (std_function.h:290)
39==1025895==    by 0x4B77FF: operator() (std_function.h:591)
40==1025895==    by 0x4B77FF: ExecuteCommand(CRPCCommand const&, JSONRPCRequest const&, UniValue&, bool) (server.cpp:512)
41==1025895==    by 0x4B8CF7: ExecuteCommands (server.cpp:477)
42==1025895==    by 0x4B8CF7: CRPCTable::execute(JSONRPCRequest const&) const (server.cpp:497)
43==1025895==    by 0x4B8DE6: JSONRPCExec(JSONRPCRequest const&, bool) (server.cpp:353)
44==1025895==    by 0x59B80B: HTTPReq_JSONRPC(std::any const&, HTTPRequest*) (httprpc.cpp:218)
45==1025895==    by 0x5A95E9: operator() (std_function.h:591)
46==1025895==    by 0x5A95E9: operator() (httpserver.cpp:62)
47==1025895==    by 0x5A95E9: WorkQueue<HTTPClosure>::Run() (httpserver.cpp:117)
48==1025895==    by 0x5A5594: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (httpserver.cpp:415)

miner: Reorg Testnet4 minimum difficulty blocks #31117