contrib/signet/miner: grind will fail for high difficulty chain #30102

issue edilmedeiros openend this issue on May 14, 2024
  1. edilmedeiros commented at 8:17 pm on May 14, 2024: contributor

    Is there an existing issue for this?

    • I have searched the existing issues

    Current behaviour

    Mining will fail with a Could not satisfy difficulty target.

     0❯ ~/2-development/bitcoin/bitcoin-core/contrib/signet/miner --cli "/Users/jose.edil/2-development/bitcoin/bitcoin-core/src/bitcoin-cli -datadir=/Users/jose.edil/2-development/bitcoin/signet-mining-experiments/signet-fix-hashing" generate --address tb1qylfujt900rjxzfxjj7sktpu7dpm2n9j60ch7jt --grind-cmd "/Users/jose.edil/2-development/bitcoin/bitcoin-core/src/bitcoin-util grind" --nbits 1d008d28 --ongoing
     12024-05-14 16:29:05 INFO Mined block at height 10079; next in -324h56m20s (mine)
     22024-05-14 16:31:22 INFO Mined block at height 10080; next in -324h56m7s (mine)
     32024-05-14 16:32:34 INFO Mined block at height 10081; next in -324h54m49s (mine)
     4Could not satisfy difficulty target
     5Traceback (most recent call last):
     6  File "/Users/jose.edil/2-development/bitcoin/bitcoin-core/contrib/signet/miner", line 545, in <module>
     7    main()
     8  File "/Users/jose.edil/2-development/bitcoin/bitcoin-core/contrib/signet/miner", line 539, in main
     9    return args.fn(args)
    10           ^^^^^^^^^^^^^
    11  File "/Users/jose.edil/2-development/bitcoin/bitcoin-core/contrib/signet/miner", line 418, in do_generate
    12    block = finish_block(block, signet_solution, args.grind_cmd)
    13            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    14  File "/Users/jose.edil/2-development/bitcoin/bitcoin-core/contrib/signet/miner", line 107, in finish_block
    15    newheadhex = subprocess.run(cmd, stdout=subprocess.PIPE, input=b"", check=True).stdout.strip()
    16                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    17  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
    18    raise CalledProcessError(retcode, process.args,
    19subprocess.CalledProcessError: Command '['/Users/jose.edil/2-development/bitcoin/bitcoin-core/src/bitcoin-util', 'grind', '0000002063d19ae45a71b26fc04ddeef152919d2ecc22ed936ea7129d3bf291f03000000703be9fb35f03fc17b9b3210ac8364c1b76888d793397b7686128107d21c0b4a39e33166ac77031d00000000']' returned non-zero exit status 1.
    

    This is a follow-up of the discussion in #30091: I started a signet with difficulty --nbits 1d008d28 and mined the first block with a date 30 days in the past. That would let the miner run as fast as it can until difficulty is so high it will converge to the 10 minute average time between blocks.

    After each 2016 blocks, the difficulty is adjusted as per the network consensus and it gets increasingly harder to find new blocks. After 5 adjustments, the grinder eventually exhausts the nonce search space and fails, making the python script to fail as well.

    I inspected the bitcoin-util grinder source, but could not find any problem there. Indeed, it does what one would expect: look for PoW and fail if the nonce search space is exhausted. That’s where the stdout message comes.

    Expected behaviour

    I tracked down the problem to this section of the code.

    The finish_block function should be able to catch the exception that can arise from the grinder subprocess and in case of failure try another block header. In case of signet, I believe it is a matter of resigning it as in and looking for PoW again until it finds something.

    I started a fix to submit a PR, but wanted to see if someone else could reproduce this behavior in the meantime. It’s not clear to me how to make a (unit) test to systematically expose the problem, though.

    Steps to reproduce

    Follow the outline discussed in #30091 and let the miner run (for a few hours) until the difficulty gets high enough so that one needs a bigger search space to find valid PoW for new blocks.

    Relevant log output

    Current chain info

     0❯ ~/2-development/bitcoin/bitcoin-core/src/bitcoin-cli -datadir="/Users/jose.edil/2-development/bitcoin/signet-mining-experiments/signet-fix-hashing" -signet getblockchaininfo
     1{
     2  "chain": "signet",
     3  "blocks": 10081,
     4  "headers": 10081,
     5  "bestblockhash": "000000031f29bfd32971ea36d92ec2ecd2192915efde4dc06fb2715ae49ad163",
     6  "difficulty": 0.2883904525532027,
     7  "time": 1714545315,
     8  "mediantime": 1714544565,
     9  "verificationprogress": 1,
    10  "initialblockdownload": true,
    11  "chainwork": "000000000000000000000000000000000000000000000000000000c3e464c5fc",
    12  "size_on_disk": 4153462,
    13  "pruned": false,
    14  "warnings": ""
    15}
    

    How did you obtain Bitcoin Core

    Compiled from source

    What version of Bitcoin Core are you using?

    v27.0

    Operating system and version

    MacOS Sonoma 14.4

    Machine specifications

    Macbook Pro M3 Pro 18GB Memory

  2. willcl-ark added the label Scripts and tools on May 15, 2024
  3. willcl-ark added the label Questions and Help on May 15, 2024
  4. bitcoin deleted a comment on May 15, 2024
  5. 1ma commented at 10:33 pm on May 15, 2024: none

    Can reproduce. Been running a signet network whose miner stalled a few dozens of blocks after the block 10080 difficulty adjustment.

    The miner (v26.1 script) is ran as a systemd service like this, and the nbits parameter was determined by running the calibrating utility set to 600s. Despite this, before block 10080 the miner used to mine a block every 2.5 min instead of 10. I don’t quite understand how it works, honestly.

    0miner --cli="bitcoin-cli" generate --address $rewards_addy --grind-cmd="bitcoin-util grind" --nbits=1d02257e --ongoing
    
  6. edilmedeiros commented at 2:59 am on May 16, 2024: contributor

    Thanks for reporting.

    About the 2m30s interval, see #30091.

  7. 1ma commented at 6:31 pm on May 16, 2024: none

    Thanks for the pointer. A bit off-topic, but since there are not so many people yet who are familiar with operating a signet let me ask you a follow up question:

    I’m don’t really understand why the signet miner has so much logic and is so opinionated. My initial idea was to keep the difficulty as low as possible then fire a cronjob every 10 minutes to insta-mine a block at the current timestamp, so the consensus algorithm would be tricked into thinking that is the real difficulty.

    I was never able to use the miner like this though, it seems to have a life of its own. No matter which nbits value I selected it mined a block every 2m30s (until it reached block 10080 and broke soon after at 10106). And if it stopped for some time it insta-mined blocks with a backdated timestamp until it catched up to the present.

    Is it possible to have a signet mining setup like that, that is not constantly raising the difficulty and wasting CPU?

  8. edilmedeiros commented at 7:27 pm on May 16, 2024: contributor

    Yeah, I had some difficulty understanding its behavior and I believe I grasped after studying the miner source code.

    It has an internal timer whose logic is capable of doing what you want. Instead of using the calibrate feature, go straight to using --min-nbits --ongoing. It will mine a block very fast and automatically wait for 10 minutes until it mines the next. That way, it keeps difficulty at minimum but can keep the network pace.

    See https://edil.com.br/blog/creating-a-custom-bitcoin-signet.

  9. edilmedeiros referenced this in commit 275d5eebc8 on May 17, 2024
  10. edilmedeiros referenced this in commit 5f9d4e1c91 on May 17, 2024
  11. edilmedeiros referenced this in commit 67fbf378ab on May 17, 2024
  12. edilmedeiros referenced this in commit 2a5bd91c76 on May 20, 2024
  13. edilmedeiros commented at 6:32 pm on May 20, 2024: contributor
    @1ma, since you seem to have a high difficulty signet chain running, may I ask for a review in #30130?
  14. edilmedeiros referenced this in commit 1cf174a295 on May 22, 2024
  15. edilmedeiros commented at 7:18 pm on September 14, 2024: contributor
    Could not reproduce this using --min-nbits with the new miner script from #28417. Seems like the difficulty won’t increase as fast as I would expect, even after 10 difficulty adjustment periods. I’ll leave it running more time to be sure and also try again using a custom difficulty as initially reported.
  16. edilmedeiros commented at 1:10 am on September 16, 2024: contributor

    Confirming this behavior is still happening after #28417. I’ll rework #30130 on top of it.

     02024-09-15 12:39:41 INFO Mined block at height 10078; next in -5784h44m40s (mine)
     12024-09-15 12:39:58 INFO Mined block at height 10079; next in -5784h42m28s (mine)
     22024-09-15 12:41:01 INFO Mined block at height 10080; next in -5784h41m0s (mine)
     32024-09-15 12:41:43 INFO Mined block at height 10081; next in -5784h39m12s (mine)
     4Traceback (most recent call last):
     5  File "/Users/jose.edil/2-development/bitcoin/issues/30102/bitcoin-core/contrib/signet/miner", line 574, in <module>
     6    main()
     7  File "/Users/jose.edil/2-development/bitcoin/issues/30102/bitcoin-core/contrib/signet/miner", line 568, in main
     8    return args.fn(args)
     9           ^^^^^^^^^^^^^
    10  File "/Users/jose.edil/2-development/bitcoin/issues/30102/bitcoin-core/contrib/signet/miner", line 444, in do_generate
    11    block = gen.mine(args.bcli, args.grind_cmd, tmpl, reward_spk)
    12            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    13  File "/Users/jose.edil/2-development/bitcoin/issues/30102/bitcoin-core/contrib/signet/miner", line 335, in mine
    14    return finish_block(block, signet_solution, grind_cmd)
    15           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    16  File "/Users/jose.edil/2-development/bitcoin/issues/30102/bitcoin-core/contrib/signet/miner", line 92, in finish_block
    17    newheadhex = subprocess.run(cmd, capture_output=True, input=b"", check=True).stdout.strip()
    18                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    19  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
    20    raise CalledProcessError(retcode, process.args,
    21subprocess.CalledProcessError: Command '['/Users/jose.edil/2-development/bitcoin/issues/30102/bitcoin-core/build/src/bitcoin-util', 'grind', '0000002041572f11c94b6ae830cbd5554128a958508176bbe42a5c0cd583410502000000d8d4e7fc8d2b6805211a9917099ce221808f9503e55d2c9718a8fdba9ccede64863da965ac77031d00000000']' returned non-zero exit status 1.
    

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me