corrupted data #30426

issue jsarenik openend this issue on July 11, 2024
  1. jsarenik commented at 6:29 am on July 11, 2024: none

    Is there an existing issue for this?

    • I have searched the existing issues

    Current behaviour

    The only peer (via connect=IP in bitcoin.conf) is another local node running aba504759caa version and that node has 94 other peers at the moment so I guess there is no problem there.

    I do not know if it is related, but since I run aba504759caa (the testnet4 commit) also for mainnet, here is link to testnet4: #29775

    Expected behaviour

    Synchronization of all the blocks with the P2P Bitcoin network.

    Steps to reproduce

    Figuring that out. It all worked well for weeks with no change.

    Relevant log output

    02024-07-11T06:06:42Z UpdateTip: new best=000000000000000000036686ffd324b99fa90f190f48c7372e14738bda739383 height=851624 version=0x2462e000 log2_work=95.031625 tx=1038976733 date='2024-07-11T02:42:12Z' progress=0.999922 cache=17.2MiB(151149txo)
12024-07-11T06:06:42Z Leaving InitialBlockDownload (latching to false)
22024-07-11T06:06:42Z initload thread exit
32024-07-11T06:06:42Z Warning: not punishing manually connected peer 0!

    How did you obtain Bitcoin Core

    Compiled from source

    What version of Bitcoin Core are you using?

    v27.1.0 from bitcoincore.org

    Operating system and version

    Pop!_OS 22.04 LTS

    Machine specifications

    Linux x86_64

    0$ free -h
1               total        used        free      shared  buff/cache   available
2Mem:           7.6Gi       1.7Gi       461Mi       359Mi       5.5Gi       5.1Gi
3Swap:          7.6Gi       985Mi       6.7Gi

    Swap is zram

  2. maflcko commented at 6:36 am on July 11, 2024: member
    You’ll have to enable at least the net debug log category to get some more details on the type of misbehaviour. (C.f. PeerManagerImpl::Misbehaving)
  3. maflcko added the label P2P on Jul 11, 2024
  4. jsarenik commented at 6:49 am on July 11, 2024: none
    Thanks, closing for now. This may be something local related to my setup. Thanks for debug=net hint as it is showing me interesting things but there is so much it wouldn’t make sense to write here until I understand it slightly better.
  5. jsarenik closed this on Jul 11, 2024
  6. jsarenik commented at 6:51 am on July 11, 2024: none

    Ah, I have something:

    02024-07-11T06:50:34Z initload thread exit
12024-07-11T06:50:34Z [net] sending getheaders (1029 bytes) peer=0
22024-07-11T06:50:34Z [net] initial getheaders (851623) to peer=0 (startheight:851650)
32024-07-11T06:50:34Z Leaving InitialBlockDownload (latching to false)
42024-07-11T06:50:34Z [net] received: headers (2188 bytes) peer=0
52024-07-11T06:50:34Z [net] Misbehaving: peer=0: invalid header received
62024-07-11T06:50:34Z Warning: not punishing manually connected peer 0!
  7. jsarenik reopened this on Jul 11, 2024
  8. jsarenik commented at 6:53 am on July 11, 2024: none

    How can I receive an invalid header from a perfectly running node?

    mainnet, I just use the same version which I use for testnet4

  9. jsarenik commented at 7:05 am on July 11, 2024: none 
    02024-07-11T07:02:21Z UpdateTip: new best=000000000000000000036686ffd324b99fa90f190f48c7372e14738bda739383 height=851624 version=0x2462e000 log2_work=95.031625 tx=1038976733 date='2024-07-11T02:42:12Z' progress=0.999901 cache=17.4MiB(151149txo)
12024-07-11T07:02:21Z initload thread exit
22024-07-11T07:02:21Z [net] sending getheaders (1029 bytes) peer=0
32024-07-11T07:02:21Z [net] initial getheaders (851623) to peer=0 (startheight:851650)
42024-07-11T07:02:21Z Leaving InitialBlockDownload (latching to false)
52024-07-11T07:02:21Z [net] sending feefilter (8 bytes) peer=0
62024-07-11T07:02:21Z [net] received: headers (2188 bytes) peer=0
72024-07-11T07:02:21Z [net] Misbehaving: peer=0 (0 -> 100) DISCOURAGE THRESHOLD EXCEEDED: invalid header received
82024-07-11T07:02:21Z Warning: not punishing manually connected peer 0!
  10. maflcko commented at 7:05 am on July 11, 2024: member
    ugh. Why is the BlockValidationState not logged?
  11. jsarenik commented at 7:06 am on July 11, 2024: none
    Already on Bitcoin 27.1 binary release from bitcoincore.org
  12. maflcko commented at 7:09 am on July 11, 2024: member

    ugh. Why is the BlockValidationState not logged?

    You’ll also have to enable validation logging. (C.f. ChainstateManager::AcceptBlockHeader)

  13. jsarenik commented at 7:16 am on July 11, 2024: none 
    02024-07-11T07:15:41Z UpdateTip: new best=000000000000000000036686ffd324b99fa90f190f48c7372e14738bda739383 height=851624 version=0x2462e000 log2_work=95.031625 tx=1038976733 date='2024-07-11T02:42:12Z' progress=0.999896 cache=17.2MiB(151149txo)
12024-07-11T07:15:41Z [validation] Enqueuing BlockConnected: block hash=000000000000000000036686ffd324b99fa90f190f48c7372e14738bda739383 block height=851624
22024-07-11T07:15:41Z [validation] Enqueuing UpdatedBlockTip: new block hash=000000000000000000036686ffd324b99fa90f190f48c7372e14738bda739383 fork block hash=00000000000000000001d5976abee369d3356e568f46c8ab839addb94779ba3a (in IBD=true)
32024-07-11T07:15:41Z [validation] MempoolTransactionsRemovedForBlock: block height=851624 txs removed=0
42024-07-11T07:15:41Z initload thread exit
52024-07-11T07:15:41Z [validation] BlockConnected: block hash=000000000000000000036686ffd324b99fa90f190f48c7372e14738bda739383 block height=851624
62024-07-11T07:15:41Z [net] sending getheaders (1029 bytes) peer=0
72024-07-11T07:15:41Z [net] initial getheaders (851623) to peer=0 (startheight:851653)
82024-07-11T07:15:41Z Leaving InitialBlockDownload (latching to false)
  14. jsarenik commented at 7:21 am on July 11, 2024: none 
    02024-07-11T07:20:02Z [validation] AcceptBlockHeader: block 00000000000000000000c29870d6f87b9aca6577e25ab2a780603e605db99ae4 is marked invalid
12024-07-11T07:20:02Z [net] Misbehaving: peer=0 (0 -> 100) DISCOURAGE THRESHOLD EXCEEDED: invalid header received
22024-07-11T07:20:02Z Warning: not punishing manually connected peer 0!
  15. jsarenik commented at 7:22 am on July 11, 2024: none
    marked invalid here on this run, but not by mempool.space https://mempool.space/block/00000000000000000000c29870d6f87b9aca6577e25ab2a780603e605db99ae4
  16. maflcko commented at 7:23 am on July 11, 2024: member

    You’ll have to check the debug log when 00000000000000000000c29870d6f87b9aca6577e25ab2a780603e605db99ae4 was first connected to see the error reason.

    But my guess would be corruption on your machine.

  17. maflcko removed the label P2P on Jul 11, 2024
  18. maflcko added the label Block storage on Jul 11, 2024
  19. maflcko added the label Data corruption on Jul 11, 2024
  20. maflcko added the label Questions and Help on Jul 11, 2024
  21. jsarenik commented at 7:26 am on July 11, 2024: none
    OK, data corruption is possible. How can I tell Bitcoin to redownload that block again?
  22. maflcko commented at 7:34 am on July 11, 2024: member

    re-downloading the block may not be enough, depending on the data corruption.

    For example, if the utxo db is corrupt, you’ll have to build it from scratch.

    I think calling the RPC gettxoutsetinfo muhash on both nodes (when they are synced to the same block) and it matches for all, then the chainstate leveldb at that point in time is probably fine.

    To recreate the block connection to get the failure again, I think you can use invalidateblock+reconsiderblock.

  23. jsarenik commented at 7:40 am on July 11, 2024: none

    Yes! Calling gettxoutsetinfo muhash triggered this:

     02024-07-11T07:36:31Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/b/.bitcoin/chainstate/13938982.ldb
 12024-07-11T07:36:31Z You can use -debug=leveldb to get more complete diagnostic messages
 22024-07-11T07:36:31Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/b/.bitcoin/chainstate/13938982.ldb
 32024-07-11T07:36:31Z Error: A fatal internal error occurred, see debug.log for details
 4Error: A fatal internal error occurred, see debug.log for details
 52024-07-11T07:36:31Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/b/.bitcoin/chainstate/13938982.ldb)
 62024-07-11T07:36:31Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/b/.bitcoin/chainstate/13938982.ldb
 72024-07-11T07:36:31Z You can use -debug=leveldb to get more complete diagnostic messages
 82024-07-11T07:36:31Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/b/.bitcoin/chainstate/13938982.ldb
 92024-07-11T07:36:31Z Error: A fatal internal error occurred, see debug.log for details
10Error: A fatal internal error occurred, see debug.log for details
112024-07-11T07:36:31Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /home/b/.bitcoin/chainstate/13938982.ldb)
  24. jsarenik commented at 7:42 am on July 11, 2024: none
    Thank you for help, @maflcko !
  25. jsarenik closed this on Jul 11, 2024
  26. jsarenik renamed this:
    not punishing manually connected peer
    corrupted data
    on Jul 11, 2024


jsarenik maflcko

Labels
Block storage Data corruption Questions and Help

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me