use -loadblock to load blk*****.dat files, but the blocks in it are not recognized #33280

issue HowHsu openend this issue on September 2, 2025
  1. HowHsu commented at 12:18 pm on September 2, 2025: none

    The background is I’m trying to measure the performance of connecting real blocks when the needed inputs are not in memory. My first idea is to get some blk*****.dat files with -stopatheight and then use -stopafterblockimport and -loadblock to load these blk files (ConnectBlock() will run when doing this) to see the performance. But I found the blk*****.dat files can not be recognized by the code. Though I now found an alternative way VerifyDB() can be leveraged to achieve my goal. I’d like still to put the above issue here. The bitcoin core code I use is at HEAD 6ca6f3b37b Merge bitcoin/bitcoin#33241: Update libmultiprocess subtree to fix build issues

    Is there an existing issue for this?

    • I have searched the existing issues

    Current behaviour

    1. I first run bitcoind -prune=1024 -stopatheight=840000 -datadir=empty_dir, then there is some block files in empty_dir/blocks.
    2. Then I copy empty_dir to empty_dir2, run bitcoind -prune=1024 -stopatheight=840050 -datadir=empty_dir2 Compare two blocks dir, I now have some blk.dat files relects height 840000 to 840050.
    3. At last I copy empty_dir to empty_dir3, and run bitcoind -datadir=empty_dir3 -assumevalid=0 -prune=10240 -stopafterblockimport -networkactive=0 -noconnect -listen=0 -dnsseed=0 -dns=0 -loadblock=blk04241.dat …… -loadblock=blk04252.dat, the result shows it finds 0 blocks from those files, and I did some test, I found the reason is those files don’t have the main Magic bytes f9beb4d9

    Anything wrong?

    Expected behaviour

    the blocks from height 840001 to 840050 are loaded from those blk files and connected to the chain.

    Steps to reproduce

    see above

    Relevant log output

    No response

    How did you obtain Bitcoin Core

    Compiled from source

    What version of Bitcoin Core are you using?

    master

    Operating system and version

    ubuntu22.04 LTS

    Machine specifications

    No response

  2. HowHsu renamed this:
    -loadblock indicated block files cannot find Magic Bytes
    -loadblock doesn't work
    on Sep 2, 2025
  3. mzumsande commented at 3:25 pm on September 2, 2025: contributor
    are the two datadirs using the same xor.dat or was it disabled (see -blocksxor)?
  4. HowHsu commented at 10:16 am on September 3, 2025: none

    are the two datadirs using the same xor.dat or was it disabled (see -blocksxor)?

    I didn’t add -blocksxor, but I compared two xor.dat in the two blocks dir (with vim -b, shows <93><80><a5><b4><fb>^_<b3><b8>), seems they are same.

  5. l0rinc commented at 6:38 pm on September 4, 2025: contributor

    It also seems to me like a xor mismatch. Could you quote the start of the logs for both source and target instance what the obfuscation key values were?

    0cat debug.log | grep 'Using obfuscation key for'
  6. HowHsu commented at 11:34 am on September 6, 2025: none

    Some clarification about my steps below:

    0
11. I first run bitcoind -prune=1024 -stopatheight=840000 -datadir=empty_dir, then there is some block files in empty_dir/blocks.
22. Then I copy empty_dir to empty_dir2, run bitcoind -prune=1024 -stopatheight=840050 -datadir=empty_dir2
3Compare two blocks dir, I now have some blk.dat files relects height 840000 to 840050.
43. At last I copy empty_dir to empty_dir3, and run bitcoind -datadir=empty_dir3 -assumevalid=0 -prune=10240 -stopafterblockimport -networkactive=0 -noconnect -listen=0 -dnsseed=0 -dns=0 -loadblock=blk04241.dat ...... -loadblock=blk04252.dat, the result shows it finds 0 blocks from those files, and I did some test, I found the reason is those files don't have the main Magic bytes f9beb4d9

    I found the reason is the xor key is different in step 2 and step 3. But I do copy the xor.dat when copying empty_dir to empty_dir2 and empty_dir3, and from the code, I saw the blocks/xor.dat is used when there are not only hidden files in the blocks dir. So seems the xor key should be the same since the xor.dat is the initial one. I’m still investigating on it.

  7. HowHsu commented at 12:20 pm on September 6, 2025: none

    Some clarification about my steps below:

    0
11. I first run bitcoind -prune=1024 -stopatheight=840000 -datadir=empty_dir, then there is some block files in empty_dir/blocks.
22. Then I copy empty_dir to empty_dir2, run bitcoind -prune=1024 -stopatheight=840050 -datadir=empty_dir2
3Compare two blocks dir, I now have some blk.dat files relects height 840000 to 840050.
43. At last I copy empty_dir to empty_dir3, and run bitcoind -datadir=empty_dir3 -assumevalid=0 -prune=10240 -stopafterblockimport -networkactive=0 -noconnect -listen=0 -dnsseed=0 -dns=0 -loadblock=blk04241.dat ...... -loadblock=blk04252.dat, the result shows it finds 0 blocks from those files, and I did some test, I found the reason is those files don't have the main Magic bytes f9beb4d9

    I found the reason is the xor key is different in step 2 and step 3. But I do copy the xor.dat when copying empty_dir to empty_dir2 and empty_dir3, and from the code, I saw the blocks/xor.dat is used when there are not only hidden files in the blocks dir. So seems the xor key should be the same since the xor.dat is the initial one. I’m still investigating on it.

    Sorry, . Checked again, the obfuscation key for *.dat files are indded the same one. So it’s not the xor issue.

    02025-09-06T11:43:23Z Using obfuscation key for blocksdir *.dat files (/work/pr_test/datatmp/blocks): '9380a5b4fb1fb3b8'
  8. HowHsu commented at 12:27 pm on September 6, 2025: none

    Reduced block ranges to only one file, run logs attached below.

      0bitcoind -datadir=datatmp -assumevalid=0 -prune=10240 -stopafterblockimport -networkactive=0 -noconnect -listen=0 -dnsseed=0 -dns=0 -loadblock=data840010/blocks/blk04241.dat -loadblock=data840010/blocks/blk04242.dat
  12025-09-06T12:24:44Z Bitcoin Core version v29.99.0-unk (release build)
  22025-09-06T12:24:44Z parameter interaction: -listen=0 -> setting -natpmp=0
  32025-09-06T12:24:44Z parameter interaction: -listen=0 -> setting -discover=0
  42025-09-06T12:24:44Z parameter interaction: -listen=0 -> setting -listenonion=0
  52025-09-06T12:24:44Z parameter interaction: -listen=0 -> setting -i2pacceptincoming=0
  62025-09-06T12:24:44Z Using the 'x86_shani(1way,2way)' SHA256 implementation
  72025-09-06T12:24:44Z Using RdSeed as an additional entropy source
  82025-09-06T12:24:44Z Using RdRand as an additional entropy source
  92025-09-06T12:24:44Z Default data directory /root/.bitcoin
 102025-09-06T12:24:44Z Using data directory /work/pr_test/datatmp
 112025-09-06T12:24:44Z Config file: /work/pr_test/datatmp/bitcoin.conf (not found, skipping)
 122025-09-06T12:24:44Z Command-line arg: assumevalid="0"
 132025-09-06T12:24:44Z Command-line arg: connect=false
 142025-09-06T12:24:44Z Command-line arg: datadir="datatmp"
 152025-09-06T12:24:44Z Command-line arg: dns="0"
 162025-09-06T12:24:44Z Command-line arg: dnsseed="0"
 172025-09-06T12:24:44Z Command-line arg: listen="0"
 182025-09-06T12:24:44Z Command-line arg: loadblock="data840010/blocks/blk04241.dat"
 192025-09-06T12:24:44Z Command-line arg: loadblock="data840010/blocks/blk04242.dat"
 202025-09-06T12:24:44Z Command-line arg: networkactive="0"
 212025-09-06T12:24:44Z Command-line arg: prune="10240"
 222025-09-06T12:24:44Z Command-line arg: stopafterblockimport=""
 232025-09-06T12:24:44Z Using at most 125 automatic connections (1024 file descriptors available)
 242025-09-06T12:24:44Z Warning: relative datadir option 'datatmp' specified, which will be interpreted relative to the current working directory '/work/pr_test'. This is fragile, because if bitcoin is started in the future from a different location, it will be unable to locate the current data files. There could also be data loss if bitcoin is started while in a temporary directory.
 252025-09-06T12:24:44Z scheduler thread start
 262025-09-06T12:24:44Z Binding RPC on address ::1 port 8332
 272025-09-06T12:24:44Z [libevent:warning] getaddrinfo: address family for nodename not supported
 282025-09-06T12:24:44Z Binding RPC on address ::1 port 8332 failed.
 292025-09-06T12:24:44Z Binding RPC on address 127.0.0.1 port 8332
 302025-09-06T12:24:44Z Generated RPC authentication cookie /work/pr_test/datatmp/.cookie
 312025-09-06T12:24:44Z Permissions used for cookie: rw-------
 322025-09-06T12:24:44Z Using random cookie authentication.
 332025-09-06T12:24:44Z Starting HTTP server with 16 worker threads
 342025-09-06T12:24:44Z Using wallet directory /work/pr_test/datatmp
 352025-09-06T12:24:44Z init message: Verifying wallet(s)…
 362025-09-06T12:24:44Z Using /16 prefix for IP bucketing
 372025-09-06T12:24:44Z init message: Loading P2P addresses…
 382025-09-06T12:24:44Z Loaded 31757 addresses from peers.dat  28ms
 392025-09-06T12:24:44Z init message: Loading banlist…
 402025-09-06T12:24:44Z SetNetworkActive: false
 412025-09-06T12:24:44Z Cache configuration:
 422025-09-06T12:24:44Z * Using 2.0 MiB for block index database
 432025-09-06T12:24:44Z * Using 8.0 MiB for chain state database
 442025-09-06T12:24:44Z * Using 440.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
 452025-09-06T12:24:44Z Script verification uses 15 additional threads
 462025-09-06T12:24:44Z Using obfuscation key for blocksdir *.dat files (/work/pr_test/datatmp/blocks): '9380a5b4fb1fb3b8'
 472025-09-06T12:24:44Z Opening LevelDB in /work/pr_test/datatmp/blocks/index
 482025-09-06T12:24:44Z Opened LevelDB successfully
 492025-09-06T12:24:44Z Using obfuscation key for /work/pr_test/datatmp/blocks/index: 0000000000000000
 502025-09-06T12:24:44Z Using 16 MiB out of 16 MiB requested for signature cache, able to store 524288 elements
 512025-09-06T12:24:44Z Using 16 MiB out of 16 MiB requested for script execution cache, able to store 524288 elements
 522025-09-06T12:24:44Z init message: Loading block index…
 532025-09-06T12:24:44Z Validating signatures for all blocks.
 542025-09-06T12:24:44Z Setting nMinimumChainWork=0000000000000000000000000000000000000000b1f3b93b65b16d035a82be84
 552025-09-06T12:24:44Z Prune configured to target 10240 MiB on disk for block and undo files.
 562025-09-06T12:24:46Z Loading block index db: last block file = 4241
 572025-09-06T12:24:46Z Loading block index db: last block file info: CBlockFileInfo(blocks=32, size=57246369, heights=839969...840013, time=2024-04-19...2024-04-20)
 582025-09-06T12:24:46Z Checking all blk files are present...
 592025-09-06T12:24:46Z Loading block index db: Block files have previously been pruned
 602025-09-06T12:24:46Z Initializing chainstate Chainstate [ibd] @ height -1 (null)
 612025-09-06T12:24:46Z Opening LevelDB in /work/pr_test/datatmp/chainstate
 622025-09-06T12:24:46Z Opened LevelDB successfully
 632025-09-06T12:24:46Z Using obfuscation key for /work/pr_test/datatmp/chainstate: 2a94feefc21ba0b3
 642025-09-06T12:24:46Z Loaded best chain: hashBestChain=0000000000000000000320283a032748cef8227873ff4872689bf23f1cda83a5 height=840000 date=2024-04-20T00:09:27Z progress=0.801499
 652025-09-06T12:24:46Z init message: Verifying blocks…
 662025-09-06T12:24:46Z Verifying last 6 blocks at level 3
 672025-09-06T12:24:46Z Verification progress: 0%
 682025-09-06T12:24:46Z Verification progress: 16%
 692025-09-06T12:24:46Z Verification progress: 33%
 702025-09-06T12:24:46Z Verification progress: 50%
 712025-09-06T12:24:46Z Verification progress: 66%
 722025-09-06T12:24:46Z Verification progress: 83%
 732025-09-06T12:24:46Z Verification progress: 99%
 742025-09-06T12:24:46Z Verification: No coin database inconsistencies in last 6 blocks (15601 transactions)
 752025-09-06T12:24:46Z Block index and chainstate loaded
 762025-09-06T12:24:46Z init message: Pruning blockstore…
 772025-09-06T12:24:46Z block tree size = 912308
 782025-09-06T12:24:46Z nBestHeight = 840000
 792025-09-06T12:24:46Z init message: Starting network threads…
 802025-09-06T12:24:46Z DNS seeding disabled
 812025-09-06T12:24:46Z initload thread start
 822025-09-06T12:24:46Z init message: Done loading
 832025-09-06T12:24:46Z Reindexing block file blk04241.dat...
 842025-09-06T12:24:46Z msghand thread start
 852025-09-06T12:24:46Z addcon thread start
 862025-09-06T12:24:46Z net thread start
 872025-09-06T12:24:46Z Loaded 0 blocks from external file in 11ms
 882025-09-06T12:24:46Z Reindexing finished
 892025-09-06T12:24:46Z Importing blocks file data840010/blocks/blk04241.dat...
 902025-09-06T12:24:46Z Loaded 0 blocks from external file in 38ms
 912025-09-06T12:24:46Z Importing blocks file data840010/blocks/blk04242.dat...
 922025-09-06T12:24:46Z Loaded 0 blocks from external file in 18ms
 93time spent for ImportBlocks: 0
 942025-09-06T12:24:46Z Stopping after block import
 952025-09-06T12:24:46Z initload thread exit
 962025-09-06T12:24:46Z Shutdown in progress...
 972025-09-06T12:24:46Z net thread exit
 982025-09-06T12:24:46Z addcon thread exit
 992025-09-06T12:24:47Z msghand thread exit
1002025-09-06T12:24:47Z scheduler thread exit
1012025-09-06T12:24:47Z Flushed fee estimates to fee_estimates.dat.
1022025-09-06T12:24:47Z Shutdown done
  9. l0rinc commented at 8:44 pm on September 6, 2025: contributor
    I’m not sure I can follow what versions you’re using in the examples and what you’re trying to achieve high level - could you please update the title and issue description and start with the zoomed out problem definition for what you’re trying to achieve and what you tried and what versions you have used for which operation etc?
  10. HowHsu renamed this:
    -loadblock doesn't work
    use -loadblock to load blk*****.dat files, but the blocks in it are not recognized
    on Sep 7, 2025
  11. HowHsu commented at 1:07 pm on September 7, 2025: none

    I’m not sure I can follow what versions you’re using in the examples and what you’re trying to achieve high level - could you please update the title and issue description and start with the zoomed out problem definition for what you’re trying to achieve and what you tried and what versions you have used for which operation etc?

    Hey I0rinc, super appreciate it for your review of this issue, I’ve updated the title and description, let me know if that makes sense or not.

  12. mzumsande commented at 4:04 pm on September 8, 2025: contributor

    I looked at the code, and it appears that loadblock currently only works with non-xor’ed block files:
    AutoFile file{fsbridge::fopen(path, “rb”)}; doesn’t use an obfuscation key.

    I don’t know if that was discussed when block obfuscation was added (FYI @maflcko). loadblock is probably not a feature that is used very much these days, but unless someone wants to add blocksxor support to it, this limitation could at least be mentioned in its help text.

    A simple workaround for the use case here should be to use -blocksxor=0 in all datadirs.

  13. HowHsu commented at 5:44 pm on September 8, 2025: none

    I looked at the code, and it appears that loadblock currently only works with non-xor’ed block files: AutoFile file{fsbridge::fopen(path, “rb”)}; doesn’t use an obfuscation key.

    I don’t know if that was discussed when block obfuscation was added (FYI @maflcko). loadblock is probably not a feature that is used very much these days, but unless someone wants to add blocksxor support to it, this limitation could at least be mentioned in its help text.

    A simple workaround for the use case here should be to use -blocksxor=0 in all datadirs.

    Hi @mzumsande , thanks for troubleshooting this one, I’ve created a PR to enrich the help text of -loadblock: PR 33343

  14. l0rinc commented at 8:01 pm on September 8, 2025: contributor

    A simple workaround for the use case here should be to use -blocksxor=0 in all datadirs.

    But that would only work for a pristine IBD, right? I have opened #33324 to be able to change the obfuscation of existing blocks - this should allow you to remove an existing obfuscation after block download.

  15. l0rinc commented at 11:00 pm on September 8, 2025: contributor
    @mzumsande, are you familiar with /contrib/linearize? I wasn’t, see #33343 (comment)
  16. maflcko commented at 3:47 pm on September 26, 2025: member

    I don’t know if that was discussed when block obfuscation was added (FYI @maflcko).

    I wasn’t aware of a use-case to load raw block files without first linearizing them. As suggested, the workaround here would be -blocksxor=0 for the node that generates block files to import. You could also hack together a trivial python script to do the xor for you, quickly. (The linearize script may also work, if you omit the unneeded block hashes, but I haven’t tried this).

  17. willcl-ark added the label RPC/REST/ZMQ on Jan 14, 2026
  18. willcl-ark added the label Block storage on Jan 14, 2026


HowHsu mzumsande l0rinc maflcko

Labels
RPC/REST/ZMQ Block storage

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-01-28 09:13 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me