Use of a wallet shouldn't be blocked in prune mode ("wallet loading failed... beyond pruned data")

GregTonoski commented at 11:54 am on March 2, 2023: none

It is impossible to load and use a wallet neither in Bitcoin-qt GUI nor console/RPC commands. The wallet is only about 2 weeks old and typical settings are used (data pruned to 2 GB).

Expected behavior

A wallet is loaded (either manually or automatically on startup) and addresses are shown in Bitcoin-qt,
successful execution of wallet RPC commands, e.g. getwalletinfo, dumpwallet, walletpassphrasechange etc.

(not different from the way it had worked and looked the previous time a user started Bitcoin Core).

Actual behavior

wallet loading failed Prune

Wallet loading failed. Prune: last wallet synchronisation goes beyond pruned data. You need to -reindex (download the whole blockchain again in case of pruned node)

Also, RPC commands fail, for example: loadwallet, getwalletinfo, dumpwallet (except for listwallets).

To reproduce

Start Bitcoin-qt
Select File->Open Wallet->[Wallet name]

Reproducible every time in prune mode. The wallet was created at the time for which data were pruned. The wallet was created in bitcoin-qt.

System information

Official binary Bitcoin Core 24.0.1, Windows 11.

GregTonoski added the label Bug on Mar 2, 2023

willcl-ark commented at 11:57 am on March 2, 2023: contributor

Possible (future) solution: https://github.com/bitcoin/bitcoin/issues/21267

sipa commented at 9:21 pm on March 2, 2023: member

@willcl-ark That still won’t work when the pruning point is later than the wallet birth date.

bitcoin deleted a comment on Mar 8, 2023

pinheadmz commented at 2:23 pm on March 22, 2023: member

@willcl-ark That still won’t work when the pruning point is later than the wallet birth date.

What if it is combined with getblockfrompeer logic?

I think we could close this issue in favor of #21267 and with the goal of essentially adding neutrino functionality to a pruning node for the sake of wallet rescans and imports ?

maflcko added the label Wallet on Mar 22, 2023

furszy commented at 3:08 pm on March 22, 2023: member

yeah @pinheadmz, same as in the other issue. Agree on unifying all this issues into a single one. So progress updates aren’t spread across the repository (I didn’t even know that this existed before) and we can make reviewers calls if the sub-projects that goes toward this goal require review.

maflcko commented at 3:26 pm on March 22, 2023: member

ok, closing for now as duplicate. Let us know if it isn’t and if this should be reopened.

maflcko closed this on Mar 22, 2023

shovas commented at 6:30 pm on December 12, 2023: none

Just wanted to chime in here and say that this is a critical blocker for anyone with a real wallet where this happens

I’m in an bad loop of loading wallet, ‘beyond pruned data…download entire blockchain’, download entire blockchain, abort on bad block, …?? Has happened on two computers where Bitcoin Core has been working great for over a year.

furszy commented at 6:36 pm on December 12, 2023: member

Just wanted to chime in here and say that this is a critical blocker for anyone with a real wallet where this happens

I’m in an bad loop of loading wallet, ‘beyond pruned data…download entire blockchain’, download entire blockchain, abort on bad block, …?? Has happened on two computers where Bitcoin Core has been working great for over a year.

In order to get this functionality merged, we need to get #27837 merged (cannot sync the wallet through the filters when the node isn’t able to track and re-request standalone blocks in case of failure). Which requires #28120 and #28170 -> which are under review currently (slow review but at least it is moving forward). I would be very happy if we can get more attention there. Would love to move forward and push the entire prune sync branch.

Also, #28955 is ready for review as well. It was decoupled from #26966 which is part of my prune sync branch for the major benefits it has in terms of testing this feature (waiting 4 hours to get the filters indexed is definitely unpleasant).

shovas commented at 7:00 pm on December 12, 2023: none

Is there a workaround in the meantime?

furszy commented at 7:21 pm on December 12, 2023: member

Is there a workaround in the meantime?

Would be good to know why your node is aborting on a bad block. Maybe you have a storage issue? Could you share the logs? (here or in a new issue is fine, just tag me when you do it).

Just as a small advice (not the best as your node should just work fine but.. having a backup plan always helps); you could have a full chain backup in a different storage. Then, if needed, transfer it to your pruned node to quickly recover from any sort of storage corruption.

shovas commented at 10:28 pm on December 13, 2023: none

Would be good to know why your node is aborting on a bad block. Maybe you have a storage issue? Could you share the logs? (here or in a new issue is fine, just tag me when you do it).

So far I have been using USB storage (128GB SanDisk Extreme Pro USB and a 1TB ADATA external ssd) whereas I have now successfully downloaded the complete blockchain to internal nvme fine, something that failed many times when trying to use a usb device as storage. Two machines, both Windows 10.

I am running badblocks on that adata drive on linux and at 4h and 60% it has no errors. A full bad sectors chkdsk on Windows didn’t find any issues on either drive. Badblocks on the 128GB SanDisk found no issues, either.

Edit: No errors from the ssd after completion.

I have a decent background in computers and programming and my spidey senses are telling me something’s weird. Like maybe it’s not a hardware issue, maybe it’s a usb protocol driver implementation or usage issue that is exposing a bug. Even the Bitcoin Core download page says it can expose bad hardware due to how intensive it is.

Here’s someone who’s basically experiencing what I’m experiencing:

https://www.reddit.com/r/intel/comments/mpazmq/usb_data_transfer_corruption_z590_11900k/

“Repeatedly diff a large binary file (e.g. 100GB) using cmd fc.exe or Beyond Compare binary comparison between system drive and USB attached drive. Result sometimes comes back identical, sometimes finds differences. Sometimes hours go by where I can’t reproduce the problem and then it just comes back and is the rule rather than the exception. A diff between a copy of the 100GB file on the system drive to the original file on the system drive has not shown up any problem in my tests so far (i.e. doing the same diff but without USB involved). Basically if I hadn’t found this USB problem I would be pretty much convinced the machine is stable.”

Another data point: 7zip reports bad data in those pruned bitcoin blockchain archives sometimes when I downloaded them straight to USB storage.

To be clear, two otherwise absolutely rock solid windows machines have both exhibited unreliable usb transmission even though the drives themselves appear reliable.

I guess what I’m saying is it feels like Windows’ usb implementation can lead to data corruption, possibly under heavy load. Given Bitcoin Core is a heavy load, and since it’s so important data be correct, maybe there’s some retry logic, some redundancy in reading/writing/verifying, some safer mode that could be leveraged just to be absolutely sure.

I’ll see if I have a log.

Just as a small advice (not the best as your node should just work fine but.. having a backup plan always helps); you could have a full chain backup in a different storage. Then, if needed, transfer it to your pruned node to quickly recover from any sort of storage corruption.

Thank you for the advice. I’ve now got a full blockchain download + backup and everything is running again.

I did have backup measures but didn’t factor in that I might need to download the whole blockchain. I thought those pruned data downloads + wallet backups would suffice.

I suppose with this git issue it will suffice in the future.

furszy commented at 2:53 pm on January 16, 2024: member

@shovas, I have created #29183 to focus our efforts and delineate the working path in the clearest possible manner. It should help pushing the project forward.

shovas commented at 11:10 pm on February 8, 2024: none

@shovas, I have created #29183 to focus our efforts and delineate the working path in the clearest possible manner. It should help pushing the project forward.

Thanks, I’m watching that issue. I’m experimenting with wallets for bitcoin, litecoin, dogecoin, and bitcoin cash using cli tools right now so maybe I can help in some way.

Use of a wallet shouldn’t be blocked in prune mode (“wallet loading failed… beyond pruned data”) #27188