corrupts wallet, does not mention anything #17690

After suffering from corrupt blockchains, we now had a corrupt wallet. Bitcoin Core does not message about this fact on startup. Only when trying to spend some BTC we are informed. Also it sees addr.dat as a wallet. So what is going on here? Can`t Bitcoin Core make a safety copy before a transaction?

Maybe add a checksum, so that even with encrypted content the file can be verified. When we add a pre-change backup of that file (which was integrity checked OK) then we have a bit more safety in the whole process.

What version are you using? What does the debug log say on startup and when you encounter the error?

You can find the debug.log in your data dir.

@udovdh

Also it sees addr.dat as a wallet.

Could you provide steps to reproduce such behavior?

Use the Bitcoin core software. It mentions corrupt blockchains, it mentions corrupt wallet not at startup but when trying to spend BTC. As this is the first discovery of the wallet corruption, and while we are downloading the blockchain again (why can't it download parts of the chain? who never thought of that scenario?) I cannot even try to reproduce this at this time.

We run Bitcoin Core 0.19.0.1 debug log holds no info over the wallet corruption; we were trying to recover the keys, etc, started the blockchain sync, etc. Thank the 3-2-1 system and a lazy admin (me) for a wallet.dat that works. Why is addr.dat seen as a wallet?

We try to shut down bitcoin core and it sits and sits: 2019-12-07T15:59:19Z UpdateTip: new best=000000000000000000d828ac3de5b8bf6d00e8d2f2da474ad86a13b1c422b9e8 height=386060 version=0x00000004 log2_work=83.670821 tx=94998574 date='2015-11-30T16:25:25Z' progress=0.196872 cache=4714.0MiB(34212576txo) 2019-12-07T15:59:19Z ERROR: ConnectBlock(): coinbase pays too much (actual=2505575669 vs limit=2505575637) 2019-12-07T15:59:19Z InvalidChainFound: invalid block=00000000000000000bf2effd303664da1edb54ce8ea07216ced45e5d001e7ab6 height=386061 log2_work=83.670851 date=2015-11-30T16:28:23Z 2019-12-07T15:59:19Z InvalidChainFound: current best=000000000000000000d828ac3de5b8bf6d00e8d2f2da474ad86a13b1c422b9e8 height=386060 log2_work=83.670821 date=2015-11-30T16:25:25Z 2019-12-07T15:59:19Z ERROR: ConnectTip: ConnectBlock 00000000000000000bf2effd303664da1edb54ce8ea07216ced45e5d001e7ab6 failed, bad-cb-amount (code 16) 2019-12-07T15:59:19Z InvalidChainFound: invalid block=00000000000000000d29b62f6e12d49db55cb05e477b6e2d48807ca259b3c6e3 height=386068 log2_work=83.671056 date=2015-11-30T17:53:17Z 2019-12-07T15:59:19Z InvalidChainFound: current best=000000000000000000d828ac3de5b8bf6d00e8d2f2da474ad86a13b1c422b9e8 height=386060 log2_work=83.670821 date=2015-11-30T16:25:25Z 2019-12-07T15:59:19Z Pre-allocating up to position 0x3000000 in blk00385.dat 2019-12-07T15:59:19Z Pre-allocating up to position 0x4000000 in blk00385.dat 2019-12-07T15:59:21Z Pre-allocating up to position 0x5000000 in blk00385.dat 2019-12-07T15:59:24Z Pre-allocating up to position 0x6000000 in blk00385.dat 2019-12-07T15:59:27Z Pre-allocating up to position 0x7000000 in blk00385.dat 2019-12-07T15:59:34Z Pre-allocating up to position 0x8000000 in blk00385.dat 2019-12-07T15:59:51Z New outbound peer connected: version: 70015, blocks=607072, peer=29 (full-relay) 2019-12-07T16:02:02Z tor: Thread interrupt 2019-12-07T16:02:02Z Shutdown: In progress... 2019-12-07T16:02:02Z addcon thread exit 2019-12-07T16:02:02Z torcontrol thread exit 2019-12-07T16:02:02Z opencon thread exit 2019-12-07T16:02:02Z net thread exit 2019-12-07T16:02:02Z msghand thread exit 2019-12-07T16:02:03Z scheduler thread interrupt 2019-12-07T16:02:03Z Dumped mempool: 6e-06s to copy, 0.023108s to dump

This is on reasonably fast storage. What is it doing? The log shows nothing now.

And of course it suddenly finds invalid blocks and chains and such on fresh startup. This is on recent hardware, redundant storage and no system log errors. So the software appears to be very unstable, unreliable, bad at handling common errors, etc.

What was the specific corrupted wallet error? What makes you think that it is reading addr.dat as a wallet?

That you are seeing multiple corruptions is more indicative of your hardware having an issue rather than the software. Recent hardware and redundant storage can still have hardware issues and Bitcoin Core is (unfortunately) good at causing hardware issues to turn up.

The error was -1. Hardware is new. amdgpu errors are on the way out, happen less often, and were the only instability here. bitcoin core cannot cope with any issue. downloading the blockchain again gets old quickly. The architecture needs to radically change to enable various error handling features.

The error was -1.

What was the error message? The error code is not particularly helpful.

Hardware is new.

New hardware can and does have errors. You can also have hardware issues where the OS reports no issues and nothing ends up in the system log.

bitcoin core cannot cope with any issue. downloading the blockchain again gets old quickly. The architecture needs to radically change to enable various error handling features.

There already is lots of error handling. But we also cannot add error handling for something that is not reliably reproduced. The vast majority of users and developers experience no corruption. Every corruption report that we have had is almost always the result of some sort of hardware failure. With hardware failures, we can't always know that something is corrupted due to the hardware. These kinds of errors cannot really be handled.

I recommend that you run hardware diagnostics instead of insisting that there is nothing wrong with your hardware. Or try Bitcoin Core on another machine.

I did not write bicoin core. It was something like: Error unlocking wallet: some keys decrypt but not all. Your wallet file may be corrupt. So I have the password of the wallet but no keys were usable after this message. Bitcoin core shows no error about the wallet on startup. Only when trying to spend it discovers this. Because of the importance of the wallet I detect some lack of care here. Same as with the blockchain, the lack of error handling there, etc. We can compile kernels and run kernels all day, but the magic sauce in bitcoin core is yet to be revealed. But the lack of error handling is already clear.

'hardware diagnostics' such as?

It was something like: Error unlocking wallet: some keys decrypt but not all. Your wallet file may be corrupt.

Can you compare the wallet file you have loaded into Bitcoin Core with your backup file? You can use the db_dump utility (you need to install BDB 4.8.30 from https://www.oracle.com/database/technologies/related/berkeleydb-release-history.html) to get a dump of the wallet files and then just compare them. You can use db_dump -t wallet.dat to get the hex interpreted as ascii. You will see lines that begin with \04ckey which are then followed by a line that does not have that prefix. Those pairs of lines are what you want to compare. The goal here is to see whether your disk has some sort of corruption that is resulting in the keys being different on disk.

If the dumps are the same, you probably have a RAM or CPU issue, or your backup was already corrupted to begin with. You can try using another machine to check the latter case.

So I have the password of the wallet but no keys were usable after this message. Bitcoin core shows no error about the wallet on startup. Only when trying to spend it discovers this.

#16946 makes it so that we will detect this on startup in the future.

I did not write bicoin core. .... Because of the importance of the wallet I detect some lack of care here. Same as with the blockchain, the lack of error handling there, etc. We can compile kernels and run kernels all day, but the magic sauce in bitcoin core is yet to be revealed. But the lack of error handling is already clear.

You say it yourself, you are not a developer on this project. As such, I don't think you should be making such claims, especially since it is clear that you don't understand the nature of the corruption happening and whether there can even be error handing. To say "lack of care" is extremely rude.

'hardware diagnostics' such as?

memtest86 to check your systems RAM. Run a S.M.A.R.T test on your hard drives to see if the drive detects any disk corruption. Try different hardware.

Bitcoin Core makes heavy use of CPU, RAM and disk IO. Hardware defects might only become visible when running Bitcoin Core. You might want to check your hardware for defects.

• memtest86 to check your RAM
• to check the CPU behaviour under load, use linpack or Prime95
• to test your storage device use smartctl or CrystalDiskInfo

Source: https://bitcoin.stackexchange.com/a/12206

The hardware mantra is unnecessary: If these errors appear on normal healthy hardware then Bitcoin Core needs to cope with them. The lack of error handling is apparent by using the software. Thanks for #16946; when will we have this fix in a released version?

The hardware mantra is unnecessary: If these errors appear on normal healthy hardware then Bitcoin Core needs to cope with them.

The point is that these errors do not appear on normal healthy hardware. If they did, we would have far more reports of corruption, and we don't.

Thanks for #16946; when will we have this fix in a released version?

0.20 probably, but it does an upgrade step so it doesn't just work for any old wallet. Wallets will need to be unlocked with the new version first so that the checksums get written.

Why unlock first? The point was that without unlocking the software could more or less guarantee integrity. Or is the unlocking (once!) just for checking initial integrity? The 'healthy' hardware label does not exist in IT hardware stores I use. Being ESD conscious the hardware here should be over average in health.

Why unlock first? The point was that without unlocking the software could more or less guarantee integrity. Or is the unlocking (once!) just for checking initial integrity?

It is to initial integrity before adding a checksum.

So we are back home and syncing the blockchain again. I wanted to test the wallet by opening it. File -> Open Wallet shows a greyed out [default wallet] and a black letterd addr.dat.1575723879.bak. It does show me an amount of BTC. There is a wallet.dat in ~/.bitcoin. Why can't I simply open it? What does the situation tell me?

I can make a backup of the wallet. (which wallet?)

How can a wallet be 'open' (if that is the case) right after startup? Supposedly there was a password...

And again:

2019-12-15T07:17:37Z LevelDB read failure: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/077170.ldb 2019-12-15T07:17:37Z Fatal LevelDB error: Corruption: block checksum mismatch: /home/udo/.bitcoin/chainstate/077170.ldb 2019-12-15T07:17:37Z You can use -debug=leveldb to get more complete diagnostic messages 2019-12-15T07:17:37Z Error: Error reading from database, shutting down.

But the gui simply sits and does nothing. No recovery, no error handling. All the other software runs OK so why would bitcoin core see corruption and force to download the whole blockchain again?

No 'hardware' mantra please. If this happens on normal off the shelf hardware that has been handled correctly w.r.t. ESD, then have the software deal with it in a more gracious manner. Where is the lightning network?

Flooding this PR with messages is not going to achieve anything. All the advice that can be given has been given.

This is open source software. You're welcome to improve the error handling and recovery. We're not your unpaid workers.

Commenting on the shape of things instead of the core issue? Flooding is subjective as this was another occurrence; some time has passed since the last event. I did not see a fix, a workaround or a plan w.r.t. these issues.

Reporting things is what this is for, but people have been giving you lots of suggestions and trying to help, and you're being accusatory and repeating yourself.