Add -flushwalletinterval command line arg #21760

Dear sirs or ladies. I'm experiencing issue with large BDB wallet: it's hammering disk io heavily (when wallet is flushing to disk every 500ms) and node requires a fast filesystem to keep working normally. To avoid situations like that, flushing can be turned off using -flushwallet=0, however it looks like walking on thin ice from point of wallet consistency in case of possible power/hardware outage.

I believe issue can be resolved less radically, if there will be a possibility to increase flush interval - this would relax disk usage significantly and at same time let us to still have some guarantees

"Flush" is somewhat poor terminology here. The data is always flushed to disk after each write. When Bitcoin Core believes the data is written to disk, it is actually written disk somewhere. The "flushing" in this context is actually compacting the wallet database so that everything is in the main database file. Without this "flushing", some wallet data will be written to a database log file (BDB uses write ahead logging). In the event of a power loss or other sudden software abort, the data should remain in those files and can be retrieved if Bitcoin Core is started normally. There is only an issue if only the wallet.dat file is copied to somewhere else following the crash event. In that case, the database log files will be missing when the wallet.dat file is loaded and so data is lost. So long as you also copy (and backup) the database log files, then you will be fine to use -flushwallet=0.

"Flush" is somewhat poor terminology here. The data is always flushed to disk after each write. When Bitcoin Core believes the data is written to disk, it is actually written disk somewhere. The "flushing" in this context is actually compacting the wallet database so that everything is in the main database file. Without this "flushing", some wallet data will be written to a database log file (BDB uses write ahead logging). In the event of a power loss or other sudden software abort, the data should remain in those files and can be retrieved if Bitcoin Core is started normally. There is only an issue if only the wallet.dat file is copied to somewhere else following the crash event. In that case, the database log files will be missing when the wallet.dat file is loaded and so data is lost. So long as you also copy (and backup) the database log files, then you will be fine to use -flushwallet=0.

I make backups with restic and copy entire walletdir during backup (which includes wallet.dat and log files), however in some scenarios (like filesystem writeback cache, for example, log file can be corrupted during power loss), especially with #18923 (If I understand correctly, transactions will be accumulated in log files until wallet finishes normally) I would prefer to have less transactions in log files and more in wallet.dat (for example flush data to wallet.dat each 2-3-5 min would be great). I think that in case of big wallet write transactions to wallet.dat every 500ms is a bad idea, at same time not to write them at all is also bad practice 🤷‍♂️ It's like MySQL which makes sql dump each 500ms or stores all data in binlog until restart)

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

No conflicts as of last run.

I will send resolve commit for #19101 if merged

@achow101 I've built binaries and performed simple test:

1. Run bitcoind node regtest built from my PR
2. Mine some coins, create some addresses and sent some TX to have some data in wallet (to be able to create visible load during flushing), wallet is 70MB now (about 45K of blocks, 50K addresses).
3. Run script which is creating new address every 5 sec. First pass with -flushinterval=500 (500ms - default), second pass with -flushinterval=120000 (120s)

Disk load is much lower, and I think it's much more acceptable behaviour (for my case of course, someone else can set anything else). In situation where last log file gets corrupted, it will be unpleasant, but nothing fatal will happen.

Hm... Even with -flushwallet=0 DB is compacting from time to time. Where in sources can I find DB compaction logic?

Yes, "flush" is a terrible name. It should really be called "compact", as the goal is to merge into one .dat file.

That said, the logic is in MaybeCompactWalletDB() which calls the unfortunately named BerkeleyDatabase::PeriodicFlush(). Mind that Sqlite has no equivalent functionality, so if you create a new-style wallet, you will get rid of this problem as well.

@laanwj With my setup I have no option to create new wallet or migrate to SQLite( Also if I understand correctly, SQLite isn't production ready in bitcoind and needs massive investigation from point of performance, right? Are there additional options to manage BerkeleyDatabase::PeriodicFlush() intervals (when I set -flushwallet=0, compaction still happens from time to time)?

(when I set -flushwallet=0, compaction still happens from time to time)?

It would be good to fix it instead of adding another option. #18923 was the last PR to touch that AFAIK

(when I set -flushwallet=0, compaction still happens from time to time)?

It would be good to fix it instead of adding another option. #18923 was the last PR to touch that AFAIK

I checked this PR and see that MaybeCompactWalletDB should work properly (never to be called with -flushwallet=0), but as I see, bdb has some kind of magic which can trigger log flush at any moment.

As I see from bdb docs there are no configuration options to affect this behaviour safely.

Closing this PR, as the conclusion was that there is no way to get around this behavior at all, so adding this option wouldn't fix the underlying issue.