In a recent blog post Wladimir writes that:
It turns out that [tests run slowly] because the wallet code does an fsync after every operation to make sure that changes to the database are safely written to disk
ThreadFlushWalletDB() in src/wallet/walletdb.cpp has roughly the following logic:
every 500 milliseconds:
if (the wallet has been modified since the last time it was flushed to disk &&
the wallet hasn't been modified for at least 2 seconds &&
we can get a database lock &&
the database isn't currently in use):
then
flush it to disk
This has a couple of problems:
-
If the wallet receives a deposit every second, the “hasn’t been modified for at least 2 seconds” condition is never met, and so the wallet is never flushed to disk. Perhaps the loop needs to have an extra check, “if the wallet wasn’t flushed in the last N1 seconds then flush it now even if it was modified in the last 2 seconds”.
-
If the wallet receives a deposit every 2.5 seconds, the “hasn’t been modified for at least 2 seconds” condition is almost always met, and so the wallet is continually flushed to disk. The wallet file grows bigger with each deposit, and the flushing causes a full rewrite of the file. This causes a quadratically increasing disk usage over time, eventually swamping the capacity of the drive. Perhaps the loop needs to have an extra check: “if the wallet was flushed in the last N2 seconds then don’t flush it now, even if it wasn’t modified in the last 2 seconds”.
Is there some way of safely updating a BDB database without rewriting the whole file to disk for every update? This seems like a major inefficiency.
I’ve been struggling with this issue of ever increasing hard drive use as wallet size grows for years now, but only recently discovered the root cause of the problem.
