Berkeley DB uses a database transaction log to ensure Durability (the D of ACID). But we already have a couple of options enabled that remove the guarantee of Durability anyways (we enable DB_TXN_WRITE_NOSYNC). So the log already isn’t as useful to us as it could be. Furthermore, we don’t really make use of the BDB’s transaction feature. Almost every single read and write that we do is itself atomic and a single transaction. We do each read and write of a record as a single transaction. Flushing is always done when a WalletBatch is destroyed, and we have many single use WalletBatch objects.
We do, in a few places, batch transactions so even after a transaction is committed, it may not be flushed to the database. In those instances, the log is useful as it does recover whatever wasn’t written. But whatever didn’t get written also doesn’t matter. If a new transaction was being written, the blockchain can be rescanned. If new metadata was being written, the user can just re-enter it. For newly generated keys, those keys didn’t see any use if their records didn’t get flushed to the db (that flushing always happens before the key generation function returns). So it’s fine to lose those keys, and for HD wallets, they will be regenerated anyways, so nothing is lost.
The single exception to the single record single transaction that we usually do is with encrypting existing private keys. This encryption is done in a single transaction so that it is atomic, and that is many records being written. But even with that, the transaction log isn’t useful. We abort before anything is flushed to disk so the wallet state is not inconsistent and all of the unencrypted private keys are still there. The encryption can then be redone. So the log is not useful there.
Additionally, we are already trying to get rid of the transaction log as much as possible. We turn on the auto remove (DB_LOG_AUTO_REMOVE) and the logs are always cleaned up on a clean shutdown. As mentioned earlier, the transaction log already has a flag that doesn’t always flush the log anyways. So I don’t think that just completely avoiding transaction logs entirely is unreasonable.
Logs are disabled by using the db flag DB_TXN_NOT_DURABLE to prevent logs being written for a db, and using the db env flag DB_LOG_IN_MEMORY to keep all db environment logs in memory.
Not having the BDB logs additionally allows us to remove the BDB 4.8 requirement. This requirement was because the Oracle keeps changing the log file format. The database file format itself has not changed in a long time. The log file format difference would supposedly make downgrades more difficult because the log files are not downgradable. However in my testing, this was not actually the case as the log files would be moved out of the way if the database could not be opened, so downgrading wasn’t that much of a problem, especially with clean shutdowns. Not having the transaction log be written at all makes this not a problem entirely as there aren’t any log files to get in the way. So we can now allow for other BDB versions.
Bumping to BDB 5.3 is done only because that’s the version that is provided by many (most?) distros. With this change, --with-incompatible-bdb is no longer needed and we can simply build and installation instructions to just “install bdb from your package manager”.
This change might be contentions, so here are some alternatives that we might want to discuss:
- Not having logs at all might make the wallet more unreliable. Maybe instead of getting rid of the logs entirely we can output compatible log files for use in BDB 4.8. But that’s clunky and just locks us into BDB 5.3 (or whatever version).
- We could possibly have our own consistent log file format which is then loaded into BDB somehow
- Instead of no logs at all, when we encounter newer log files, copy them (as we do now) and the wallet.dat file to a backup location in the walletdir so that users can recover them if they see consistency issues with the wallet file. This still lets us change BDB.
