gui: Pruning keeps getting reenabled #19291

<!-- This issue tracker is only for technical issues related to Bitcoin Core. General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange at https://bitcoin.stackexchange.com. For reporting security issues, please read instructions at https://bitcoincore.org/en/contact/. If the node is "stuck" during sync or giving "block checksum mismatch" errors, please ensure your hardware is stable by running memtest and observe CPU temperature with a load-test tool such as linpack before creating an issue! -->

<!-- Describe the issue -->

Expected behavior

I want my Bitcoin Core node to never prune the blockchain.

<!--- What behavior did you expect? -->

Actual behavior

Occasionally, Bitcoin Core will enable the pruning option. This causes it to prune the blockchain, and forces me to resync when I disable it. I started seeing this in 0.19 and it still persists in 0.20. It may also be related to my moving to a new computer which upgraded me to Catalina. I'm not really sure.

<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

To reproduce

I think it might be related to the fact that my data directory is on an external hard drive. If I start Bitcoin Core before mounting in the drive, it will popup the dialog as if it is my first time starting Bitcoin Core, asking me where to store the block chain. In the dialog, the datadir location is defaulted to put it in the location where my external hard drive mount would be, like /Volumes/Drive/Bitcoin. If I then mount the drive and choose next in the dialog, the sync works, but pruning is enabled and it immediately prunes the blockchain. I can go into the settings at this point and uncheck pruning, but at that point, it requires a full resync. And it ends up being enabled again the next time I start Bitcoin Core. Needless to say, this is extremely annoying and makes Bitcoin Core almost unusable for me.

I even tried setting prune=0 in my bitcoin.conf file, which is on the external drive, but it has no effect.

I suspect what is happening is that pruning is enabled by default, and since it can't find my datadir when it launches, Bitcoin Core thinks that it is doing a fresh sync and sets it to true. This setting is then not updated, even from the bitcoin.conf (which itself lives on the external drive) once that datadir is loaded. The fact that it remembers where the datadir is implies that some settings are stored locally, though I can't really tell where. Is it possible for me to store my no-prune setting locally as well?

Note that I used to do this exact workflow of opening Bitcoin Core before mounting my drive all the time in previous node versions and macOS versions, and it never enabled pruning on me.

(as an aside, I feel like Bitcoin Core should be smart enough to recognize that a datadir in /Volumes on macOS is an external drive, even when it doesn't currently exist).

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

System information

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->

Bitcoin Core 0.20.0 macOS GUI version. Downloaded from https://bitcoincore.org/.

<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

macOS Catalina 10.15.5.

<!-- GUI-related issue? What is your operating system and its version? If Linux, what is your desktop environment and graphical shell? -->

<!-- Any extra information that might be useful in the debugging process. -->

<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

Contents of /Volume/Drive/Bitcoin/bitcoin.conf:

server=1
dbcache=4096
rpcport=8332
prune=0

What does "Bitcoin Core will enable the pruning option" mean? How do you observe that it is in fact pruning?

No setting should ever change automatically.

"Smart" pruning was introduced to the welcome screen in #16714. If the welcome screen is shown, it will enable pruning if there it believes there is not enough space for the blockchain. This pruning setting is stored in the Qt settings which is stored independently from the datadir and is platform specific. The welcome screen will be shown if the configured datadir does not exist and wasn't specified in the command line options.

If you add the -datadir=/Volume/Drive/Bitcoin to the shortcut that you use to launch Bitcoin Core, it should just throw an error if the datadir isn't mounted and the welcome screen won't be shown so you pruning won't be enabled.

What does "Bitcoin Core will enable the pruning option" mean? How do you observe that it is in fact pruning?

I observe it in the debug log, as well as by the fact that the option becomes checked in the settings.

If you add the -datadir=/Volume/Drive/Bitcoin to the shortcut that you use to launch Bitcoin Core, it should just throw an error if the datadir isn't mounted and the welcome screen won't be shown so you pruning won't be enabled.

I just launch the .app GUI on macOS. How can I add the option to that?

I just launch the .app GUI on macOS. How can I add the option to that?

Haven't tried, but googling turned up this https://superuser.com/questions/16750/how-can-i-run-an-application-with-command-line-arguments-in-mac-os

This appears to be a real bug. Thank you for taking the time to report this!

During startup, Intro::showIfNeeded will be called to show the intro if the datadir doesn't exist. The Intro dialog will check or uncheck the prune checkbox automatically depending on how much disk space it thinks is free.

If I start Bitcoin Core before mounting in the drive, it will popup the dialog as if it is my first time starting Bitcoin Core,

If you manually uncheck the pruning checkbox in the intro dialog that pops up when the drive is not mounted you will avoid this problem, but if you forget to uncheck the pruning box, leaving it checked will enable pruning (setting gets applied with BitcoinApplication::InitializePruneSetting force=true) and overrides the bitcoin.conf setting.

There are a few changes we can can make to fix this. One we should probably make is let the bitcoin.conf setting always take precedence, ignoring whether pruning is checked or unchecked in the intro dialog. This would be an improvement because bitcoin.conf already takes precedence over the gui setting in the normal case where bitcoin app is started and the intro dialog doesn't execute. Obviously though, it is not great to show the pruning checkbox in the intro dialog and then ignore it.

I think a better fix would be to change Intro::showIfNeeded to not show the intro dialog straight away when a gui datadir path is set (QSettings "strDataDir" entry exists) and the directory doesn't exist. I think in this case it would be better to show a different dialog "Error: Bitcoin data directory "/Volume/Drive/Bitcoin" does not exist" with options "Choose a new data directory" "Quit" to give more clarity about what is going on and give the user a chance to mount the drive and avoid the intro dialog mess.

Additionally it would also be good if the intro dialog didn't just check for free disk space in the selected data dir, but also loaded any bitcoin.conf file there to see if it contains a pruning setting. If it does contain a pruning setting it should disable the checkbox and make it unclickable, leaving it checked if automatic pruning is enabled in the bitcoin.conf, unchecked if it is disabled. This is actually partially implemented already. There is currently code to disable and force check the checkbox when pruning is explicitly enabled, but no code to disable and force-uncheck the checkbox when it's explicitly disabled, and also no code to read the contents of bitcoin.conf file from the currently selected datadir (right now it is just looking at the command line -prune value).

Note: #15936 indirectly relates to this bug because it currently changes settings precedence from gui settings < bitcoin.conf settings < command line settings to bitcoin.conf settings < gui settings (settings.json) < command line settings. So with #15936 the GUI setting always overrides the bitcoin.conf setting instead of sometimes overriding it (not inconsistently depending on whether intro dialog executed). #15936 actually makes the bug worse here since the gui setting is wrong in this case, but this can be fixed with changes above, and changing the precendence does have some other benefits I need to write down somewhere, probably https://github.com/bitcoin-core/bitcoin-devwiki/wiki/Settings-design-questions

I think a better fix would be to change Intro::showIfNeeded to not show the intro dialog straight away when a gui datadir path is set (QSettings "strDataDir" entry exists) and the directory doesn't exist. I think in this case it would be better to show a different dialog "Error: Bitcoin data directory "/Volume/Drive/Bitcoin" does not exist" with options "Choose a new data directory" "Quit" to give more clarity about what is going on and give the user a chance to mount the drive and avoid the intro dialog mess.

So to be clear, is this workflow of loading an existing datadir into the "new custom data directory" location after Bitcoin Core starts actually supported. It has always worked for me in the past (sans this issue), but maybe there are other bitcoin.conf settings that wouldn't get retroactively changed either?

So to be clear, is this workflow of loading an existing datadir into the "new custom data directory" location after Bitcoin Core starts actually supported. It has always worked for me in the past (sans this issue), but maybe there are other bitcoin.conf settings that wouldn't get retroactively changed either?

If the question is whether bitcoin.conf settings in the datadir will get applied after choosing the datadir in the intro dialog, the answer is yes. This already works for every setting except the pruning setting, because of the current startup sequence where bitcoin.conf is only first loaded after the intro dialog is closed. The reason pruning is a special case is because we have special code that runs after the intro dialog box is closed and after bitcoin.conf is loaded, that goes back and applies the intro dialog prune checkbox value on top of the bitcoin.conf value, if the intro dialog had been shown. The three fixes I suggested for this are:

1. Make bitcoin.conf setting always override intro dialog pruning checkbox value if present, by narrowing the special case code mentioned above.
2. Avoid showing the intro dialog altogether when a datadir location is missing, and instead show a "Data directory not found" dialog. This way a user who just forgot to mount an external drive isn't blindly thrown into the intro sequence again, unless they actually request to choose a new data location.
3. Fix the pruning checkbox default checked state in the intro dialog to reflect the bitcoin.conf pruning setting in an existing datadir instead of ignoring it.

For unusual circumstances it needs to look at the free space it has in it's block storage directory, since pruning loves to start up overriding the conf file. Thankfully, getting the guisettings to popup was the fix that worked.

As this is pretty much a pure GUI issue, does it make sense to move it over to that repo @hebasto ?

Moved to https://github.com/bitcoin-core/gui/issues/709.