“Unable to parse settings.json” at startup after disk was full #21340

1. 
   laanwj commented at 6:37 pm on March 2, 2021: member

   This was reported to me by someone who had a crash issue after the disk was full, ostensibly unable to write settings.json and ending up with a truncated file. The solution was obvious in retrospect: delete it. But not everyone is going to figure this out by themselves.

   Should this error be fatal at all? Could it be ignored and logged? Or is a warning then deleting it better (at least in the GUI, that’s harder to do for the daemon).

   Another thing to look into might be to write the file atomically so that this doesn’t occur.

2. laanwj added the label Utils/log/libs on Mar 2, 2021
3. 
   mat888 commented at 1:17 am on March 3, 2021: none

   So the settings.json file is becoming corrupted after the disk is full? That seems like a problem which should be avoided. I’m not sure deleting it, and removing any settings which may be important to the user should be the official solution.
4. 
   laanwj commented at 12:01 pm on March 3, 2021: member

   Yes, it is a problem that should be avoided (hence “Another thing to look into might be to write the file atomically so that this doesn’t occur.”), but even then corruption can still happen and there needs to be a way to recover. I do agree silently deleting it would not be a good thing. A dialog at startup to continue with “reset settings” might do the trick better.
5. ryanofsky referenced this in commit cf472050f2 on Jul 8, 2021
6. 
   ryanofsky commented at 7:35 pm on July 8, 2021: contributor

   I think bitcoin-core/gui#379 should resolve this. It clarifies the error and adds an option in the gui to reset the file
7. 
   kristapsk commented at 10:13 pm on July 8, 2021: contributor

   So the settings.json file is becoming corrupted after the disk is full?

   When disk is totally full on Linux, any file randomly can be truncated to 0 size (the one you will try to overwrite at that moment). Guess solution could be to rename old file to temporary name before writing the new one and deleting old one only after the new one is saved correctly?

8. 
   ryanofsky commented at 11:02 pm on July 8, 2021: contributor

   When disk is totally full on Linux, any file randomly can be truncated to 0 size (the one you will try to overwrite at that moment).

   Interesting. Is this documented somewhere? ~It definitely suggests regardless of any write errors that the settings read code should be treating an empty file the same as a non-existent file, and just ignoring it instead of triggering a fatal error.~ (EDIT: I don’t think this anymore, see below)

   This is an easy change to make but I’d also be curious to hear from @laanwj if he knows whether the file was truncated to size 0.

   Guess solution could be to rename old file to temporary name before writing the new one and deleting old one only after the new one is saved correctly?

   I could be misunderstanding but I think this describes what happens currently: https://github.com/bitcoin/bitcoin/blob/df2b5dadfb2dc21b05e5a60b71987fb5b96201f5/src/util/system.cpp#L572-L579, but maybe there could be additional syncs or checks added?

9. 
   kristapsk commented at 11:09 pm on July 8, 2021: contributor

   Interesting. Is this documented somewhere?

   Haven’t looked at the kernel code in detail for this, just my personal experience (had old blog written in PHP long long time ago where everything was saved in text files instead of proper database and file got truncated to zero bytes when partition ran out of space, IIRC code just read everything into memory, modified and then tried to overwrite old file).

10. ryanofsky referenced this in commit ebfd619e3c on Jul 13, 2021
11. 
    ryanofsky commented at 7:20 pm on July 14, 2021: contributor

    It definitely suggests regardless of any write errors that the settings read code should be treating an empty file the same as a non-existent file, and just ignoring it instead of triggering a fatal error.

    Thinking about this more, I disagree with what I wrote earlier. Even if a settings file is truncated to size 0 and the old settings are completely lost (as opposed to partially corrupted), I still think the safest thing for bitcoind to do is refuse to start until the corrupt file is cleared away. It would unsafe to assume no important settings were saved before the corruption happened and just start anyway, because there could have been critical settings affecting security (sandbox settings) or privacy (tor settings, wallet settings) or performance (pruning, coins cache size) that would be important not to ignore before starting. So I don’t think it would be good to treat a size 0 settings.json file differently than any other corrupt settings.json file.

    I also don’t know of a way of avoiding corruption when writing to a full disk that would work better then the current approach of renaming a new file over the old one. Maybe bitcoin should generally shutdown or avoid writing things if disk space gets too low, though.

    I think since bitcoind is non-interactive, the best thing it can do on startup when settings are corrupt is show an error about the corruption, and exit like it is currently doing. Since bitcoin-qt is interactive, I think the change in bitcoin-core/gui#379 adding a dialog button to reset settings provides a decent way to recover there, and resolves this issue as well as we can for now

    

12. ryanofsky referenced this in commit 1ee6d0b01a on Jul 29, 2021
13. hebasto closed this on Aug 6, 2021

    ---

14. hebasto referenced this in commit 7ebc4c6689 on Aug 6, 2021
15. sidhujag referenced this in commit 1f3f166be5 on Aug 6, 2021
16. stratospher referenced this in commit e395501624 on Aug 14, 2021
17. luke-jr referenced this in commit 0ce2f5b846 on Oct 17, 2021
18. janus referenced this in commit ad36bf5dde on Nov 5, 2021
19. 
    kristapsk commented at 8:08 am on July 20, 2022: contributor

    I think since bitcoind is non-interactive, the best thing it can do on startup when settings are corrupt is show an error about the corruption, and exit like it is currently doing.

    Had this on a Debian based machine recently. This error isn’t written in debug.log (makes sense, settings haven’t been loaded yet, so you don’t know location of a file), on Systemd based machines only way to see this error will be to sudo su - bitcoin and then manually try to launch bitcoind process. Which isn’t too user friendly.

     0$ sudo systemctl status bitcoind
     1* bitcoind.service - Bitcoin daemon
     2     Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; vendor preset: enabled)
     3     Active: activating (auto-restart) (Result: exit-code) since Wed 2022-07-20 10:00:17 CEST; 23s ago
     4    Process: 1354 ExecStart=/usr/local/bin/bitcoind -daemon -pid=/run/bitcoind/bitcoind.pid -conf=/home/bitcoin/.bitcoin/bitcoin.conf -datadir=/home/bitcoin/.bitcoin -startupnotify=chmod g+>
     5
     6Jul 20 10:00:17 hostname systemd[1]: bitcoind.service: Control process exited, code=exited, status=1/FAILURE
     7Jul 20 10:00:17 hostname systemd[1]: bitcoind.service: Failed with result 'exit-code'.
     8Jul 20 10:00:17 hostname systemd[1]: Failed to start Bitcoin daemon.
     9$ sudo journalctl -u bitcoin
    10-- Logs begin at Wed 2022-07-20 09:57:42 CEST, end at Wed 2022-07-20 10:00:26 CEST. --
    11-- No entries --
    12$ sudo su - bitcoin
    13$ bitcoind -daemon -pid=/run/bitcoind/bitcoind.pid -conf=/home/bitcoin/.bitcoin/bitcoin.conf -datadir=/home/bitcoin/.bitcoin
    14Error: Failed loading settings file:
    15- Unable to parse settings file /home/bitcoin/.bitcoin/settings.json
    

20. 
    kristapsk commented at 8:37 am on July 20, 2022: contributor

    Another thing - in this case settings.json was anyways “empty” (just { }). Probably we should avoid creating it at all if there are no settings to save there?
21. 
    ryanofsky commented at 4:50 pm on July 28, 2022: contributor

    Another thing - in this case settings.json was anyways “empty” (just { }). Probably we should avoid creating it at all if there are no settings to save there?

    I don’t have a firm opinion on this, but the settings file is written on startup unconditionally (unless -nosettings is specified) to:

    • Show an error message as early as possible if the file is not writable, instead of later when wallets are loaded or when settings are changing, so changes are not lost if there is a write error and problems can be addressed more easily.
    • Help debug configuration issues. Let someone see what settings file is in use from the directory listing and file modification time and avoid confusion about what settings path is in use
22. 
    ryanofsky commented at 4:56 pm on July 28, 2022: contributor

    Had this on a Debian based machine recently. This error isn’t written in debug.log (makes sense, settings haven’t been loaded yet, so you don’t know location of a file), on Systemd based machines only way to see this error will be to sudo su - bitcoin and then manually try to launch bitcoind process. Which isn’t too user friendly.

    This seems bad. I would expect the message also to be written to stderr and show up in the journald log file. I noticed the command you used is journalctl -u bitcoin and maybe that should be journalctl -u bitcoind? But maybe also there is problem with the debian systemd file not capturing stderr. This seems like something that could be addressed and discussed further if you have more information and want to open an issue about the missing error.

23. 
    dongcarl commented at 3:37 pm on March 20, 2023: contributor

    One of my boxes running v24.0.1 just encountered a low storage situation, and /var/lib/bitcoind/settings.json was truncated to 0 size.

    From my journalctl:

    0Mar 20 11:33:05 hostname bitcoind[935629]: Error: Failed loading settings file:
    1Mar 20 11:33:05 hostname bitcoind[935629]: - Unable to parse settings file /var/lib/bitcoind/settings.json
    2Mar 20 11:33:05 hostname systemd[1]: bitcoind.service: Control process exited, code=exited, status=1/FAILURE
    

    I’m not sure what to do in a situation like this, is trying to proceed without the settings applied safe? Is it possible the user might have set pruning settings in settings.json and proceeding without the settings applied would worsen the low storage situation?

    This seems bad. I would expect the message also to be written to stderr and show up in the journald log file.

    Unfortunately the current bitcoind.service in the repo does not print logs to journald, which I think is unfortunate. Here’s my fix: #25975

24. PastaPastaPasta referenced this in commit 358df3762d on Feb 13, 2024
25. PastaPastaPasta referenced this in commit 8f230727d2 on Feb 13, 2024
26. PastaPastaPasta referenced this in commit b2722e92db on Feb 14, 2024
27. PastaPastaPasta referenced this in commit 65b7ec3366 on Feb 14, 2024
28. bitcoin locked this on Mar 19, 2024

Contributors
laanwj mat888 ryanofsky kristapsk dongcarl

Labels
Utils/log/libs