Fix >2GB debug.log on 32bit and trunc'ing debug.log on all platforms #439

This patch does two bug fixing things and has two documentation corrections:

bug fix 1) It allows 32bit builds to write to debug.log past 2GB. I need someone to test this on bsd and win32. bug fix 2) Change GetFilesize to use stat() instead of fseek() to prevent an integer overflow that would prevent >2GB debug.logs from getting trimmed on most (any?) platforms. This function is only currently called by ShrinkDebugFile which has been updated to pass a string instead of a FILE.

documentation update) Add -logtimestamps and -printtoconsole to -? output.

This can be tested with:

dd if=/dev/zero of=debug.log bs=$((1024 * 1024)) count=2048; tail -f debug.log & bitcoind -daemon;

If logs get appended to where debug.log continues past 2GB it is working. Must be tested on a 32bit build.

debug.log should also no longer be truncated on startup as that should be left to the user. This can be tested by restarting bitcoind after the fact and ensuring it is not truncated.

The additional help output is pretty straight forward to test: is it there? :)

Your GetFileSize change is a regression, introducing a minor race window. If we have an open FILE*, then we may fstat via fileno. The current code implicitly does this, by working on the already-open file.

Your change refers to the file by name, which may have been renamed, and may no longer refer to the already-open file.

win32 does not have a fstat() that I can find. That is why I did it this way. There doesn't seem to be a way to easily guarantee that fseek()/ftell() will give you a 64bit int. Suggestions?

Also it actually doesn't add any new kind of race condition. It's only first opened a line before the call (and this is the only caller) and gets closed and reopened immediately afterwards if the file with that filename at the time is longer than the specified hardcoded value.

Worst case we'll get old logs from a different file appended to the top of the new debug.log than was actually in the old debug.log we had opened if a user changes debug.log between open and Getfilesize call. Since the contents of debug.log are never parsed by bitcoin it doesn't add any form of attack/dos vector that isn't accomplished by just rm debug.log which you would have to already be able to perform in order to take advantage.

Windows supports _ftelli64.

The ftell approach remains superior because it avoids the aforementioned regression.

There we go, race condition gone.

Seems to me this is fixing the symptom, not the disease. The disease is writing way too much to debug.log by default.

I agree that the default logging level probably needs to be reworked; but, this is a problem in itself. No software should just stop logging out of the blue unless it has no space on the device it's logging to.

Added test instructions to description:

This can be tested with:

dd if=/dev/zero of=debug.log bs=$((1024 * 1024)) count=2048; tail -f debug.log & bitcoind -daemon;

If logs get appended to where debug.log continues past 2GB it is working. Must be tested on a 32bit build.

I have to agree that we shouldn't have a 2GB+ logfile in the first place. For the majority of people, I would think that the advantage of having verbose logs is outweighed by the inconvenience of their size.

#515 - logfile size reference

#125 - log filters #520 - streamlining the output

I'm definitely not qualified to appreciate the inner-workings of the logfile. That being said, I would think a 2GB cap is sensible (intentional or not). Please correct me if I'm mistaken.

I agree with @jrmithdobbs here. >2GB log files are a problem in itself, but it should not simply stop logging after that, that can be dangerous and lead to non-debuggable issues.

This seems to have been forgotten over time. Looking at it again, it seems to remove the automatic pruning of debug.log. Is this intentional/wanted?

Hm I understood from the bug reported was that it stopped logging after 2Gb. It's good to solve that.

Auto-pruning should not be disabled though, IMO, as most users don't want to be concerned with the debug.log file at all. So having it grow to fill their disk is not so nice. Pruning at startup when the file is big is a good compromise.

But processing a 2GB file should be quite time-intense and will for sure hurt user-experience!?

Processing how? It's never read back in let alone "processed."

My removing of the pruning was intentional. Auto-pruning logs is just bad practice. What if I need to look at something in my log from a month ago? Things log way too verbosely by default, sure, fix that don't mangle the log.

Feel free to close this. It needs fixing but I have no interest in rebasing/etc.

I meant Auto-pruning a large log-file could be time-intense, no?