Move logging to background thread #17216

To increase parallelism, the logging could be moved to a background thread.

Currently, writing to the debug log is done in-line (synchronous) and unbuffered. See https://github.com/bitcoin/bitcoin/blob/a22b62481aae95747830bd3c0db3227860b12d8e/src/logging.cpp#L56 That means that on slow disks (non-ssd, mostly), calls to LogPrint*() are potentially expensive and blocking further execution of the calling thread (potentially the net processing or validation thread).

Moving logging to a background thread should be fine, except for some edge cases:

• We use assert_debug_log in the functional tests, which assume that a message is written to the debug log synchronous. However, the recently added polling to assert_debug_log should work around that.
• The logging thread should probably be started when logging is started and shut down when the node is shut down. Calls to logging during shutdown should probably be in-line.

Like #13200?

Ah, yes. "Up for grabs" I guess.

I have a branch of a rebased #13200. I could use some help benchmarking to make a good case for reopening.

The instructions on how to run bitcoinperf are here: https://github.com/chaincodelabs/bitcoinperf#example-local-usage-no-docker

I'd suggest to not run it on your normal machine, as it installs quite a bunch of crap.

Ran some benchmarks on an HDD machine:

In all, we saw a 2.6% speed-up from for the first 200,000 blocks and a negligible slow down for blocks 500,000 to 525,000 with a 1% reduction in peak memory usage.

First 200_000 blocks

async-log vs. master (absolute)

async-log vs. master (relative)

Blocks 500_000 to 525_000

async-log vs. master (absolute)

async-log vs. master (relative)

Ran on:

cpu_model_name         Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz
ram_gb                 7.712368011474609
os                     ['Debian GNU/Linux', '9', 'stretch']
arch                   x86_64
kernel                 4.9.0-8-amd64
read_iops (/tmp)       214
write_iops (/tmp)      71
read_iops (.)          255
write_iops (.)         84

There is evidence that logging slows down some tasks by a factor of 5 (500% overhead) on non-intel low-end machines:

• #18538 (comment)

The feature request didn't seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.

Pull requests with improvements are always welcome.

Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.

Isn't that a point of labels and milestones? You can leave issues open to accumulate various perspectives for reference, and just mark the particular issues being worked on for the current milestones?

The issue will be around forever. It is just closed, not removed. But if there is little gain and interest, it doesn't make sense to keep it open forever. Otherwise the list would be unbounded, because there are unlimited ideas that one can come up with that provide a little edge-case gain and have trade-offs.