Background chainstate compaction (#35465) filled the disk; node then hung after shutdown (v32.0rc3) #36585

This is my first issue on bitcoin/bitcoin, so apologies in advance if anything is in the wrong place. I'm happy to provide more data or run tests.

Is there an existing issue for this?

I searched and found none. The change involved is #35465 ("coins: compact chainstate regularly"). It fixes #35298 and was backported to 29.4, 30.3 and 31.1 (#35450, #35452, #35331). I observed this on v32.0rc3. As far as I can tell, v32.0rc4 and current master have no changes to this code.

Current behaviour

A pruned mainnet node ran out of disk space during what appears to be the new periodic background chainstate compaction. The node detected the low disk space and ran its shutdown sequence, but the process never exited, so the node stayed down until it was stopped by hand about 5.7 hours later.

The same node had completed a full compaction about 10 hours earlier using very little extra space. The failing compaction needed more than 5 GiB extra.

Setup

• 25 GiB volume (24.9 GiB usable), XFS. The datadir is on the same volume.
• 5.03 GiB free (79.8% used) immediately before the failing compaction.
• Chainstate: 10.5 GiB in 327 .ldb files.

Disk figures are from telegraf, sampled once a minute, so short peaks between samples are not captured.

First compaction on this node: completed normally (2026-10-19)

All 327 chainstate .ldb files were rewritten between 20:45 and 20:48 UTC.

Second compaction on this node: filled the disk (2026-10-20)

Nothing was logged about either compaction at the default log level, because the start and finish messages are LogDebug(BCLog::COINDB). debug.log has no entries between 06:48:41 and the first error at 07:01:43. The timing above comes from file modification times and the disk metrics.

After restart. When the node was restarted, LevelDB removed the partial outputs and the chainstate returned to 10.5 GiB. The node then synced normally.

A second node. This node has the same version and configuration and the same 25 GiB volume. It did a full compaction on 2026-10-17 (all 329 files rewritten, 11:29-11:32 UTC) and completed it. However, about 2 GiB of extra disk usage remained for more than two hours after the compaction's file writes had stopped:

debug.log on that node has nothing around 11:33 or 13:48 apart from normal block processing. Nothing on either node calls gettxoutsetinfo, scantxoutset or dumptxoutset.

Comparison with the PR measurements. In #35465, a simulation of 22 compactions reported at most 1.06 GiB of temporary extra disk space. The first compaction on the failing node matches that. The second one wrote more than 5 GiB of new table files without releasing any input files, and its true peak was higher because it did not finish.

Possible explanation (from reading the code, not confirmed)

I have not been able to confirm this, because -debug=leveldb was not enabled. From reading the LevelDB code in the tree:

• Obsolete .ldb files are only removed in DBImpl::DeleteObsoleteFiles(), which runs in three places: after a memtable compaction, after each background or manual compaction step, and at DB::Open.
• Any file still referenced by a live Version at that moment is kept. When the last reference is dropped later, Version::Unref() frees the Version but does not trigger deletion, so the file stays on disk until the next DeleteObsoleteFiles() call.
• The chainstate is only written during Core's periodic flushes (every 50-70 minutes), so the next call can be an hour or more away.

That could explain the second node: the final step of the full compaction has no later step to clean up after it, and the ~2 GiB held matches the size of that step's output (58 files, 1.99 GB, written at 11:32). The space was released at 13:48, which is about two flush intervals after the compaction (it was triggered by a flush at about 11:29). I don't know why it wasn't released at the first flush after the compaction, or what was holding a reference.

If something held an old Version for a long time on the failing node, it could also explain why none of the 327 input files were deleted while 191 output files were written, even though every compaction step calls DeleteObsoleteFiles(). The other possibility is that one or more steps were very large. My guess is that it matters that this compaction ran only about 10 hours after a previous full compaction, when all files are in the bottom level. The data I have can't tell these apart.

The post-shutdown hang. This may be a separate issue. When systemd finally killed the process, the remaining threads were the main thread (named b-shutoff) and two b-capnp-loop threads. This is a multiprocess build, and a mining client was connected over IPC at the time. The hang turned what would have been a short outage (the restart recovered immediately) into a 5.7-hour one. I'm happy to open it as its own issue if that's preferred.

Expected behaviour

Periodic maintenance compaction should not be able to take a running node down. For example:

• Check available disk space (for example, against the chainstate's on-disk size) before starting a full compaction, and skip or postpone it if there isn't enough headroom.
• Make sure obsolete table files are released promptly once a compaction finishes, rather than waiting for a later write.
• Log the start and finish of the compaction at the default log level, so operators can relate disk usage spikes to it.
• Optionally, provide a way for operators to disable or defer the compaction.

Separately, after a fatal error and Shutdown done, the process should exit so that a service manager can restart it.

Steps to reproduce

Not deterministic, because the trigger is a random 1/320 chance after each full flush. Conditions here:

• Pruned node (prune=550).
• 25 GiB volume with about 5 GiB free.
• A full compaction about 10 hours before the failing one.

I can try to reproduce on a copy of the chainstate (for example with -forcecompactdb and -debug=leveldb), or run with -debug=leveldb -debug=coindb until the next compaction. If there's a lower-overhead way to capture the relevant LevelDB events on a production node, I'd be glad to use it.

Relevant log output

debug.log:

2026-10-20T07:01:43.207490Z [error] A fatal internal error occurred, see debug.log for details: Disk space is too low!
2026-10-20T07:01:43.210613Z [error] FindNextBlockPos failed for FlatFilePos(nFile=-1, nPos=0) while writing block
2026-10-20T07:01:43.212811Z Shutdown in progress...
2026-10-20T07:01:43.212507Z [error] ProcessNewBlock: AcceptBlock FAILED (AcceptBlock: Failed to find position to write new block to disk)
2026-10-20T07:01:43.306973Z [error] SerializeFileDB: Failed to open file /data/.bitcoin/peers.fb8d
2026-10-20T07:01:43.307578Z [error] SerializeFileDB: Failed to open file /data/.bitcoin/anchors.bb45
2026-10-20T07:01:46.092854Z [error] A fatal internal error occurred, see debug.log for details: Disk space is too low!
2026-10-20T07:01:46.092888Z [warning] Failed to force flush state (Disk space is too low!)
2026-10-20T07:01:46.106919Z [error] A fatal internal error occurred, see debug.log for details: Disk space is too low!
2026-10-20T07:01:46.107061Z [warning] Failed to force flush state (Disk space is too low!)
2026-10-20T07:01:46.899647Z Shutdown done

systemd (no entries for the service between 07:01 and 12:42):

12:42:01 systemd[1]: Stopping bitcoin-node.service...
12:44:01 systemd[1]: bitcoin-node.service: State 'stop-sigterm' timed out. Killing.
12:44:01 systemd[1]: bitcoin-node.service: Killing process 3463 (b-shutoff) with signal SIGKILL.
12:44:01 systemd[1]: bitcoin-node.service: Killing process 3466 (b-capnp-loop) with signal SIGKILL.
12:44:01 systemd[1]: bitcoin-node.service: Killing process 3486 (bitcoin-node) with signal SIGKILL.
12:44:01 systemd[1]: bitcoin-node.service: Killing process 3303146 (b-capnp-loop) with signal SIGKILL.
12:44:01 systemd[1]: bitcoin-node.service: Main process exited, code=killed, status=9/KILL
12:44:01 systemd[1]: bitcoin-node.service: Failed with result 'timeout'.

Chainstate .ldb files by modification time, while the process was hung:

2026-10-19T20:45   72 files   2.49 GB
2026-10-19T20:46  106 files   3.66 GB
2026-10-19T20:47  107 files   3.70 GB
2026-10-19T20:48   42 files   1.44 GB
2026-10-20T06:55   55 files   0.78 GB
2026-10-20T06:56  113 files   3.91 GB
2026-10-20T06:57   23 files   0.76 GB

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v32.0rc3 (2633a1ded5) plus two local commits that don't touch src/leveldb, dbwrapper, txdb, coins or validation. Multiprocess build, run as bitcoin-node -ipcbind=unix under systemd (Type=simple, Restart=always).

Operating system and version

Amazon Linux 2023

Machine specifications

AWS EC2 instance with a 25 GiB EBS volume (XFS).

Relevant config:

prune=550
dbcache=2500
maxmempool=300
persistmempool=1

Thank you for the excellent and unusually detailed report. The disk measurements and table timestamps made the failure much easier to understand.

A 25 GiB volume is very optimistic for a long-running node, even with pruning. The current mainnet parameters assume a maximum chainstate size of 14 GiB. In the worst case, full compaction can temporarily require nearly another copy, so two maximum-size chainstates alone would need about 28 GiB, already exceeding the entire usable volume before accounting for the 550 MiB pruning target or anything else stored on the filesystem. Pruning limits block storage, not the chainstate, so the configured pruning target does not provide the headroom compaction needs. Before seek-triggered compaction was disabled, LevelDB combined ordinary size-triggered compactions with frequent incremental compactions caused by reads. Those smaller compactions reclaimed obsolete data continuously, but they also caused too much write amplification. #35465 subsequently added occasional full-range compactions, which reduce regular writes but can produce much larger temporary spikes that nearly double the chainstate’s disk usage. Simply disabling compaction would let obsolete data accumulate again, while postponing it would not guarantee that more space becomes available. A free-space check could prevent an unsafe full compaction, but nodes without enough headroom for a full copy would also need a way to reclaim data incrementally, with each step bounded so that old tables can be removed before the next step begins.

We should document this temporary-space requirement more clearly and consider checking that the chainstate filesystem has enough free space before compaction, in addition to the existing block-storage checks. Automatic compaction should also be logged at info level so operators can understand sudden changes in disk usage.

Finally, the hang after Shutdown done appears unrelated to compaction. Two related IPC shutdown bugs are already tracked privately, and I have pinged @ryanofsky on both reports with a link to this issue.

In the worst case, full compaction can temporarily require nearly another copy, so two maximum-size chainstates alone would need about 28 GiB

I wonder if the gui's first-time warning should be adjusted to account for that? Also, the daemon doesn't have a first-time warning about this, so a warning could be added there as well?

Thanks for the quick and thorough response, that makes sense. I've grown the volumes and will size for roughly two maximum-size chainstates going forward.

I agree with all three of your suggestions, and from an operator's side each one would have made a real difference here:

• Documentation: I sized the volume from the pruning target and the current chainstate size. I couldn't find anything that says a full compaction can temporarily need close to a second copy of the chainstate: not in doc/, the -prune help text, or the release notes (29.4, 30.3 and 31.1 list #35465 as a one-line changelog entry, and the draft v32.0 notes don't mention it). A note in the -prune help text and next to the chainstate/ entry in doc/files.md, plus a release-notes entry, would help. +1 to @maflcko's suggestion on the first-run warnings: the GUI currently estimates a pruned node's requirement as prune target + m_assumed_chain_state_size (src/qt/intro.cpp), which leaves no room for a compaction, and a similar warning in the daemon would have caught my setup.
• Free-space check: with a check against the chainstate filesystem, this compaction would have been skipped and the node would have stayed up.
• Info-level logging: it took a fair amount of digging through file timestamps and disk metrics to work out what had happened. A single "Starting chainstate compaction" line in debug.log would have pointed straight at the cause.

One follow-up that may be relevant to the bounded-step approach: on the second node, about 2 GiB stayed allocated for over two hours after the compaction had finished writing, and was then released all at once (11:33 to 13:48 in the table above). On the failing node, none of the 327 input tables were removed while 191 output tables were written. Is that expected? If tables can remain on disk after their compaction step completes, bounding each step might not by itself guarantee they're reclaimed before the next one starts.

Happy to test a patch, or to run the compaction on a copy of the chainstate if that would help.