coins: warn of low disk space before chainstate compaction #36586

pull l0rinc wants to merge 3 commits into bitcoin:master from l0rinc:l0rinc/chainstate-compaction-headroom changing 6 files +28 −14
  1. l0rinc commented at 9:34 PM on October 20, 2026: contributor

    Problem: Full chainstate compaction can temporarily require nearly another copy of the database before old files are deleted.

    Fix: Log automatic full compaction start and finish at info level. Before starting, check the free space on the chainstate filesystem, and if it is below m_assumed_chain_state_size, log a warning and skip this compaction. Skipping avoids the temporary extra disk use but leaves obsolete data taking up space.

    Fixes #36585

  2. DrahtBot added the label UTXO Db and Indexes on Oct 20, 2026
  3. DrahtBot commented at 9:34 PM on October 20, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    External sites

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    ACK optout21, stickies-v, danielabrozzoni
    Concept ACK sedited, stringintech

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #36598 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36598.svg"></sub> ([Demo] validation: remove AssumeUTXO by l0rinc)
    • #34132 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/34132.svg"></sub> (coins,dbwrapper: make point-read and txindex lookup failures fatal by l0rinc)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  4. andrewtoth commented at 11:01 PM on October 20, 2026: contributor

    Compaction continues after the warning because postponing cleanup would allow obsolete data to accumulate.

    Would it make more sense to just skip compaction if we don't think there's enough free disk space to do it? This would have the tradeoff that some extra data would remain, but it would let users like the issue reporter continue running their node for likely several years without any problems.

  5. l0rinc force-pushed on Oct 21, 2026
  6. l0rinc commented at 6:47 AM on October 21, 2026: contributor

    Thanks @andrewtoth, updated as suggested.

  7. sedited commented at 6:50 AM on October 21, 2026: contributor

    Concept ACK

  8. stickies-v commented at 3:26 PM on October 21, 2026: contributor

    Concept ACK

  9. in src/txdb.cpp:222 in a3de791941
     222 | -            LogInfo("Starting chainstate compaction of %s", fs::PathToString(m_db_params.path));
     223 | -            m_db->CompactFull();
     224 | -            LogInfo("Finished chainstate compaction of %s", fs::PathToString(m_db_params.path));
     225 | +            if (m_db_params.memory_only || fs::space(m_db_params.path).available > min_free_space) {
     226 | +                LogInfo("Starting chainstate compaction of %s", fs::PathToString(m_db_params.path));
     227 | +                m_db->CompactFull();
    


    stickies-v commented at 5:03 PM on October 21, 2026:

    My understanding of the worst case here is that this happens when all the data is on a single level L, with a very sparse (<32MiB) level L-1 above it. Because L-1 is so sparse, all L-1 files (with keys across the entire L level) get compacted in one go, causing the entire dense level L (~the entire database) to be temporarily copied, correct?

    It seems like we're already being more adventurous with our LevelDB usage than we perhaps should be (the CompactRange docstring states "This operation should typically only be invoked by users who understand the underlying implementation."), but perhaps an alternative approach could be to call CompactRange on slices, instead of on the entire (nullptr, nullptr) range in one go?


    l0rinc commented at 10:39 AM on October 22, 2026:

    Thanks for the hint, we should investigate that - I tried compacting smaller key ranges locally, which reduced peak disk usage in some cases but could still produce large spikes, defeating the purpose. Partial compaction could be investigated separately, but it seems fairly complicated, and I don't think the benefit is worth adding that complexity to this PR.


    stickies-v commented at 9:25 AM on October 23, 2026:

    I don't think the benefit is worth adding that complexity to this PR.

    Yeah, I think I agree.

  10. l0rinc force-pushed on Oct 22, 2026
  11. l0rinc commented at 10:39 AM on October 22, 2026: contributor

    I've separated the free-space check from the database operation and unified the compaction logging, as @optout21 suggested in the original PR.

  12. in src/validation.cpp:2839 in 694c535312
    2832 | @@ -2833,7 +2833,13 @@ bool Chainstate::FlushStateToDisk(
    2833 |  
    2834 |          if (!m_chainman.m_interrupt && ShouldCompactChainstate(m_chainman.IsInitialBlockDownload())) {
    2835 |              try {
    2836 | -                CoinsDB().CompactFullAsync();
    2837 | +                const fs::path path{StoragePath()};
    2838 | +                const uint64_t min_free_space{m_chainman.GetParams().AssumedChainStateSize() * 1_GiB};
    2839 | +                if (!CoinsDB().IsMemoryOnly() && fs::space(path).available < min_free_space) {
    2840 | +                    LogWarning("Skipping chainstate compaction of %s: need %u GiB free", fs::PathToString(path), CeilDiv(min_free_space, 1_GiB));
    


    optout21 commented at 11:18 AM on October 22, 2026:

    694c535 coins: skip compaction if disk space is low:

    Additionally logging here the currently observed available space is marginally helpful (guideline: if a log is a result of a decision based on several values, log each).


    optout21 commented at 11:23 AM on October 22, 2026:

    694c535 coins: skip compaction if disk space is low:

    Could we add some more explanatory and call-to-action info here? Describing that the uncompacted DB takes up more disk space, and that the user should free up some disk space. Also, I suggest changing the wording "Skipping" to "Deferring", to suggest that it's not without consequences.


    l0rinc commented at 8:49 PM on October 23, 2026:

    Done, thanks


    l0rinc commented at 8:49 PM on October 23, 2026:

    Done

  13. optout21 commented at 11:30 AM on October 22, 2026: contributor

    Concept ACK Left minor comments on the wording of the warning.

  14. in src/validation.cpp:2838 in ed0f6de810
    2832 | @@ -2833,7 +2833,13 @@ bool Chainstate::FlushStateToDisk(
    2833 |  
    2834 |          if (!m_chainman.m_interrupt && ShouldCompactChainstate(m_chainman.IsInitialBlockDownload())) {
    2835 |              try {
    2836 | -                CoinsDB().CompactFullAsync();
    2837 | +                const fs::path path{StoragePath()};
    2838 | +                const uint64_t min_free_space{m_chainman.GetParams().AssumedChainStateSize() * 1_GiB};
    2839 | +                if (!CoinsDB().IsMemoryOnly() && fs::space(path).available < min_free_space) {
    


    stickies-v commented at 9:09 AM on October 23, 2026:

    nit: any reason we're not using CheckDiskSpace() here?


    stickies-v commented at 9:15 AM on October 23, 2026:

    For an in-memory database, the compaction memory usage problem is arguably even worse than the on-disk one? Since memory-only is quite ephemeral in nature, I think it's uncontroversial to just skip compaction entirely in that case?


    l0rinc commented at 8:49 PM on October 23, 2026:

    Guess the extra 50 MiB reserve is fine here (seemed imprecise to me) - changed.


    l0rinc commented at 8:49 PM on October 23, 2026:

    Sure, I was also wondering if someone would comment on bubbling this check up - thanks, done.


    optout21 commented at 4:56 AM on October 24, 2026:

    AssumedChainStateSize() is also rather imprecise here.

  15. in doc/release-notes-36586.md:4 in ed0f6de810
       0 | @@ -0,0 +1,6 @@
       1 | +Disk space
       2 | +----------
       3 | +
       4 | +- Automatic full chainstate compaction now logs its start and finish by default.
    


    stickies-v commented at 9:23 AM on October 23, 2026:

    nit: I found the "may need another database copy" a bit confusing, and think the rest of the phrasing could be a bit more clear as well. Proposed alternative, if you like:

    Periodic background chainstate compaction is now logged by default. It can temporarily use about as much extra disk space as the chainstate, so it is skipped with a warning if there is not enough free space, leaving obsolete data to take up unnecessary disk space.


    l0rinc commented at 8:49 PM on October 23, 2026:

    Did something similar, thanks.

  16. stickies-v approved
  17. stickies-v commented at 9:31 AM on October 23, 2026: contributor

    ACK ed0f6de8104d67454c0b9547fd19113d4e95abf2

  18. DrahtBot requested review from optout21 on Oct 23, 2026
  19. DrahtBot requested review from sedited on Oct 23, 2026
  20. logging: share full compaction logs
    Automatic chainstate compaction can use substantial disk space and take minutes, but its start and finish are hidden unless `-debug=coindb` is enabled.
    Log both at info level in `CDBWrapper::CompactFull()` so forced and automatic compaction share the same messages.
    
    Co-authored-by: optout <13562139+optout21@users.noreply.github.com>
    f57a6973da
  21. coins: skip compaction if disk space is low
    Full chainstate compaction can temporarily need about as much extra disk space as the existing database.
    Before scheduling it, check the chainstate filesystem using `AssumedChainStateSize()` and the standard free-space reserve, and report available space when deferring compaction.
    Deferring avoids the temporary extra disk use but leaves obsolete data taking up space until the operator frees enough space for cleanup.
    Skip periodic compaction for in-memory chainstates, which do not need to reclaim disk space.
    
    Co-authored-by: stickies-v <stickies-v@protonmail.com>
    cef76701a8
  22. doc: add chainstate compaction release note
    Co-authored-by: stickies-v <stickies-v@protonmail.com>
    2fcfa1dae7
  23. l0rinc force-pushed on Oct 23, 2026
  24. l0rinc commented at 8:51 PM on October 23, 2026: contributor

    Thanks for the reviews, updated to skip automatic in-memory compaction, use CheckDiskSpace() with 50 mb additional overhead, include available space and recovery guidance in the warning, and clarify the release note.

  25. optout21 commented at 5:03 AM on October 24, 2026: contributor

    crACK 2fcfa1dae75ccd11062a6e6073600e67eb20164c

  26. DrahtBot requested review from stickies-v on Oct 24, 2026
  27. stickies-v commented at 10:06 AM on October 26, 2026: contributor

    ACK 2fcfa1dae75ccd11062a6e6073600e67eb20164c

  28. BenWestgate referenced this in commit 2fcced24ee on Oct 29, 2026
  29. BenWestgate referenced this in commit 4dea4b14b8 on Oct 29, 2026
  30. in src/validation.cpp:2841 in 2fcfa1dae7
    2838 | +                const fs::path path{StoragePath()};
    2839 | +                const uint64_t min_free_space{m_chainman.GetParams().AssumedChainStateSize() * 1_GiB};
    2840 | +                if (CheckDiskSpace(path, min_free_space)) {
    2841 | +                    CoinsDB().CompactFullAsync();
    2842 | +                } else {
    2843 | +                    LogWarning("Deferring chainstate compaction of %s: %.2f GiB free, %u GiB needed. Free up some space so compaction can run.",
    


    andrewtoth commented at 7:46 PM on October 31, 2026:

    I'm not sure we should warn about this. Running on a system with < 2x the chainstate disk space should be supported. If that is the case there is nothing the user can do, and they shouldn't be getting warnings about it.


    l0rinc commented at 5:40 PM on November 1, 2026:

    Running on a system with < 2x the chainstate disk space should be supported

    We can't expect a node to run without compactions, so we have to let the user know that it's a temporary allowance and that they should do something about it: relocate to a bigger disk, delete some files, prune harder, etc.


    andrewtoth commented at 6:49 PM on November 1, 2026:

    We can't expect a node to run without compactions

    Why not? From my understanding the compactions are just a way to reduce disk usage in the chainstate db, but don't affect node operation otherwise.

    Right now the chainstate db is ~11 GB. With pruning we can run it on an instance with < 20 GB of disk space. In this scenario the user will get a warning for every attempted compaction. What is the benefit of having this user relocate to a bigger disk, just to save disk space? Is there a way to determine the total disk size and skip warning if it is < 2x chainstate size?


    l0rinc commented at 8:52 PM on November 2, 2026:

    It might be fine to keep running for a while without regular full compaction, which is why this is a warning rather than an abort. But the operator should be aware that the machine lacks the disk headroom for normal database maintenance.

    Allowing tombstones to accumulate can also make full database scans more expensive, since those records still need to be traversed and skipped.

    I don't think we should encourage sizing a node's disk around the chainstate staying at roughly 10 GB (hypothetically, it could even double in a month) - nodes need some headroom, hence the warning.


    stringintech commented at 1:59 PM on November 9, 2026:

    I was wondering whether, if we want the node operator to act on the warning, we could surface it in a more visible/persistent way as well. But it looks like we reserve persistent warnings for more critical/consensus-related cases? (While looking into how we currently emit the more persistent warnings, I also noticed that the first-startup disk space check doesn't account for the space needed by the chainstate, so I opened #36811 to get feedback. If that lands, maybe a follow-up could extend the check with the headroom needed for compaction?)

  31. in doc/release-notes-36586.md:6 in 2fcfa1dae7
       0 | @@ -0,0 +1,6 @@
       1 | +Disk space
       2 | +----------
       3 | +
       4 | +- Periodic background chainstate compaction is now logged by default.
       5 | +  Compaction can temporarily need about as much extra disk space as the chainstate itself.
       6 | +  If there is not enough free space, it is deferred with a warning and obsolete data stays on disk.
    


    andrewtoth commented at 7:48 PM on October 31, 2026:

    I don't think this change warrants a release note. If others disagree, at least the last part of this line should be struck."obsolete data" is not something a user reading the release notes will understand.


    l0rinc commented at 5:41 PM on November 1, 2026:

    Do they need to understand more than garbage collection needs a bit more space?

  32. andrewtoth changes_requested
  33. in src/validation.cpp:2834 in 2fcfa1dae7
    2830 | @@ -2831,9 +2831,16 @@ bool Chainstate::FlushStateToDisk(
    2831 |              m_chainman.m_options.signals->ChainStateFlushed(this->GetRole(), GetLocator(m_last_flushed_block));
    2832 |          }
    2833 |  
    2834 | -        if (!m_chainman.m_interrupt && ShouldCompactChainstate(m_chainman.IsInitialBlockDownload())) {
    2835 | +        if (!m_chainman.m_interrupt && !CoinsDB().IsMemoryOnly() && ShouldCompactChainstate(m_chainman.IsInitialBlockDownload())) {
    


    stringintech commented at 12:22 PM on November 9, 2026:

    nit: From commit cef76701a81fde56ad143b4ea559468a6b4dc626 description:

    Skip periodic compaction for in-memory chainstates, which do not need to reclaim disk space.

    Maybe here we can more accurately reflect the intention of skipping in-memory chainstates (stated in #36586 (review)). Because if it weren't for compaction's temporary extra space coming out of RAM (even scarcer than disk) and in-memory chainstates being mostly used for short-lived purposes, compaction would also have been helpful to reclaim memory space?

  34. stringintech commented at 2:08 PM on November 9, 2026: contributor

    Concept ACK.

  35. danielabrozzoni commented at 12:11 PM on November 10, 2026: member

    light tACK 2fcfa1dae75ccd11062a6e6073600e67eb20164c

    Very new to this part of the code, but this looks good to me. I tested on my own system by:

    1. Changing ShouldCompactChainstate to always returns true if not in IBD
    2. (Almost) filling my disk using fallocate :)
    3. Calling bitcoin-cli gettxoutsetinfo to force flush to disk

    And saw the log:

    2026-11-10T11:49:02Z [warning] Deferring chainstate compaction of /home/daniela/.bitcoin/chainstate: 8.38 GiB free, 14 GiB needed. Free up some space so compaction can run.
    
  36. DrahtBot requested review from stringintech on Nov 10, 2026

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-11-16 00:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me