Flush dbcache early if system is under memory pressure

luke-jr commented at 6:29 pm on September 4, 2020: member

No point forcing memory to get pushed out to swap just to cache db changes when we can write the db changes out instead

~~Chasing Concept ACKs and opinions on approach taken~~ (no longer draft)

One eventual goal might be to (on systems where this is supported) allow dbcache to be unlimited.

DrahtBot added the label Build system on Sep 4, 2020

DrahtBot added the label Utils/log/libs on Sep 4, 2020

DrahtBot added the label Validation on Sep 4, 2020

laanwj added the label UTXO Db and Indexes on Sep 4, 2020

laanwj removed the label Build system on Sep 4, 2020

laanwj removed the label Utils/log/libs on Sep 4, 2020

laanwj removed the label Validation on Sep 4, 2020

laanwj added the label Linux/Unix on Sep 4, 2020

laanwj added the label Windows on Sep 4, 2020

in src/validation.cpp:2098 in 1f62bebb27 outdated

2318+        bool fCacheCritical = false;
2319+        if (mode == FlushStateMode::IF_NEEDED) {
2320+            if (cache_state >= CoinsCacheSizeState::CRITICAL) {
2321+                // The cache is over the limit, we have to write now.
2322+                fCacheCritical = true;
2323+            } else if (util::SystemNeedsMemoryReleased()) {

laanwj commented at 6:48 am on September 8, 2020:

Something I’m slightly worried about is how this is pull, not push. How often does this polling happen? Will it result in a lot of extra syscalls?

laanwj commented at 6:51 am on September 8, 2020:

Please log a message when this happens (possibly with a only-once check against flooding).

luke-jr commented at 2:08 pm on October 24, 2020:

Should be at most once per block.

laanwj commented at 6:48 am on September 8, 2020: member

Concept ACK I think this is a neat idea.

You do need a way to disable this during the tests though, to assure deterministic behaviour that doesn’t depend on system circumstances (see Travis).

jb55 commented at 3:25 pm on September 10, 2020: contributor

Concept ACK

luke-jr renamed this:
~~[WIP] Flush dbcache early if system is under memory pressure~~
Flush dbcache early if system is under memory pressure
on Oct 24, 2020

luke-jr marked this as ready for review on Oct 24, 2020

luke-jr commented at 2:45 pm on October 24, 2020: member

Added logging, a config option, and fixed test (and took out of Draft)

DrahtBot commented at 8:37 pm on October 24, 2020: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#23732 (refactor: Remove gArgs from bdb.h and sqlite.h by kiminuo)
#22564 (refactor: Move mutable globals cleared in ::UnloadBlockIndex to BlockManager by dongcarl)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

luke-jr force-pushed on Nov 19, 2020

luke-jr force-pushed on Dec 3, 2020

DrahtBot added the label Needs rebase on Apr 13, 2021

Flush dbcache early if system is under memory pressure

No point forcing memory to get pushed out to swap just to cache db changes when we can write the db changes out instead

e029ae11ea

util: Log reasoning when returning true from SystemNeedsMemoryReleased bb29aa9f06

Make lowmem threshold configurable 141a05afe4

Disable lowmem flushing in test that needs determinism b9da34cec3

luke-jr force-pushed on Aug 29, 2021

DrahtBot removed the label Needs rebase on Aug 29, 2021

DrahtBot commented at 7:26 am on March 24, 2022: contributor

🐙 This pull request conflicts with the target branch and needs rebase.

Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a “draft”.

DrahtBot added the label Needs rebase on Mar 24, 2022

achow101 commented at 6:15 pm on October 12, 2022: member

Closing this as it has not had any activity in a while. If you are interested in continuing work on this, please leave a comment so that it can be reopened.

achow101 closed this on Oct 12, 2022

luke-jr commented at 11:58 pm on November 12, 2022: member

This continues to be updated and maintained for Knots.

Feel free to reopen it if anyone shows interest in reviewing for Core. (Ping me if I need to push an older commit temporarily to reopen)

bitcoin locked this on Nov 12, 2023

Flush dbcache early if system is under memory pressure #19873

Conflicts