validation: do not wipe utxo cache for stats/scans/snapshots #30610

pull sipa wants to merge 1 commits into bitcoin:master from sipa:202408_force_sync changing 8 files +26 −24
  1. sipa commented at 5:49 pm on August 8, 2024: member

    Since #28280, the cost of a non-wiping sync of the UTXO cache is only proportional to the number of dirty entries, rather than proportional to the size of the entire cache. Because of that, there is no reason to perform a wiping flush in case the contents of the cache is still useful.

    Split the FlushStateMode::ALWAYS mode into a FORCE_SYNC (non-wiping) and a FORCE_FLUSH (wiping), and then use the former in scantxoutset, gettxoutsetinfo, and in the currently-unused CreateUTXOSnapshot.

  2. DrahtBot commented at 5:49 pm on August 8, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/30610.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept ACK andrewtoth, mzumsande

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31382 (kernel: Flush in ChainstateManager destructor by TheCharlatan)
    • #30611 (validation: write chainstate to disk every hour by andrewtoth)
    • #29641 (scripted-diff: Use LogInfo over LogPrintf [WIP, NOMERGE, DRAFT] by maflcko)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Validation on Aug 8, 2024
  4. in src/validation.cpp:6041 in 2a9c93871c outdated
    6022@@ -6023,7 +6023,7 @@ util::Result<void> ChainstateManager::PopulateAndValidateSnapshot(
    6023         // returns in `ActivateSnapshot()`, when `MaybeRebalanceCaches()` is
    6024         // called, since we've added a snapshot chainstate and therefore will
    6025         // have to downsize the IBD chainstate, which will result in a call to
    6026-        // `FlushStateToDisk(ALWAYS)`.
    6027+        // `FlushStateToDisk(FORCE_FLUSH)`.
    


    sipa commented at 5:50 pm on August 8, 2024:
    It is unclear to me what the desired behavior is here, so I opted not to change it, but opinions welcome.
  5. andrewtoth commented at 6:36 pm on August 8, 2024: contributor

    Concept ACK

    Indeed, the FlushStateMode::ALWAYS variant is now confusing. It could mean we want to always write the chainstate to disk, or we want to always erase the dbcache. Splitting it up into the two cases makes sense.

  6. DrahtBot added the label CI failed on Aug 8, 2024
  7. DrahtBot commented at 7:12 pm on August 8, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/28531257299

    Make sure to run all tests locally, according to the documentation.

    The failure may happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  8. l0rinc commented at 1:04 pm on August 9, 2024: contributor

    I understand this is still WIP, but checked quickly how much the current impl would speed up the IBD (first 500k blocks with prune, between https://github.com/bitcoin/bitcoin/commit/27a770b34b8f1dbb84760f442edb3e23a0c2420b and https://github.com/bitcoin/bitcoin/commit/2a9c93871cea8c3b1d927a2559ef69ea76c9faf9).

    0hyperfine \
    1--runs 3 \
    2--parameter-list COMMIT 27a770,2a9c93 \
    3--prepare 'git checkout {COMMIT} && git clean -fxd && git reset --hard && ./autogen.sh && ./configure && make -j8 && rm -rf /mnt/sda1/BitcoinData/*' \
    4'COMMIT={COMMIT} ./src/bitcoind -datadir=/mnt/sda1/BitcoinData -dbcache=16384 -prune=550 -stopatheight=500000 -printtoconsole=0'
    
    0Benchmark 1: COMMIT=27a770 ./src/bitcoind -datadir=/mnt/sda1/BitcoinData -dbcache=16384 -prune=550 -stopatheight=500000 -printtoconsole=0
    1  Time (mean ± σ):     7999.675 s ± 63.767 s    [User: 9070.085 s, System: 750.952 s]
    2  Range (min  max):   7926.519 s  8043.491 s    3 runs
    3
    4Benchmark 2: COMMIT=2a9c93 ./src/bitcoind -datadir=/mnt/sda1/BitcoinData -dbcache=16384 -prune=550 -stopatheight=500000 -printtoconsole=0
    5  Time (mean ± σ):     9210.362 s ± 539.519 s    [User: 8944.738 s, System: 728.552 s]
    6  Range (min  max):   8713.643 s  9784.349 s    3 runs
    
    0Summary
    1 'COMMIT=27a770 ./src/bitcoind -datadir=/mnt/sda1/BitcoinData -dbcache=16384 -prune=550 -stopatheight=500000 -printtoconsole=0' ran
    2   1.15 ± 0.07 times faster than 'COMMIT=2a9c93 ./src/bitcoind -datadir=/mnt/sda1/BitcoinData -dbcache=16384 -prune=550 -stopatheight=500000 -printtoconsole=0'
    

    i.e. if the new impl seems consistently 15% slower than HEAD^1.

  9. sipa force-pushed on Aug 9, 2024
  10. sipa commented at 1:23 pm on August 9, 2024: member

    @paplorinc Huh, this PR shouldn’t affect IBD speed at all.

    EDIT: is this an IBD test from random peers? That will have way too much variance to conclude anything from 3 runs. I’d suggest a reindex test to avoid influence from randomly-selected peers on the internet.

  11. sipa force-pushed on Aug 9, 2024
  12. andrewtoth commented at 2:30 pm on August 9, 2024: contributor
    @paplorinc I don’t see anything in this PR that would affect reindexing. This PR seems like a refactor only. Oh, actually it will affectscantxoutset and gettxoutsetinfo rpcs.
  13. DrahtBot removed the label CI failed on Aug 9, 2024
  14. sipa force-pushed on Aug 9, 2024
  15. l0rinc commented at 8:55 pm on August 12, 2024: contributor

    Because of that, there is no reason to perform a wiping flush in case the contents of the cache is still useful.

    I assumed (without diving into the code) that this meant an optimization.

    0  Time (mean ± σ):     15396.778 s ± 31.337 s    [User: 81183.230 s, System: 364.223 s]
    1  Range (min … max):   15363.334 s … 15427.586 s    5 runs
    2
    3  Time (mean ± σ):     15410.168 s ± 20.832 s    [User: 81268.923 s, System: 363.370 s]
    4  Range (min … max):   15381.363 s … 15429.772 s    5 runs
    5
    6Summary
    7  'COMMIT=f40365460247e3ce14dd59ca441fc243e679f228 ./src/bitcoind -datadir=/mnt/sda1/BitcoinData -dbcache=16384 -reindex -stopatheight=500000 -printtoconsole=0' ran
    8    1.00 ± 0.00 times faster than 'COMMIT=27a770b34b8f1dbb84760f442edb3e23a0c2420b ./src/bitcoind -datadir=/mnt/sda1/BitcoinData -dbcache=16384 -reindex -stopatheight=500000
    9-printtoconsole=0'
    
  16. sipa commented at 8:57 pm on August 12, 2024: member
    @paplorinc Yes, it is an optimization, but only affects the scantxoutset and gettxoutsetinfo RPCs. Normal synchronization should be unaffected.
  17. sipa force-pushed on Aug 15, 2024
  18. sipa force-pushed on Aug 16, 2024
  19. sipa force-pushed on Aug 16, 2024
  20. validation: do not wipe utxo cache for stats/scans/snapshots 1ede4803f1
  21. sipa force-pushed on Aug 16, 2024
  22. mzumsande commented at 4:29 pm on September 3, 2024: contributor

    Concept ACK

    The test interface_usdt_utxocache.py fails because generate() in wallet.py calls scantxoutset, which doesn’t wipe the cache anymore. As a result, the number of utxos flushed to disk in later flushes changes.

    Re: “currently-unused CreateUTXOSnapshot” in the PR description - it’s used to create snapshots (dumptxoutset).

  23. achow101 commented at 3:47 pm on October 15, 2024: member
  24. DrahtBot added the label CI failed on Oct 23, 2024
  25. DrahtBot removed the label CI failed on Oct 25, 2024
  26. l0rinc commented at 4:17 pm on November 11, 2024: contributor

    The test interface_usdt_utxocache.py fails

    I haven’t reviewed the code in detail yet (since the build is still failing).

     0
     1> apt install systemtap-sdt-dev
     2> cmake -B build -DCMAKE_BUILD_TYPE=Release -DWITH_USDT=ON && cmake --build build -j$(nproc)
     3> build/test/functional/interface_usdt_utxocache.py
     4
     52024-11-11T15:47:49.211000Z TestFramework (INFO): handle_utxocache_add(): UTXOCacheChange(outpoint=d09ca39938b434b19a3338ef6b410071d85d4450dd7dc1f7db1bf7d95a53709a:0, height=201, value=4999968800sat, is_coinbase=False)
     62024-11-11T15:47:49.211000Z TestFramework (INFO): handle_utxocache_spent(): UTXOCacheChange(outpoint=fab2396e3f726e1801b9e7303cf294783a3fd7e19ddc8f2dfa55e25b01575b3e:0, height=77, value=5000000000sat, is_coinbase=True)
     72024-11-11T15:47:49.211000Z TestFramework (INFO): check that we successfully traced 2 adds and 1 spent
     82024-11-11T15:47:49.229000Z TestFramework (INFO): test the utxocache:flush tracepoint API
     92024-11-11T15:47:49.229000Z TestFramework (INFO): hook into the utxocache:flush tracepoint
    102024-11-11T15:47:50.288000Z TestFramework (INFO): stop the node to flush the UTXO cache
    112024-11-11T15:47:50.390000Z TestFramework (INFO): handle_utxocache_flush(): UTXOCacheFlush(duration=11670, mode=FORCE_FLUSH, size=3, memory=262464, for_prune=False)
    12Exception ignored on calling ctypes callback function: <function PerfEventArray._open_perf_buffer.<locals>.raw_cb_ at 0x76705f695c60>
    13Traceback (most recent call last):
    14  File "/usr/lib/python3/dist-packages/bcc/table.py", line 991, in raw_cb_
    15    callback(cpu, data, size)
    16  File "/mnt/my_storage/bitcoin/build/test/functional/interface_usdt_utxocache.py", line 351, in handle_utxocache_flush
    17    expected_flushes.remove({
    18ValueError: list.remove(x): x not in list
    192024-11-11T15:47:50.390000Z TestFramework (INFO): handle_utxocache_flush(): UTXOCacheFlush(duration=8205, mode=FORCE_FLUSH, size=0, memory=262240, for_prune=False)
    202024-11-11T15:47:50.408000Z TestFramework (INFO): check that we don't expect additional flushes
    212024-11-11T15:47:50.408000Z TestFramework (ERROR): Assertion failed
    22Traceback (most recent call last):
    23  File "/mnt/my_storage/bitcoin/test/functional/test_framework/test_framework.py", line 132, in main
    24    self.run_test()
    25  File "/mnt/my_storage/bitcoin/build/test/functional/interface_usdt_utxocache.py", line 151, in run_test
    26    self.test_flush()
    27  File "/mnt/my_storage/bitcoin/build/test/functional/interface_usdt_utxocache.py", line 380, in test_flush
    28    assert_equal(0, len(expected_flushes))
    29  File "/mnt/my_storage/bitcoin/test/functional/test_framework/util.py", line 77, in assert_equal
    30    raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
    31AssertionError: not(0 == 1)
    322024-11-11T15:47:50.460000Z TestFramework (INFO): Stopping nodes
    

    it is an optimization, but only affects the scantxoutset and gettxoutsetinfo RPCs

    0hyperfine \
    1--show-output \
    2--warmup 1 --runs 10 \
    3--export-json /mnt/gettxoutsetinfo-30610.json \
    4--parameter-list COMMIT 900b17239fb25750fd30b4af6e6c38096374b71f,226b6c7d6e69c1f032079c8bc5af5710d8e54efb \
    5--prepare 'killall -q bitcoind; git checkout {COMMIT} && git clean -fxd && git reset --hard && cmake -B build && cmake --build build -j$(nproc) && ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -daemon -connect=0 -rpcuser=benchmark -rpcpassword=benchmark && sleep 10' \
    6--cleanup './build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark stop && sleep 10 && killall -9 bitcoind || true' \
    7'COMMIT={COMMIT} ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData gettxoutsetinfo >/dev/null'
    
    0"command": "COMMIT=900b17239fb25750fd30b4af6e6c38096374b71f ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData gettxoutsetinfo >/dev/null",
    1"times": [ 194.38279658410002, 259.16496413609997, 209.65035661110002, 207.9064897821, 245.2373909041, 200.04822835410002, 227.2078635091, 192.6374527831, 160.2041179331, 141.44879581310002 ],
    2
    3"command": "COMMIT=226b6c7d6e69c1f032079c8bc5af5710d8e54efb ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData gettxoutsetinfo >/dev/null",
    4"times": [ 138.6323105461, 138.80884732910002, 137.4915245871, 137.2168089111, 139.4260206461, 144.2785081071, 136.9866460371, 137.38376074410002, 138.23486413010002, 136.55915986710002 ],
    
    0Summary
    1  'COMMIT=226b6c7d6e69c1f032079c8bc5af5710d8e54efb ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData gettxoutsetinfo >/dev/null' ran
    2    1.47 ± 0.26 times faster than 'COMMIT=900b17239fb25750fd30b4af6e6c38096374b71f ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData gettxoutsetinfo >/dev/null'
    

    0hyperfine \
    1--show-output \
    2--warmup 1 --runs 10 \
    3--export-json /mnt/scantxoutset-30610.json \
    4--parameter-list COMMIT 900b17239fb25750fd30b4af6e6c38096374b71f,226b6c7d6e69c1f032079c8bc5af5710d8e54efb \
    5--prepare 'killall -q bitcoind; git checkout {COMMIT} && git clean -fxd && git reset --hard && cmake -B build && cmake --build build -j$(nproc) && ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -daemon -connect=0 -rpcuser=benchmark -rpcpassword=benchmark && sleep 10' \
    6--cleanup './build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark stop && sleep 10 && killall -9 bitcoind || true' \
    7"COMMIT={COMMIT} ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData scantxoutset start '[\"addr(32ixEdVJWo3kmvJGMTZq5jAQVZZeuwnqzo)\"]' >/dev/null"
    
    0"command": "COMMIT=900b17239fb25750fd30b4af6e6c38096374b71f ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData scantxoutset start '[\"addr(32ixEdVJWo3kmvJGMTZq5jAQVZZeuwnqzo)\"]' >/dev/null",
    1"times": [ 62.82799487242, 60.64005598342, 60.99546379442, 60.31839828842, 64.55913160942, 63.55735657542, 62.07198356042, 62.17213781942, 63.035974236419996, 61.785472937419996 ],
    2
    3"command": "COMMIT=226b6c7d6e69c1f032079c8bc5af5710d8e54efb ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData scantxoutset start '[\"addr(32ixEdVJWo3kmvJGMTZq5jAQVZZeuwnqzo)\"]' >/dev/null",
    4"times": [ 61.64325996342, 61.09686494542, 61.077064575419996, 61.28694446342, 60.94913395142, 61.24807362942, 61.204911535419996, 61.23621690842, 61.174668652419996, 61.04969869342 ],
    
    0Summary
    1  'COMMIT=226b6c7d6e69c1f032079c8bc5af5710d8e54efb ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData scantxoutset start '["addr(32ixEdVJWo3kmvJGMTZq5jAQVZZeuwnqzo)"]' >/dev/null' ran
    2    1.02 ± 0.02 times faster than 'COMMIT=900b17239fb25750fd30b4af6e6c38096374b71f ./build/src/bitcoin-cli -rpcuser=benchmark -rpcpassword=benchmark -datadir=/mnt/my_storage/BitcoinData scantxoutset start '["addr(32ixEdVJWo3kmvJGMTZq5jAQVZZeuwnqzo)"]' >/dev/null'
    
  27. andrewtoth commented at 4:25 pm on November 11, 2024: contributor
    The tests need to be fixed per #30610#pullrequestreview-2277942711. @l0rinc what this PR accomplishes is not wiping the dbcache if calling those RPCs. It should not affect performance of the RPCs, but performance of connecting new blocks after calling the RPCs.
  28. in src/validation.cpp:2872 in 1ede4803f1
    2868@@ -2869,7 +2869,7 @@ bool Chainstate::FlushStateToDisk(
    2869         // It's been very long since we flushed the cache. Do this infrequently, to optimize cache usage.
    2870         bool fPeriodicFlush = mode == FlushStateMode::PERIODIC && nNow > m_last_flush + DATABASE_FLUSH_INTERVAL;
    2871         // Combine all conditions that result in a full cache flush.
    2872-        fDoFullFlush = (mode == FlushStateMode::ALWAYS) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune;
    2873+        fDoFullFlush = (mode == FlushStateMode::FORCE_FLUSH) || (mode == FlushStateMode::FORCE_SYNC) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune;
    


    l0rinc commented at 10:55 am on November 14, 2024:

    It seems to me this complex condition contains a repeated hidden sub-expression that’s needed later as empty_cache. Could we split this up to reduce duplication and simplify the expressions?

    0bool should_empty_cache = (mode == FlushStateMode::FORCE_FLUSH) || fCacheLarge || fCacheCritical;
    1bool fDoFullFlush = should_empty_cache || (mode == FlushStateMode::FORCE_SYNC) || fPeriodicFlush || fFlushForPrune;
    2...
    3// Flush the chainstate (which may refer to block index entries).
    4if (should_empty_cache ? !CoinsTip().Flush() : !CoinsTip().Sync()) {
    5    return FatalError(m_chainman.GetNotifications(), state, _("Failed to write to coin database."));
    6}
    
     0diff --git a/src/validation.cpp b/src/validation.cpp
     1--- a/src/validation.cpp	(revision 1ede4803f1f5dffa55644d908062b52bf71394e7)
     2+++ b/src/validation.cpp	(date 1731581531144)
     3@@ -2807,7 +2807,6 @@
     4     try {
     5     {
     6         bool fFlushForPrune = false;
     7-        bool fDoFullFlush = false;
     8 
     9         CoinsCacheSizeState cache_state = GetCoinsCacheSizeState();
    10         LOCK(m_blockman.cs_LastBlockFile);
    11@@ -2869,7 +2868,8 @@
    12         // It's been very long since we flushed the cache. Do this infrequently, to optimize cache usage.
    13         bool fPeriodicFlush = mode == FlushStateMode::PERIODIC && nNow > m_last_flush + DATABASE_FLUSH_INTERVAL;
    14         // Combine all conditions that result in a full cache flush.
    15-        fDoFullFlush = (mode == FlushStateMode::FORCE_FLUSH) || (mode == FlushStateMode::FORCE_SYNC) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune;
    16+        bool should_empty_cache = (mode == FlushStateMode::FORCE_FLUSH) || fCacheLarge || fCacheCritical;
    17+        bool fDoFullFlush = should_empty_cache || (mode == FlushStateMode::FORCE_SYNC) || fPeriodicFlush || fFlushForPrune;
    18         // Write blocks and block index to disk.
    19         if (fDoFullFlush || fPeriodicWrite) {
    20             // Ensure we can write block index
    21@@ -2917,8 +2917,7 @@
    22                 return FatalError(m_chainman.GetNotifications(), state, _("Disk space is too low!"));
    23             }
    24             // Flush the chainstate (which may refer to block index entries).
    25-            const auto empty_cache{(mode == FlushStateMode::FORCE_FLUSH) || fCacheLarge || fCacheCritical};
    26-            if (empty_cache ? !CoinsTip().Flush() : !CoinsTip().Sync()) {
    27+            if (should_empty_cache ? !CoinsTip().Flush() : !CoinsTip().Sync()) {
    28                 return FatalError(m_chainman.GetNotifications(), state, _("Failed to write to coin database."));
    29             }
    30             m_last_flush = nNow;
    
  29. in src/validation.h:451 in 1ede4803f1
    447@@ -448,7 +448,8 @@ enum class FlushStateMode {
    448     NONE,
    449     IF_NEEDED,
    450     PERIODIC,
    451-    ALWAYS
    452+    FORCE_SYNC,
    


    l0rinc commented at 10:58 am on November 14, 2024:

    We should likely update the related script and docs as well:

     0diff --git a/doc/tracing.md b/doc/tracing.md
     1--- a/doc/tracing.md	(revision 1ede4803f1f5dffa55644d908062b52bf71394e7)
     2+++ b/doc/tracing.md	(date 1731581879407)
     3@@ -126,8 +126,8 @@
     4
     5 Arguments passed:
     6 1. Time it took to flush the cache microseconds as `int64`
     7-2. Flush state mode as `uint32`. It's an enumerator class with values `0`
     8-   (`NONE`), `1` (`IF_NEEDED`), `2` (`PERIODIC`), `3` (`ALWAYS`)
     9+2. Flush state mode as `uint32`. It's an enumerator class with values
    10+   `0` (`NONE`), `1` (`IF_NEEDED`), `2` (`PERIODIC`), `3` (`FORCE_SYNC`), `4` (`FORCE_FLUSH`)
    11 3. Cache size (number of coins) before the flush as `uint64`
    12 4. Cache memory usage in bytes as `uint64`
    13 5. If pruning caused the flush as `bool`
    14
    15diff --git a/contrib/tracing/README.md b/contrib/tracing/README.md
    16--- a/contrib/tracing/README.md	(revision 1ede4803f1f5dffa55644d908062b52bf71394e7)
    17+++ b/contrib/tracing/README.md	(date 1731581127426)
    18@@ -247,10 +246,10 @@
    19
    20 Logging utxocache flushes. Ctrl-C to end...
    21-Duration (µs)   Mode       Coins Count     Memory Usage    Prune
    22-730451          IF_NEEDED  22990           3323.54 kB      True
    23-637657          ALWAYS     122320          17124.80 kB     False
    24-81349           ALWAYS     0               1383.49 kB      False
    25+Duration (µs)   Mode        Coins Count     Memory Usage    Prune
    26+730451          IF_NEEDED   22990           3323.54 kB      True
    27+637657          FORCE_FLUSH 122320          17124.80 kB     False
    28+81349           FORCE_FLUSH 0               1383.49 kB      False
    29
    30
    31diff --git a/contrib/tracing/log_utxocache_flush.py b/contrib/tracing/log_utxocache_flush.py
    32--- a/contrib/tracing/log_utxocache_flush.py	(revision 1ede4803f1f5dffa55644d908062b52bf71394e7)
    33+++ b/contrib/tracing/log_utxocache_flush.py	(date 1731582100621)
    34@@ -45,7 +45,8 @@
    35     'NONE',
    36     'IF_NEEDED',
    37     'PERIODIC',
    38-    'ALWAYS'
    39+    'FORCE_SYNC',
    40+    'FORCE_FLUSH',
    41 ]
    
  30. in src/validation.cpp:2944 in 1ede4803f1
    2940@@ -2941,10 +2941,10 @@ bool Chainstate::FlushStateToDisk(
    2941     return true;
    2942 }
    2943 
    2944-void Chainstate::ForceFlushStateToDisk()
    2945+void Chainstate::ForceFlushStateToDisk(bool wipe_cache)
    


    l0rinc commented at 11:11 am on November 14, 2024:
    Given that ForceFlushStateToDisk(false) does a FORCE_SYNC now, does the method name still reflect its role?
  31. in src/validation.h:681 in 1ede4803f1
    678@@ -678,7 +679,7 @@ class Chainstate
    679         int nManualPruneHeight = 0);
    680 
    681     //! Unconditionally flush all changes to disk.
    


    l0rinc commented at 11:13 am on November 14, 2024:
    Same question here, is Unconditionally still the case here, given that ForceFlushStateToDisk(false) does a sync instead?
  32. in test/functional/interface_usdt_utxocache.py:369 in 1ede4803f1
    368@@ -368,8 +369,8 @@ def handle_utxocache_flush(_, data, __):
    369         # A node shutdown causes two flushes. One that flushes UTXOS_IN_CACHE
    


    l0rinc commented at 12:37 pm on November 14, 2024:

    Updating to UTXOS_IN_CACHE = 3 and expected_flushes.append({"mode": "NONE", "for_prune": True, "size": BLOCKS_TO_MINE}) makes this test pass again.

     0diff --git a/test/functional/interface_usdt_utxocache.py b/test/functional/interface_usdt_utxocache.py
     1--- a/test/functional/interface_usdt_utxocache.py	(revision eb1257260b2f4ede605241f1fb514452d63305c8)
     2+++ b/test/functional/interface_usdt_utxocache.py	(date 1731587669301)
     3@@ -365,7 +365,7 @@
     4         bpf["utxocache_flush"].open_perf_buffer(handle_utxocache_flush)
     5 
     6         self.log.info("stop the node to flush the UTXO cache")
     7-        UTXOS_IN_CACHE = 2 # might need to be changed if the earlier tests are modified
     8+        UTXOS_IN_CACHE = 3 # might need to be changed if the earlier tests are modified
     9         # A node shutdown causes two flushes. One that flushes UTXOS_IN_CACHE
    10         # UTXOs and one that flushes 0 UTXOs. Normally the 0-UTXO-flush is the
    11         # second flush, however it can happen that the order changes.
    12@@ -395,7 +395,7 @@
    13         bpf["utxocache_flush"].open_perf_buffer(handle_utxocache_flush)
    14 
    15         self.log.info(f"prune blockchain to trigger a flush for pruning")
    16-        expected_flushes.append({"mode": "NONE", "for_prune": True, "size": 0})
    17+        expected_flushes.append({"mode": "NONE", "for_prune": True, "size": BLOCKS_TO_MINE})
    18         self.nodes[0].pruneblockchain(315)
    19 
    20         bpf.perf_buffer_poll(timeout=500)
    
  33. l0rinc changes_requested
  34. l0rinc commented at 2:39 pm on November 14, 2024: contributor

    It should not affect performance of the RPCs

    It seems I have misunderstood multiple times what to benchmark here (IBD, reindex, RPC), so if there’s anything that actually needs benchmarking, please describe it in more detail.

  35. andrewtoth commented at 4:02 pm on November 14, 2024: contributor
    @l0rinc a way to test that this PR is working properly is to first run master and sync a few blocks. In the UpdateTip logs you will see a cache= value that increases. Running either of these RPCs should make the next UpdateTip reset a few MB at most. When doing the same test on this branch, the next UpdateTip value should be the same or higher than the previous.

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me