[IBD] flush UTXOs in bigger batches #31645

pull l0rinc wants to merge 1 commits into bitcoin:master from l0rinc:l0rinc/utxo-dump-batching changing 1 files +1 −1
  1. l0rinc commented at 8:08 pm on January 12, 2025: contributor

    This change is part of [IBD] - Tracking PR for speeding up Initial Block Download

    Summary

    The final UTXO set is written to disk in batches to avoid a gigantic spike at flush time. There is already a -dbbatchsize config option to change this value, this PR adjusts the default only. By increasing the default batch size, we can reduce overhead from repeated compaction cycles, minimize constant overhead per batch, and achieve more sequential writes.

    Context

    The UTXO set has grown significantly since 2017, when the original 16 MiB batch size was chosen. Flushing it from memory to LevelDB often takes 20-30 minutes on commodity hardware after a successful IBD with large dbcache values. Bigger batch allows more LevelDB optimizations before flushing while minimally increasing memory footprint (the UTXO set is >1000x times bigger the batch size). This is especially true now that we’ve bumped the LevelDB max file size.

    Considerations

    As noted by @sipa, this will temporarily increase the required memory exactly when our memory needs are the greatest. This will be less problematic after validation: write chainstate to disk every hour, where we can lose at most the last hour of work. While 128 and 256 MiB batches were often faster, 64 MiB was chosen since it seems to achieve a reasonable speedup at a small memory cost.

    Measurements

    Experiments with different batch sizes (loaded via AssumeUTXO 840k and the new 880k, then measuring final flush time) on different operating system show that 64 MiB batches significantly reduce flush time without notably increasing memory usage (both smaller and bigger ones are usually slower).

    dbbatchsize flush_sum (ms)
    8 « 20 236993.73
    8 « 20 239557.79
    8 « 20 244149.25
    8 « 20 246116.93
    8 « 20 243496.98
    16 « 20 209673.01
    16 « 20 225029.97
    16 « 20 230826.61
    16 « 20 230312.84
    16 « 20 235912.83
    32 « 20 201898.77
    32 « 20 196676.18
    32 « 20 198958.81
    32 « 20 196230.08
    32 « 20 199105.84
    64 « 20 150691.51
    64 « 20 151072.18
    64 « 20 151465.16
    64 « 20 150403.59
    64 « 20 150342.34
    128 « 20 155917.81
    128 « 20 156121.83
    128 « 20 156514.6
    128 « 20 155616.36
    128 « 20 156398.24
    256 « 20 166843.39
    256 « 20 166226.37
    256 « 20 166351.75
    256 « 20 166197.15
    256 « 20 166755.22
    512 « 20 186020.24
    512 « 20 186689.18
    512 « 20 186895.21
    512 « 20 185427.1
    512 « 20 186105.48
    1 « 30 185488.98
    1 « 30 185963.51
    1 « 30 185754.25
    1 « 30 186993.17
    1 « 30 186145.73

    Checking the impact of a -reindex-chainstate with -stopatheight=878000 and -dbcache=30000 gives:

    On SSD:

    02025-01-12T07:31:05Z [warning] Flushing large (26 GiB) UTXO set to disk, it may take several minutes
12025-01-12T07:53:51Z Shutdown: done

    Flush time before: 22 minutes and 46 seconds

    02025-01-12T18:30:00Z [warning] Flushing large (26 GiB) UTXO set to disk, it may take several minutes
12025-01-12T18:44:43Z Shutdown: done

    Flush time after: 14 minutes and 43 seconds

    On HDD:

    02025-01-12T04:31:40Z [warning] Flushing large (26 GiB) UTXO set to disk, it may take several minutes
12025-01-12T05:02:39Z Shutdown: done

    Flush time before: 30 minutes and 59 seconds

    02025-01-12T20:22:24Z [warning] Flushing large (26 GiB) UTXO set to disk, it may take several minutes
12025-01-12T20:42:57Z Shutdown: done

    Flush time after: 20 minutes and 33 seconds

    Reproducer:

    You can either do a full IBD or a reindex(-chainstate) and check the final logs flush the blocks or load the UTXO set from the AssumeUTXO until 840k or 880k and use them for the measurements.

     0# Build Bitcoin Core
 1cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j$(nproc)
 2
 3# Set up a clean demo environment
 4mkdir -p demo && rm -rfd demo/chainstate demo/chainstate_snapshot demo/debug.log
 5
 6# Start bitcoind with minimal settings without mempool and internet connection
 7build/bin/bitcoind -datadir=demo -stopatheight=1
 8build/bin/bitcoind -datadir=demo -daemon -blocksonly=1 -connect=0 -dbcache=30000
 9
10# Load the AssumeUTXO snapshot, making sure the path is correct
11# Expected output includes `"coins_loaded": 184821030`
12build/bin/bitcoin-cli -datadir=demo -rpcclienttimeout=0 loadtxoutset ~/utxo-880000.dat
13
14# Stop the daemon and verify snapshot flushes in the logs
15build/bin/bitcoin-cli -datadir=demo stop
16grep "FlushSnapshotToDisk: completed" demo/debug.log
  2. l0rinc commented at 8:11 pm on January 12, 2025: contributor

    Visual representation of the AssumeUTXO 840k measurements (16MiB was the previous default, 64MiB is the proposed one):

     0set -e
 1
 2killall bitcoind 2>/dev/null || true
 3cd /mnt/my_storage/bitcoin
 4mkdir -p demo
 5
 6for i in 1 2; do
 7  for dbcache in 440 5000 30000; do
 8    for dbbatchsize in 4194304 8388608 16777216 33554432 67108864 134217728 268435456; do
 9      cmake -B build -DCMAKE_BUILD_TYPE=Release > /dev/null 2>&1
10      cmake --build build -j"$(nproc)" > /dev/null 2>&1
11
12      build/bin/bitcoin-cli -datadir=demo stop 2>/dev/null || true
13      killall bitcoind 2>/dev/null || true
14      sleep 10
15
16      rm -rfd demo/chainstate demo/chainstate_snapshot
17      build/bin/bitcoind -datadir=demo -stopatheight=1 -printtoconsole=0 && rm -f demo/debug.log
18
19      echo "Starting bitcoind with dbcache=$dbcache"
20      build/bin/bitcoind -datadir=demo -blocksonly=1 -connect=0 -dbcache="$dbcache" -dbbatchsize="$dbbatchsize" -daemon -printtoconsole=0
21      sleep 10
22
23      echo "Loading UTXO snapshot..."
24      build/bin/bitcoin-cli -datadir=demo -rpcclienttimeout=0 loadtxoutset /mnt/my_storage/utxo-880000.dat | grep -q '"coins_loaded": 184821030' || { echo "ERROR: Wrong number of coins loaded"; exit 1; }
25      build/bin/bitcoin-cli -datadir=demo stop
26      killall bitcoind 2>/dev/null || true
27      sleep 10
28
29      out_file="results_i_${i}_dbcache_${dbcache}_dbbatchsize_${dbbatchsize}.log"
30      echo "Collecting logs in ${out_file}"
31      grep "FlushSnapshotToDisk: completed" demo/debug.log | tee -a "${out_file}"
32      echo "---" >> "${out_file}"
33
34      echo "Done with i=${i}, dbcache=${dbcache}, dbbatchsize=${dbbatchsize}"
35      echo
36    done
37  done
38done
39
40echo "All runs complete. Logs are saved in results_dbcache*.log files."

     0import re
 1import sys
 2
 3
 4def parse_bitcoin_debug_log(file_path):
 5    results = []
 6
 7    flush_sum = 0.0
 8    flush_count = 0
 9
10    version_pattern = re.compile(r"Bitcoin Core version")
11    flush_pattern = re.compile(r'FlushSnapshotToDisk: completed \(([\d.]+)ms\)')
12
13    def finalize_current_block():
14        nonlocal flush_sum, flush_count
15        if flush_count > 0:
16            results.append((flush_sum, flush_count))
17        flush_sum = 0.0
18        flush_count = 0
19
20    try:
21        with open(file_path, 'r') as file:
22            for line in file:
23                if version_pattern.search(line):
24                    finalize_current_block()
25                    continue
26
27                match_flush = flush_pattern.search(line)
28                if match_flush:
29                    flush_ms = float(match_flush.group(1))
30                    flush_sum += flush_ms
31                    flush_count += 1
32    except Exception as e:
33        print(f"Error reading file: {e}")
34        sys.exit(1)
35
36    finalize_current_block()
37
38    return results
39
40
41if __name__ == "__main__":
42    if len(sys.argv) < 2:
43        print("Usage: python3 script.py <path_to_debug_log>")
44        sys.exit(1)
45
46    file_path = sys.argv[1]
47    parsed_results = parse_bitcoin_debug_log(file_path)
48
49    for total_flush_time, total_flush_calls in parsed_results:
50        print(f"{total_flush_time:.2f},{total_flush_calls}")

    Edit: Reran the benchmarks on a Hetzner Linux machine with the new AssumeUTXO 880k set, parsing the logged flush times and plotting the results. For different dbcache (440, 5000, 30000) and dbbatchsize (4-256MiB range, 16MiB (current) and 64MiB (proposed) are highlighted and trendline is added for clarity), each one twice for stability:

    Sorting the same measurements (and calculating the sums) might give us a better understanding of the trends:

  3. DrahtBot commented at 8:11 pm on January 12, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31645.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK ryanofsky, jonatack

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    No conflicts as of last run.

  4. laanwj added the label UTXO Db and Indexes on Jan 13, 2025
  5. sipa commented at 3:52 pm on January 13, 2025: member

    FWIW, the reason for the existence of the batch size behavior (as opposed to just writing everything at once) is that it causes a memory usage spike at flush time. If that spike exceeds the memory the process can allocate it causes a crash, at a particularly bad time (may require a replay to fix, which may be slower than just reprocessing the blocks).

    Given that changing this appears to improve performance it’s worth considering of course, but it is essentially a trade-off between speed and memory usage spiking.

  6. 1440000bytes commented at 6:00 pm on January 13, 2025: none

    If there are tradeoffs (speed, memory usage etc.) involved in changing default batch size, then it could remain the same.

    Maybe a config option can be provided to change it.

  7. sipa commented at 6:02 pm on January 13, 2025: member
    There is a config option. This is about changing the dedault.
  8. 1440000bytes commented at 6:12 pm on January 13, 2025: none

    There is a config option. This is about changing the dedault.

    Just realized dbbatchsize already exists.

  9. l0rinc commented at 6:58 pm on January 13, 2025: contributor

    If that spike exceeds the memory the process can allocate it causes a crash

    Thanks for the context, @sipa. On the positive side, the extra allocation is constant (or at least non-proportional with the usage) and it’s narrowing the window for other crashes during flushing (https://github.com/bitcoin/bitcoin/pull/30611 will also likely help here). This change may also enable another one (that I’m currently re-measuring to be sure), which seems to halve the remaining flush time again (by sorting the values in descending order before adding them to the batch), e.g. from 30 minutes (on master) to 10 (with this change included).

  10. luke-jr commented at 9:55 pm on January 14, 2025: member
    Can we predict the memory usage spike size? Presumably as we flush, that releases memory, which allows for a larger and larger batch size?
  11. l0rinc commented at 10:32 am on January 16, 2025: contributor

    Since profilers may not catch these short-lived spikes, I’ve instrumented the code, loaded the UTXO set (as described the in the PR), parsed the logged flushing times and memory usages and plotted them against each other to see the effect of the batch size increase.

      0diff --git a/src/txdb.cpp b/src/txdb.cpp
  1--- a/src/txdb.cpp	(revision d249a353be58868d41d2a7c57357038ffd779eba)
  2+++ b/src/txdb.cpp	(revision bae884969d35469320ed9967736eb15b5d87edff)
  3@@ -90,7 +90,81 @@
  4     return vhashHeadBlocks;
  5 }
  6
  7+/*
  8+ * Author:  David Robert Nadeau
  9+ * Site:    http://NadeauSoftware.com/
 10+ * License: Creative Commons Attribution 3.0 Unported License
 11+ *          http://creativecommons.org/licenses/by/3.0/deed.en_US
 12+ */
 13+#if defined(_WIN32)
 14+#include <windows.h>
 15+#include <psapi.h>
 16+
 17+#elif defined(__unix__) || defined(__unix) || defined(unix) || (defined(__APPLE__) && defined(__MACH__))
 18+#include <unistd.h>
 19+#include <sys/resource.h>
 20+
 21+#if defined(__APPLE__) && defined(__MACH__)
 22+#include <mach/mach.h>
 23+
 24+#elif (defined(_AIX) || defined(__TOS__AIX__)) || (defined(__sun__) || defined(__sun) || defined(sun) && (defined(__SVR4) || defined(__svr4__)))
 25+#include <fcntl.h>
 26+#include <procfs.h>
 27+
 28+#elif defined(__linux__) || defined(__linux) || defined(linux) || defined(__gnu_linux__)
 29+#include <stdio.h>
 30+
 31+#endif
 32+
 33+#else
 34+#error "Cannot define  getCurrentRSS( ) for an unknown OS."
 35+#endif
 36+
 37+/**
 38+ * Returns the current resident set size (physical memory use) measured
 39+ * in bytes, or zero if the value cannot be determined on this OS.
 40+ */
 41+size_t getCurrentRSS( )
 42+{
 43+#if defined(_WIN32)
 44+    /* Windows -------------------------------------------------- */
 45+    PROCESS_MEMORY_COUNTERS info;
 46+    GetProcessMemoryInfo( GetCurrentProcess( ), &info, sizeof(info) );
 47+    return (size_t)info.WorkingSetSize;
 48+
 49+#elif defined(__APPLE__) && defined(__MACH__)
 50+    /* OSX ------------------------------------------------------ */
 51+    struct mach_task_basic_info info;
 52+    mach_msg_type_number_t infoCount = MACH_TASK_BASIC_INFO_COUNT;
 53+    if ( task_info( mach_task_self( ), MACH_TASK_BASIC_INFO,
 54+        (task_info_t)&info, &infoCount ) != KERN_SUCCESS )
 55+        return (size_t)0L;      /* Can't access? */
 56+    return (size_t)info.resident_size;
 57+
 58+#elif defined(__linux__) || defined(__linux) || defined(linux) || defined(__gnu_linux__)
 59+    /* Linux ---------------------------------------------------- */
 60+    long rss = 0L;
 61+    FILE* fp = NULL;
 62+    if ( (fp = fopen( "/proc/self/statm", "r" )) == NULL )
 63+        return (size_t)0L;      /* Can't open? */
 64+    if ( fscanf( fp, "%*s%ld", &rss ) != 1 )
 65+    {
 66+        fclose( fp );
 67+        return (size_t)0L;      /* Can't read? */
 68+    }
 69+    fclose( fp );
 70+    return (size_t)rss * (size_t)sysconf( _SC_PAGESIZE);
 71+
 72+#else
 73+    /* AIX, BSD, Solaris, and Unknown OS ------------------------ */
 74+    return (size_t)0L;          /* Unsupported. */
 75+#endif
 76+}
 77+
 78 bool CCoinsViewDB::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &hashBlock) {
 79+    const auto start = std::chrono::steady_clock::now();
 80+    size_t max_mem{getCurrentRSS()};
 81+
 82     CDBBatch batch(*m_db);
 83     size_t count = 0;
 84     size_t changed = 0;
 85@@ -129,7 +203,11 @@
 86         it = cursor.NextAndMaybeErase(*it);
 87         if (batch.SizeEstimate() > m_options.batch_write_bytes) {
 88             LogDebug(BCLog::COINDB, "Writing partial batch of %.2f MiB\n", batch.SizeEstimate() * (1.0 / 1048576.0));
 89+
 90+            max_mem = std::max(max_mem, getCurrentRSS());
 91             m_db->WriteBatch(batch);
 92+            max_mem = std::max(max_mem, getCurrentRSS());
 93+
 94             batch.Clear();
 95             if (m_options.simulate_crash_ratio) {
 96                 static FastRandomContext rng;
 97@@ -146,8 +224,16 @@
 98     batch.Write(DB_BEST_BLOCK, hashBlock);
 99
100     LogDebug(BCLog::COINDB, "Writing final batch of %.2f MiB\n", batch.SizeEstimate() * (1.0 / 1048576.0));
101+
102+    max_mem = std::max(max_mem, getCurrentRSS());
103     bool ret = m_db->WriteBatch(batch);
104+    max_mem = std::max(max_mem, getCurrentRSS());
105+
106     LogDebug(BCLog::COINDB, "Committed %u changed transaction outputs (out of %u) to coin database...\n", (unsigned int)changed, (unsigned int)count);
107+    if (changed > 0) {
108+        const auto end{std::chrono::steady_clock::now()};
109+        LogInfo("BatchWrite took=%dms, maxMem=%dMiB", duration_cast<std::chrono::milliseconds>(end - start).count(), max_mem >> 20);
110+    }
111     return ret;
112 }

      0import os
  1import re
  2import shutil
  3import statistics
  4import subprocess
  5import time
  6import datetime
  7import argparse
  8import matplotlib.pyplot as plt  # python3.12 -m pip install matplotlib --break-system-packages
  9
 10# Regex to parse logs
 11BATCHWRITE_REGEX = re.compile(r"^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) BatchWrite took=(\d+)ms, maxMem=(\d+)MiB")
 12
 13
 14def parse_log(archive):
 15    """Parse the log file to extract elapsed times, flush times, and memory usage."""
 16    start_time = None
 17    elapsed, batchwrite_times, usage_snapshots = [], [], []
 18    with open(archive, "r") as f:
 19        for line in f:
 20            if m := BATCHWRITE_REGEX.search(line):
 21                dt = datetime.datetime.strptime(m.group(1), "%Y-%m-%dT%H:%M:%SZ")
 22                if start_time is None:
 23                    start_time = dt
 24                elapsed.append((dt - start_time).total_seconds())
 25                batchwrite_times.append(int(m.group(2)))
 26                usage_snapshots.append(int(m.group(3)))
 27    return elapsed, batchwrite_times, usage_snapshots
 28
 29
 30def plot_results(results, output_dir):
 31    """Create separate plots for flush times and memory usage."""
 32    if len(results) != 2:
 33        print("plot_results() requires exactly 2 runs for comparison.")
 34        return
 35
 36    (dbbatch0, elapsed0, flush0, mem0) = results[0]
 37    (dbbatch1, elapsed1, flush1, mem1) = results[1]
 38
 39    # Compute percentage differences
 40    avg_flush0, avg_flush1 = statistics.mean(flush0), statistics.mean(flush1)
 41    max_mem0, max_mem1 = max(mem0), max(mem1)
 42    flush_improvement = round(((avg_flush0 - avg_flush1) / avg_flush0) * 100, 1)
 43    mem_increase = round(((max_mem1 - max_mem0) / max_mem0) * 100, 1)
 44
 45    # Plot flush times
 46    plt.figure(figsize=(16, 8))
 47    plt.plot(elapsed0, flush0, color="red", linestyle="-", label=f"Flush Times (dbbatch={dbbatch0})")
 48    plt.axhline(y=avg_flush0, color="red", linestyle="--", alpha=0.5, label=f"Mean ({dbbatch0})={avg_flush0:.1f}ms")
 49    plt.plot(elapsed1, flush1, color="orange", linestyle="-", label=f"Flush Times (dbbatch={dbbatch1})")
 50    plt.axhline(y=avg_flush1, color="orange", linestyle="--", alpha=0.5, label=f"Mean ({dbbatch1})={avg_flush1:.1f}ms")
 51    plt.title(f"Flush Times (dbbatch {dbbatch0} vs {dbbatch1}) — {abs(flush_improvement)}% {'faster' if flush_improvement > 0 else 'slower'}")
 52    plt.xlabel("Elapsed Time (seconds)")
 53    plt.ylabel("Flush Times (ms)")
 54    plt.legend()
 55    plt.grid(True)
 56    plt.tight_layout()
 57    flush_out_file = os.path.join(output_dir, "plot_flush_times.png")
 58    plt.savefig(flush_out_file)
 59    print(f"Flush Times plot saved as {flush_out_file}")
 60    plt.close()
 61
 62    # Plot memory usage
 63    plt.figure(figsize=(16, 8))
 64    plt.plot(elapsed0, mem0, color="blue", linestyle="-", label=f"Memory (dbbatch={dbbatch0})")
 65    plt.axhline(y=max_mem0, color="blue", linestyle="--", alpha=0.5, label=f"Max Mem ({dbbatch0})={max_mem0}MiB")
 66    plt.plot(elapsed1, mem1, color="green", linestyle="-", label=f"Memory (dbbatch={dbbatch1})")
 67    plt.axhline(y=max_mem1, color="green", linestyle="--", alpha=0.5, label=f"Max Mem ({dbbatch1})={max_mem1}MiB")
 68    plt.title(f"Memory Usage (dbbatch {dbbatch0} vs {dbbatch1}) — {abs(mem_increase)}% {'higher' if mem_increase > 0 else 'lower'}")
 69    plt.xlabel("Elapsed Time (seconds)")
 70    plt.ylabel("Memory Usage (MiB)")
 71    plt.legend()
 72    plt.grid(True)
 73    plt.tight_layout()
 74    mem_out_file = os.path.join(output_dir, "plot_memory_usage.png")
 75    plt.savefig(mem_out_file)
 76    print(f"Memory Usage plot saved as {mem_out_file}")
 77    plt.close()
 78
 79
 80def loadtxoutset(dbbatchsize, datadir, bitcoin_cli, bitcoind, utxo_file):
 81    """Load the UTXO set and run the Bitcoin node."""
 82    archive = os.path.join(datadir, f"results_dbbatch-{dbbatchsize}.log")
 83
 84    # Skip if logs already exist
 85    if os.path.exists(archive):
 86        print(f"Log file {archive} already exists. Skipping loadtxoutset for dbbatchsize={dbbatchsize}.")
 87        return
 88
 89    os.makedirs(datadir, exist_ok=True)
 90    debug_log = os.path.join(datadir, "debug.log")
 91
 92    try:
 93        print("Cleaning up previous run")
 94        for subdir in ["chainstate", "chainstate_snapshot"]:
 95            shutil.rmtree(os.path.join(datadir, subdir), ignore_errors=True)
 96
 97        print("Preparing UTXO load")
 98        subprocess.run([bitcoind, f"-datadir={datadir}", "-stopatheight=1"], cwd=bitcoin_core_path)
 99        os.remove(debug_log)
100
101        print(f"Starting bitcoind with dbbatchsize={dbbatchsize}")
102        subprocess.run([bitcoind, f"-datadir={datadir}", "-daemon", "-blocksonly=1", "-connect=0", f"-dbbatchsize={dbbatchsize}", f"-dbcache={440}"], cwd=bitcoin_core_path)
103        time.sleep(5)
104
105        print("Loading UTXO set")
106        subprocess.run([bitcoin_cli, f"-datadir={datadir}", "loadtxoutset", utxo_file], cwd=bitcoin_core_path)
107    except Exception as e:
108        print(f"Error during loadtxoutset for dbbatchsize={dbbatchsize}: {e}")
109        raise
110    finally:
111        print("Stopping bitcoind...")
112        subprocess.run([bitcoin_cli, f"-datadir={datadir}", "stop"], cwd=bitcoin_core_path, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
113        time.sleep(5)
114
115    shutil.copy2(debug_log, archive)
116    print(f"Archived logs to {archive}")
117
118
119if __name__ == "__main__":
120    # Parse script arguments
121    parser = argparse.ArgumentParser(description="Benchmark Bitcoin dbbatchsize configurations.")
122    parser.add_argument("--utxo-file", required=True, help="Path to the UTXO snapshot file.")
123    parser.add_argument("--bitcoin-core-path", required=True, help="Path to the Bitcoin Core project directory.")
124    args = parser.parse_args()
125
126    utxo_file = args.utxo_file
127    bitcoin_core_path = args.bitcoin_core_path
128    datadir = os.path.join(bitcoin_core_path, "demo")
129    debug_log = os.path.join(datadir, "debug.log")
130    bitcoin_cli = os.path.join(bitcoin_core_path, "build/src/bitcoin-cli")
131    bitcoind = os.path.join(bitcoin_core_path, "build/src/bitcoind")
132
133    # Build Bitcoin Core
134    print("Building Bitcoin Core...")
135    subprocess.run(["cmake", "-B", "build", "-DCMAKE_BUILD_TYPE=Release"], cwd=bitcoin_core_path, check=True)
136    subprocess.run(["cmake", "--build", "build", "-j", str(os.cpu_count())], cwd=bitcoin_core_path, check=True)
137
138    # Run tests for each dbbatchsize
139    results = []
140    for dbbatchsize in [16777216, 67108864]:  # Original and proposed
141        loadtxoutset(dbbatchsize, datadir, bitcoin_cli, bitcoind, utxo_file)
142        archive = os.path.join(datadir, f"results_dbbatch-{dbbatchsize}.log")
143        elapsed, batchwrite_times, usage_snapshots = parse_log(archive)
144        results.append((dbbatchsize, elapsed, batchwrite_times, usage_snapshots))
145
146    # Plot results
147    plot_results(results, bitcoin_core_path)
148    print("All configurations processed.")

    For standard dbcache values the results are very close (though the memory measurements aren’t as scientific as I’d like them to be (probably because there is still enough memory), some runs even indicate that 16MiB consumes a bit more memory than the 64MiB version), but the trend seems to be clear from the produced plots: the batch writes are faster (and seem more predictable) with bigger batches, while the memory usage is only slightly higher.

    plot_flush_times plot_memory_usage

    Is there any other way that you’d like me to test this @sipa, @luke-jr, @1440000bytes?

  12. DrahtBot added the label Needs rebase on Jan 16, 2025
  13. coins: bump default LevelDB write batch size to 64 MiB 
    The UTXO set has grown significantly, and flushing it from memory to LevelDB often takes over 20 minutes after a successful IBD with large dbcache values.
The final UTXO set is written to disk in batches, which LevelDB sorts into SST files.
By increasing the default batch size, we can reduce overhead from repeated compaction cycles, minimize constant overhead per batch, and achieve more sequential writes.

Experiments with different batch sizes (loaded via assumeutxo at block 840k, then measuring final flush time) show that 64 MiB batches significantly reduce flush time without notably increasing memory usage:

| dbbatchsize | flush_sum (ms) |
|-------------|----------------|
| 8 MiB       | ~240,000       |
| 16 MiB      | ~220,000       |
| 32 MiB      | ~200,000       |
| *64 MiB*    | *~150,000*     |
| 128 MiB     | ~156,000       |
| 256 MiB     | ~166,000       |
| 512 MiB     | ~186,000       |
| 1 GiB       | ~186,000       |

Checking the impact of a `-reindex-chainstate` with `-stopatheight=878000` and `-dbcache=30000` gives:
16 << 20
```
2025-01-12T07:31:05Z Flushed fee estimates to fee_estimates.dat.
2025-01-12T07:31:05Z [warning] Flushing large (26 GiB) UTXO set to disk, it may take several minutes
2025-01-12T07:53:51Z Shutdown: done
```
Flush time: 22 minutes and 46 seconds

64 >> 20
```
2025-01-12T18:30:00Z Flushed fee estimates to fee_estimates.dat.
2025-01-12T18:30:00Z [warning] Flushing large (26 GiB) UTXO set to disk, it may take several minutes
2025-01-12T18:44:43Z Shutdown: done
```
Flush time: ~14 minutes 43 seconds.
    868413340f
  14. l0rinc force-pushed on Jan 16, 2025
  15. luke-jr commented at 10:56 pm on January 16, 2025: member

    I think those graphs need to be on height rather than seconds. The larger dbbatchsize making it faster means it gets further in the chain, leading to the higher max at the end…

    I would expect both lines to be essentially overlapping except during flushes.

  16. l0rinc commented at 10:14 am on January 17, 2025: contributor

    I would expect both lines to be essentially overlapping except during flushes.

    I was only measuring the memory here during flushes. There is no direct height available there, but if we instrument UpdateTipLog instead (and fetch some data from the assumeUTXO height), we’d get:

    dbbatchsize=16MiB:

    image

    dbbatchsize=64MiB (+ experimental sorting):

    image

    overlapped (blue 16, green 64): image

  17. DrahtBot removed the label Needs rebase on Jan 17, 2025
  18. luke-jr referenced this in commit aa56eaaa78 on Feb 22, 2025
  19. l0rinc renamed this:
    optimization: increase default LevelDB write batch size to 64 MiB
    [IBD] Flush UXTOs in bigger batches
    on Mar 12, 2025
  20. l0rinc renamed this:
    [IBD] Flush UXTOs in bigger batches
    [IBD] flush UXTOs in bigger batches
    on Mar 12, 2025
  21. ryanofsky approved
  22. ryanofsky commented at 7:32 pm on March 12, 2025: contributor

    Code review ACK 868413340f8d6058d74186b65ac3498d6b7f254a

    If that spike exceeds the memory the process can allocate it causes a crash, at a particularly bad time (may require a replay to fix, which may be slower than just reprocessing the blocks).

    It is difficult for me to have a sense of how safe this change is but I’d hope we are not currently pushing systems so close to the edge that using an extra 48mb will cause them to start crashing. This does seem like a nice performance improvment if it doesn’t cause crashes.

    In theory, we could dynamically limit the batch size based on available memory to mitigate the risk of crashes. However, since the batch size is already small and further increases don’t provide much additional benefit (per the commit message), that added complexity probably isn’t worth it here.

  23. jonatack commented at 4:08 pm on March 13, 2025: member

    Concept ACK on raising the default from 16 to 67 megabytes (note that -dbbatchsize is a hidden -help-debug config option). Testing.

    0$ ./build/bin/bitcoind -help-debug | grep -A2 dbbatch
1  -dbbatchsize
2       Maximum database write batch size in bytes (default: 67108864)
  24. jonatack commented at 8:22 pm on March 13, 2025: member

    Review and testing-under-way ACK 868413340f8d6058d74186b65ac3498d6b7f254a

    As described in the pull description, this value was set in 2017 in #10148 and hasn’t been changed since.

    Maybe a config option can be provided to change it.

    Just realized dbbatchsize already exists.

    This is a hidden config option and the rationale might be #10148 (review):

    “the relevant constraint is the memory usage peak from allocating the batch, which depends on the batch memory usage, not dbcache memory usage. Also, I don’t think anyone will need to change this property (except for tests, where it’s very useful to get much more frequent partial flushes).”

    Though, if an older machine might be constrained in memory during IBD after upgrading bitcoin core, that might be an argument to unhide -dbbatchsize with the larger default (and document its use in the conf file; I don’t recall if conf file doc gen is automated yet).

  25. l0rinc commented at 1:24 pm on March 14, 2025: contributor

    Reran the benchmarks on a Hetzner Linux machine using GCC with the new AssumeUTXO 880k set, parsing the logged flush times and plotting the results. For different dbcache (440, 5000, 30000) and dbbatchsize (4-256MiB range, 16MiB (current) and 64MiB (proposed) are highlighted and trendline is added for clarity) sizes, each one twice for stability:

    Sorting the same measurements (+ sums) might give us a better understanding of the trends:

    Edit: running it on my Mac M4 Max gives more varied results in general:

    I will investigate if it depends on the compiler and operating system - can anyone else check Windows?

  26. Eunovo commented at 3:01 pm on March 20, 2025: contributor

    I used valgrind --tool=massif valgrind massif to profile memory usage of bitcoin-core IBD(840002 to 850000) for 16MiB and 64MiB batch sizes. Tested on an Ubuntu 22.04 Server and compiled with GCC 11.4.0.

    Memory Usage Tests

    Method

     0mkdir demo
 1build/bin/bitcoind -datadir=demo -stopatheight=1
 2build/bin/bitcoind -datadir=demo -daemon -blocksonly=1 -stopatheight=840001
 3build/bin/bitcoin-cli -datadir=demo -rpcclienttimeout=0 loadtxoutset ~/utxo-840000.dat
 4mkdir demo-backup
 5cp -r demo/* demo-backup
 6
 7valgrind --tool=massif build/bin/bitcoind -datadir=demo -daemon -blocksonly=1 -dbbatchsize=16777216 -stopatheight=850000
 8
 9rm -rf demo
10mkdir demo
11cp -r demo-backup/* demo
12
13valgrind --tool=massif build/bin/bitcoind -datadir=demo -daemon -blocksonly=1 -dbbatchsize=67108864 -stopatheight=850000

    Results

    Running ms_print on the massif output files produces

     0--------------------------------------------------------------------------------
 1Command:            build/bin/bitcoind -datadir=../demo -daemon -blocksonly=1 -dbbatchsize=16777216 -stopatheight=850000
 2Massif arguments:   (none)
 3ms_print arguments: massif.out.674977
 4--------------------------------------------------------------------------------
 5
 6
 7    MB
 8811.4^        #                                                               
 9     |     :: #                                                       :       
10     |     :  #                                                       :       
11     |     :  #             ::               ::                       :       
12     |  :  :  # @           :                :           :   :        :       
13     |  :  :  # @          ::  :   ::        :   @@      :   :   :    :  ::  @
14     |  :  : :# @   :  ::  ::  :   :        ::   @   :   :   :   :    :  :   @
15     | ::  : :# @ :::  :   ::  :   :  ::    ::   @   :   :  ::   :  :::  :   @
16     | ::  : :# @ : :  :  :::  :   :  :   :::: ::@   :  ::  ::   :  : :  :   @
17     | ::  : :#:@:: :  :  :::  : :::  :   : :: : @   :  ::  :: :::  : :  :   @
18     | ::::: :#:@:: :  :  ::: @: : :  :   : :: : @   :  ::  :: : :  : ::::   @
19     | ::: : :#:@:: :::: :::: @: : : ::   : :: : @   :  :::::: : :::: :: : ::@
20     | ::: : :#:@:: :: : :::: @: : : :: ::: :: : @   :@@::: :::: :: : :: : : @
21     | ::: : :#:@:: :: : :::: @::: : :: : : :: : @ :::@ ::: :::: :: : :: : : @
22     | ::: : :#:@:: :: : :::: @::: : :: : : :: : @ : :@ ::: :::: :: : :: : : @
23     | ::: : :#:@:: :: : :::: @::: : :: : : :: : @ : :@ ::: :::: :: : :: : : @
24     | ::: : :#:@:: :: : :::: @::: : :: : : :: : @ : :@ ::: :::: :: : :: : : @
25     | ::: : :#:@:: :: : :::: @::: : :: : : :: : @ : :@ ::: :::: :: : :: : : @
26     | ::: : :#:@:: :: : :::: @::: : :: : : :: : @ : :@ ::: :::: :: : :: : : @
27     | ::: : :#:@:: :: : :::: @::: : :: : : :: : @ : :@ ::: :::: :: : :: : : @
28   0 +----------------------------------------------------------------------->Ti
29     0                                                                   8.610

     0--------------------------------------------------------------------------------
 1Command:            build/bin/bitcoind -datadir=../demo -daemon -blocksonly=1 -dbbatchsize=67108864 -stopatheight=850000
 2Massif arguments:   (none)
 3ms_print arguments: massif.out.682470
 4--------------------------------------------------------------------------------
 5
 6
 7    GB
 81.005^          ##                                                            
 9     |          #                        ::                                   
10     |          #               :        :   ::                               
11     |          #               :        :   :                        :       
12     |          #               :        :   :                        :       
13     |          #               :        :   :                        :      :
14     |          #               :        :   :        :               :      :
15     |  :  ::   #      :    :   :        :  ::   :    : ::   :   ::   :   :: :
16     |  :  : @  #      :    :  ::  :   :::  ::   :    : :    :   :  :::   :  @
17     |  :  : @  # ::   :    : :::  :   : : :::  ::  ::: :  :::  ::  : : :::  @
18     |  :  : @  # :   ::  ::: :::  :   : : :::  ::  : : :  : :  ::  : : : : :@
19     | ::::: @::# :   ::  : : :::  :  :: : ::: :::  : :::  : :  :: :: ::: : :@
20     | ::: : @: # :   ::  : : ::::::  :: : ::: :::  : :::  : :  :: :: ::: : :@
21     | ::: : @: # : ::::  : :::::: :  :: : ::: :::::: :::  : ::::: :: ::: : :@
22     | ::: : @: # : : ::::: :::::: ::::: : ::: :::: : ::: :: :: :: :: ::: : :@
23     | ::: : @: # : : ::: : :::::: :: :: : ::: :::: : ::: :: :: :: :: ::: : :@
24     | ::: : @: # : : ::: : :::::: :: :: : ::: :::: : ::: :: :: :: :: ::: : :@
25     | ::: : @: # : : ::: : :::::: :: :: : ::: :::: : ::: :: :: :: :: ::: : :@
26     | ::: : @: # : : ::: : :::::: :: :: : ::: :::: : ::: :: :: :: :: ::: : :@
27     | ::: : @: # : : ::: : :::::: :: :: : ::: :::: : ::: :: :: :: :: ::: : :@
28   0 +----------------------------------------------------------------------->Ti
29     0                                                                   8.477

    The graphs show that with the default dbcache size, bitcoin-core memory usage peaks at 811.4MB for 16MiB dbbatchsize and 1.005GB for 64MiB dbbatchsize .

    If I’m not mistaken, Bitcoin-core runs on really small devices like Raspberry Pi with only 1GB RAM, if we raise the default dbbatchsize to 64GB, there’s a chance that Bitcoin-core could OOM if there’s not enough Swap space. Also note that if the bitcoin-core process uses swap space, it will run slower thereby negating potential speed gains from raising the dbbatchsize.

    Speed Tests

    Method

    Apply the following diff to output Batch write time data to debug.log.

     0diff --git a/src/txdb.cpp b/src/txdb.cpp
 1index 1622039d63..ef73378ce6 100644
 2--- a/src/txdb.cpp
 3+++ b/src/txdb.cpp
 4@@ -91,6 +91,7 @@ std::vector<uint256> CCoinsViewDB::GetHeadBlocks() const {
 5 }
 6 
 7 bool CCoinsViewDB::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &hashBlock) {
 8+    const auto start = std::chrono::steady_clock::now();
 9     CDBBatch batch(*m_db);
10     size_t count = 0;
11     size_t changed = 0;
12@@ -147,7 +148,9 @@ bool CCoinsViewDB::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &hashB
13 
14     LogDebug(BCLog::COINDB, "Writing final batch of %.2f MiB\n", batch.SizeEstimate() * (1.0 / 1048576.0));
15     bool ret = m_db->WriteBatch(batch);
16+    const auto end{std::chrono::steady_clock::now()};
17     LogDebug(BCLog::COINDB, "Committed %u changed transaction outputs (out of %u) to coin database...\n", (unsigned int)changed, (unsigned int)count);
18+    LogDebug(BCLog::COINDB, "BatchWrite: completed in %dms\n", duration_cast<std::chrono::milliseconds>(end - start).count());
19     return ret;
20 }

     0mkdir demo
 1build/bin/bitcoind -datadir=demo -stopatheight=1
 2build/bin/bitcoind -datadir=demo -daemon -blocksonly=1 -stopatheight=840001
 3build/bin/bitcoin-cli -datadir=demo -rpcclienttimeout=0 loadtxoutset ~/utxo-840000.dat
 4mkdir demo-backup
 5cp -r demo/* demo-backup
 6
 7build/bin/bitcoind -datadir=demo -daemon -blocksonly=1 -dbbatchsize=16777216 -stopatheight=850000 -debug=coindb
 8cp demo/debug.log defaultcache_16MiB_dbbatchsize.log
 9
10rm -rf demo
11mkdir demo
12cp -r demo-backup/* demo
13
14build/bin/bitcoind -datadir=demo -daemon -blocksonly=1 -dbbatchsize=67108864 -stopatheight=850000 -debug=coindb
15cp demo/debug.log defaultcache_64MiB_dbbatchsize.log
16
17// Parse log files using https://github.com/Eunovo/31645-data/blob/main/extract_batch_write_data.py
18cat defaultcache_64MiB_dbbatchsize.log | grep "BatchWrite" -B 1 | xargs ./extract_batch_write_data.py > defaultcache_64MiB_dbbatchsize.result

    Results

     0Logfile: defaultcache_16MiB_dbbatchsize.log
 1--------------------------------------------
 2Average num_txouts_per_ms: 1018.2494977251025
 3--------------------------------------------
 4num_txouts,write_times_ms,num_txouts_per_ms
 52862776,2830,1011.5816254416961
 62934562,2851,1029.3097158891617
 72921143,2753,1061.0762804213584
 82940809,2861,1027.8954910870325
 92948759,2870,1027.4421602787456
102979042,2967,1004.0586450960566
112956426,2917,1013.5159410353102
123048466,3060,996.2307189542483
132978184,2940,1012.9877551020409
142944033,2945,999.6716468590832
152973528,2970,1001.1878787878788
162970816,2889,1028.3198338525442
172982352,2898,1029.1069703243616
182960008,2776,1066.28530259366
193016540,2952,1021.8631436314363
203007010,3033,991.4309264754369
212976775,2982,998.2478202548625
223084773,3085,999.92641815235
232069630,2016,1026.6021825396826

     0Logfile: defaultcache_64MiB_dbbatchsize.log
 1--------------------------------------------
 2Average num_txouts_per_ms: 735.7374156918239
 3--------------------------------------------
 4num_txouts,write_times_ms,num_txouts_per_ms
 52862776,3881,737.6387528987375
 62895729,3942,734.5837138508372
 72941575,3704,794.161717062635
 82941158,3924,749.5305810397554
 92965901,4129,718.3097602325018
102978043,4095,727.2388278388279
112946445,4041,729.1375897055184
123034443,4319,702.5799953692984
132993584,4154,720.6509388541165
142947744,4113,716.6895210308777
152979400,4072,731.679764243615
162968346,3819,777.2573972244043
172991885,3932,760.9066632756867
182960911,3844,770.2682101977107
192942739,4000,735.68475
203031336,4282,707.9252685660906
213007839,4118,730.4125789218067
223057094,4257,718.133427296218
232137921,2985,716.2214405360133

    Running with 64MiB dbbatchsize turns out to be about 27% slower. I was only able to oberserve a speedup when doing a large flush which doesn’t happen often with the default 450MiB cache size.

  27. sipa commented at 3:41 pm on March 20, 2025: member
    Maybe it’s a better approach to scale the (default) batch size in function of the dbcache itself? That way, people who have configured their node to run with a low memory footprint to begin with will have relatively small flushing spikes, but people who run on beefy systems and thus likely can tolerate bigger spikes too get to enjoy the performance benefit those may be provide?
  28. l0rinc renamed this:
    [IBD] flush UXTOs in bigger batches
    [IBD] flush UTXOs in bigger batches
    on Mar 25, 2025
  29. l0rinc commented at 3:23 pm on March 26, 2025: contributor

    Thanks for your measurements @Eunovo!

    I’m not sure if we support 1GiB of total memory or not, but even 8 years ago that required a lower dbcache that 450MiB.

    It took me a bit, but I finally finished comparing the 16 and 64 MiB batches with valgrind --tool=massif for full IBD until block 888888.

     0COMMITS="998386d4462f5e06412303ba559791da83b913fb 5cefc4bbadaa06f22089dd7991fac17ed5b283ca"; \
 1STOP_HEIGHT=888888; DBCACHE=1000; \
 2C_COMPILER=gcc; CXX_COMPILER=g++; \
 3DATA_DIR="/mnt/my_storage/BitcoinData"; \
 4LOG_DIR="/mnt/my_storage/logs"; \
 5mkdir -p $LOG_DIR; \
 6for COMMIT in $COMMITS; do \
 7  git fetch --all -q && git fetch origin $COMMIT && git log -1 --oneline $COMMIT && \
 8  killall bitcoind || true && \
 9  rm -rf $DATA_DIR/* && \
10  git checkout $COMMIT && \
11  git clean -fxd && \
12  git reset --hard && \
13  cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF -DCMAKE_C_COMPILER=$C_COMPILER -DCMAKE_CXX_COMPILER=$CXX_COMPILER && \
14  cmake --build build -j$(nproc) --target bitcoind && \
15  ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=1 || true && \
16  valgrind --tool=massif --time-unit=ms \
17    --massif-out-file="$LOG_DIR/massif-$COMMIT-$STOP_HEIGHT-$DBCACHE.out" \
18    ./build/bin/bitcoind -datadir=$DATA_DIR \
19    -stopatheight=$STOP_HEIGHT \
20    -dbcache=$DBCACHE \
21    -blocksonly=1 \
22    -printtoconsole=1 && \
23  cp $DATA_DIR/debug.log "$LOG_DIR/debug-$COMMIT-$STOP_HEIGHT-$DBCACHE.log" && \
24  ms_print "$LOG_DIR/massif-$COMMIT-$STOP_HEIGHT-$DBCACHE.out" | tee "$LOG_DIR/massif-$COMMIT-$STOP_HEIGHT-$DBCACHE.txt"; \
25done

    The memory difference is measurable and non-negligible (~200MiB, seems the buffer may be copied 4 times), especially for small memory usecases. I have opened https://github.com/google/leveldb/pull/1259 to enable pre-sizing the memory, which might help us in reducing the spikes slightly.

    16 » 10 with 1GiB dbcache:

    64 » 10 with 1GiB dbcache:

    I like @sipa’s suggestion of making the batch size depend on cache size, I also thought of that while investigating it, but that may mean that the hidden dbbatchsize has to be changed. Basically we could have a fixed number of batch fills, currently 28 GiB is batched into 16 MiB chunks, we could set it to 1000 fixed writes - or something similar, I’ll investigate.

    And about the speedups, I found that on Mac the differences were tiny (see attachments), but on Linux they were basically always faster, regardless of the dbcache size. We have to find out in which case it’s faster and when it’s slower, thanks for helping me with that.

  30. Eunovo commented at 6:05 am on March 27, 2025: contributor

    The memory difference is measurable and non-negligible (~200MiB, seems the buffer may be copied 4 times), especially for small memory usecases. I have opened google/leveldb#1259 to enable pre-sizing the memory, which might help us in reducing the spikes slightly. @l0rinc you can also check leveldb.approximate-memory-usage. Use db.GetProperty before and after batch writes. I noticed that it shows a non-negligible spike in memory usage. Could be useful to quickly check if https://github.com/google/leveldb/pull/1259 works

  31. l0rinc marked this as a draft on Mar 30, 2025
  32. l0rinc commented at 3:47 pm on March 30, 2025: contributor

    I’ve experimented a bit with changing the fixed write batch size to a dynamic value that scales with dbcache size, basically something like:

    0static int64_t GetDefaultDbBatchSize(int64_t dbcache_bytes) {
1    return std::min<int64_t>(
2        MAX_DB_CACHE_BATCH, // 512 MiB upper limit
3        std::max<int64_t>(
4            MIN_DB_CACHE,   // 4 MiB lower limit
5            dbcache_bytes * DEFAULT_KERNEL_CACHE_BATCH_SIZE / DEFAULT_KERNEL_CACHE
6        )
7    );
8}

    Corresponding to:

    dbcache (MiB) Before (Fixed) After (Dynamic)
    Batch Size Batch Count Batch Size Batch Count
    4 16 MiB 1,546 4 MiB 3,111
    10 16 MiB 1,546 4 MiB 3,090
    45 16 MiB 780 4 MiB 3,090
    100 16 MiB 929 4 MiB 3,085
    450 (default) 16 MiB 775 19.6 MiB 646
    1,000 16 MiB 788 44 MiB 285
    4,500 16 MiB 774 200 MiB 69
    10,000 16 MiB 772 444 MiB 33
    45,000 16 MiB 771 512 MiB 28

    This would:

    • Provide smaller batches (4 MiB) for memory-constrained systems (likely making them a bit slower).
    • Maintains a similar batch size for default dbcache (20 MiB instead of 16 MiB, making it a bit faster, using a bit more memory).
    • Scales to larger batches for high-memory systems (up to 512 MiB, making it a lot faster, using a lot more memory).

    The chart illustrates how the dynamic batch sizing maintains approximately a 1:22 ratio (~4.5%) between batch size and dbcache size, while capping at 4 MiB minimum for memory-constrained systems and 512 MiB maximum for high-memory systems (I chose this simply to maintain a similar batch size for the default value (20 MiB instead of 16 MiB for dbcache=440) while scaling linearly for other values).

    I have drafted the PR to fine-tune these values based on performance and memory usage.


l0rinc DrahtBot sipa 1440000bytes luke-jr ryanofsky jonatack Eunovo

Labels
UTXO Db and Indexes

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-03-31 09:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me