bench: `WalletLoadingDescriptors` slowdown #36510

Locally running this takes ~180s:

tool_bench_sanity_check.py --bench=WalletEncryptDescriptors                      | ✓ Passed  | 3 s
tool_bench_sanity_check.py --bench=WalletIsMineDescriptors                       | ✓ Passed  | 1 s
tool_bench_sanity_check.py --bench=WalletIsMineMigratedDescriptors               | ✓ Passed  | 2 s
tool_bench_sanity_check.py --bench=WalletLoadingDescriptors                      | ✓ Passed  | 186 s
tool_bench_sanity_check.py --bench=WalletMigration                               | ✓ Passed  | 6 s
tool_bench_sanity_check.py --bench=WriteBlockBench                               | ✓ Passed  | 0 s

Looks like it's also slowed down in the CI, compared to all other benches (which avg ~0s): https://github.com/bitcoin/bitcoin/actions/runs/38328088251/job/115071335648#step:9:4651:

tool_bench_sanity_check.py --bench=WalletEncryptDescriptors                      | ✓ Passed  | 1 s
tool_bench_sanity_check.py --bench=WalletIsMineDescriptors                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletIsMineMigratedDescriptors               | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletLoadingDescriptors                      | ✓ Passed  | 17 s
tool_bench_sanity_check.py --bench=WalletMigration                               | ✓ Passed  | 1 s
tool_bench_sanity_check.py --bench=WriteBlockBench                               | ✓ Passed  | 0 s

I guess this is just sanitizer/debug overhead? Or is this a recent regression?

just sanitizer/debug overhead

This is running without sanitizers/debug.

Locally, a libc++ Debug build (with -O3) passes in less than 2s. And one with -DCMAKE_CXX_COMPILER='clang++;-stdlib=libc++;-D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_DEBUG' passes in slightly over 2s. Maybe this is a macOS issue?

Maybe this is a macOS issue?

I'm not sure, as the same slowdown occurs on Alpine, where the bench is drastically slower than all other benches: https://github.com/bitcoin/bitcoin/actions/runs/38328088251/job/115071335763#step:11:4677:

[673](https://github.com/bitcoin/bitcoin/actions/runs/38328088251/job/115071335763#step:11:4674)
tool_bench_sanity_check.py --bench=WalletEncryptDescriptors                      | ✓ Passed  | 3 s
tool_bench_sanity_check.py --bench=WalletIsMineDescriptors                       | ✓ Passed  | 0 s
tool_bench_sanity_check.py --bench=WalletIsMineMigratedDescriptors               | ✓ Passed  | 2 s
tool_bench_sanity_check.py --bench=WalletLoadingDescriptors                      | ✓ Passed  | 92 s
tool_bench_sanity_check.py --bench=WalletMigration                               | ✓ Passed  | 2 s
tool_bench_sanity_check.py --bench=WriteBlockBench                               | ✓ Passed  | 0 s

Looking at https://willcl-ark.github.io/bitcoin-core-ci-stats/test-timings/ and filtering by WalletLoadingDescriptors, it shows the same slowdown for many other jobs:

<img width="725" height="566" alt="Image" src="https://github.com/user-attachments/assets/60cedfac-cc03-44bc-8a94-8b51b9c78545" />

I tried with the Alpine CI, but only found a speed-up from 44s->40s on arm64.

The diff I used:

root@ubuntu-32gb-hel1-1:~/b-c# uname --machine
aarch64
root@ubuntu-32gb-hel1-1:~/b-c# git diff -U1
diff --git a/ci/test/00_setup_env_native_alpine_musl.sh b/ci/test/00_setup_env_native_alpine_musl.sh
index 09b07b682c..cf94a61943 100755
--- a/ci/test/00_setup_env_native_alpine_musl.sh
+++ b/ci/test/00_setup_env_native_alpine_musl.sh
@@ -11,5 +11,6 @@ export CI_BASE_PACKAGES="build-base musl-dev pkgconf curl ccache make ninja git
 export PIP_PACKAGES="--break-system-packages pyzmq pycapnp"
-export DEP_OPTS="DEBUG=1"
+export DEP_OPTS="DEBUG=1 NO_QT=1"
 printf -v BITCOIN_CONFIG "%q " \
  --preset=dev-mode \
+ -DBUILD_GUI=OFF \
  -DREDUCE_EXPORTS=ON \
diff --git a/ci/test/03_test_script.sh b/ci/test/03_test_script.sh
index 5d741936ae..72dbab4276 100755
--- a/ci/test/03_test_script.sh
+++ b/ci/test/03_test_script.sh
@@ -127,2 +127,3 @@ fi
 # shellcheck disable=SC2086
+BUILD_TARGETS="bench_bitcoin"
 cmake --build "${BASE_BUILD_DIR}" "$MAKEJOBS" --target $BUILD_TARGETS || (
@@ -133,2 +134,7 @@ cmake --build "${BASE_BUILD_DIR}" "$MAKEJOBS" --target $BUILD_TARGETS || (
 )
+for i in {1..3}; do
+  ( time "${BASE_BUILD_DIR}/bin/bench_bitcoin" -filter=WalletLoadingDescriptors -sanity-check ) &
+done
+wait
+exit 0
 

I also tried with tsan on the same machine, but it too was not changed:

rm -rf ./bld-cmake/   &&  cmake -B ./bld-cmake -DAPPEND_CXXFLAGS='-O3 -g2' -DAPPEND_CFLAGS='-O3 -g2' -DCMAKE_BUILD_TYPE=Debug -DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=mold -DCMAKE_C_COMPILER='clang' -DCMAKE_CXX_COMPILER='clang++' --preset=dev-mode -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DSANITIZERS=thread -DAPPEND_CPPFLAGS="-DARENA_DEBUG -DDEBUG_LOCKCONTENTION"                               && cmake --build ./bld-cmake --parallel  $(nproc) --target=bench_bitcoin && for i in {1..3}; do ( time ./bld-cmake/bin/bench_bitcoin -filter=WalletLoadingDescriptors -sanity-check ) & done; wait   

(both on tmpfs and inside a container)

It would be good to provide the exact steps to reproduce and the exact machine hardware to reproduce, ideally on Linux.

Can you bisect for when this started happening?

On a macOS 26.2 VM (Apple M3 Max Virtual, 8 vCPUs, 32 GiB RAM), the patch below reduced the direct WalletLoadingDescriptors sanity-check runtime from 39–41 seconds to 2.63 seconds. The functional-test wrapper also passed in 2.22 seconds.

Tested at commit badc9574810c23c05ac17a5285c2edf416d2d66b, using a Release build with -O2 and no sanitizers.

<details> <summary>diff</summary>

diff --git a/src/bench/wallet_loading.cpp b/src/bench/wallet_loading.cpp
index 6b1208ea0b..b2b2d7bd29 100644
--- a/src/bench/wallet_loading.cpp
+++ b/src/bench/wallet_loading.cpp
@@ -50,6 +50,9 @@ static void WalletLoadingDescriptors(benchmark::Bench& bench)
     DatabaseOptions options;
     options.require_format = DatabaseFormat::SQLITE;
     options.require_create = true;
+    // Avoid expensive disk syncs while preparing the wallet. Re-enable syncs
+    // below so that the measured loading uses the normal database settings.
+    options.use_unsafe_sync = true;
     bilingual_str error;
     auto database = MakeWalletDatabase("", options, status, error);
     auto wallet = TestCreateWallet(std::move(database), context, create_flags);
@@ -61,6 +64,7 @@ static void WalletLoadingDescriptors(benchmark::Bench& bench)
 
     options.require_create = false;
     options.require_existing = true;
+    options.use_unsafe_sync = false;
 
     bench.epochs(5)
         .setup([&] {

</details>

<details> <summary> Commands used</summary>

# Direct sanity check
./build-investigation/bin/bench_bitcoin \
  -filter=WalletLoadingDescriptors -sanity-check

# Functional-test wrapper
python3 ./test/functional/tool_bench_sanity_check.py \
  --configfile=./build-investigation/test/config.ini \
  --bench=WalletLoadingDescriptors

# Regular benchmark
./build-investigation/bin/bench_bitcoin \
  -filter=WalletLoadingDescriptors \
  -output-csv=/tmp/bitcoin-fixed-regular.csv

</details>

It would be useful to run this patch on the original reporter’s machine and the affected CI runners to confirm the improvement there.

Can you bisect for when this started happening?

Apparently before/after #35018

Ok, so the issue only happens when the tmpdir is on slow storage (like CI machines or macOS VMs)?