[31.x] Backports #35331

pull fanquake wants to merge 26 commits into bitcoin:31.x from fanquake:more_31_x_backports changing 46 files +860 −324
  2. psbt, test: remove address type restrictions in test
    Because the corresponding Taproot fields were added in PSBT in PR 22558, so
these restrictions are no longer necessary.

Github-Pull: #35279
Rebased-From: 81348576cc411a63da295a10a3846302419a18e0
    101071722e
  3. DrahtBot added the label Backport on May 20, 2026
  4. fanquake added this to the milestone 31.1 on May 20, 2026
  5. DrahtBot commented at 1:03 PM on May 20, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35331.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK sedited, marcofleon

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

    LLM Linter (✨ experimental)

    Possible typos and grammar issues:

    • use the proxy if overriden -> use the proxy if overridden [misspelling of “overridden”]

    Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):

    • [MaxInputWeight(*desc, CTxIn{outpoint}, coin_control, true, can_grind_r)] in [src/wallet/spend.cpp]

    <sup>2026-06-22 11:06:46</sup>

  6. wallet: use outpoint when estimating input size
    `CalculateMaximumSignedInputSize()` is passed the outpoint being sized, but that context was not used when estimating the signed input size.
Pass the outpoint through so externally selected inputs are not underestimated.

Co-authored-by: Antoine Poinsot <darosior@protonmail.com>

Github-Pull: #35228
Rebased-From: cd8d3bd937b5515ea000408eb07d2ae3cd1aa417
    671e6c2c33
  7. fanquake force-pushed on May 20, 2026
  8. DrahtBot added the label CI failed on May 20, 2026
  9. DrahtBot removed the label CI failed on May 20, 2026
  10. musig: Reject empty pubkey list in GetMuSig2KeyAggCache
    Github-Pull: #35316
Rebased-From: 8ce84321ceaf16c0ee3418d30011c357fdc46deb
    d61687a2ac
  11. fanquake force-pushed on May 22, 2026
  12. DrahtBot added the label CI failed on May 22, 2026
  13. ci: switch to GitHub cache for all runners
    Cirrus is winding down and github now offers more than 10GB cache.

Switch to GH cache for all runner-types. Simplify docker build arg
construction, and reduce the number of needed action permissions.

Github-Pull: #35348
Rebased-From: c03107acf50aeada1bacdfdbf632ffce2957bff0
    3440027b7d
  14. fanquake force-pushed on May 22, 2026
  15. DrahtBot removed the label CI failed on May 22, 2026
  16. fanquake force-pushed on May 28, 2026
  17. DrahtBot added the label CI failed on May 28, 2026
  18. DrahtBot commented at 4:05 PM on May 28, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task lint: https://github.com/bitcoin/bitcoin/actions/runs/26580148981/job/78317916896</sub> <sub>LLM reason (✨ experimental): CI failed the lint check “subtree” because a subtree directory was modified without a corresponding subtree merge.</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  19. fanquake force-pushed on May 28, 2026
  20. fanquake force-pushed on May 29, 2026
  21. fanquake force-pushed on May 29, 2026
  22. fanquake force-pushed on Jun 2, 2026
  23. fanquake force-pushed on Jun 4, 2026
  24. fanquake force-pushed on Jun 4, 2026
  25. fanquake marked this as ready for review on Jun 12, 2026
  26. fanquake requested review from marcofleon on Jun 12, 2026
  27. DrahtBot commented at 8:36 AM on June 15, 2026: contributor

    Maybe turn into draft while CI is red?

  28. fanquake commented at 11:31 AM on June 15, 2026: member

    That is expected. This is reviewable.

  29. maflcko commented at 7:09 PM on June 16, 2026: member

    Ok, fair enough. The lint CI will continue to fail, going forward on this branch then. It could make sense to disable the subtree lint, if the folder is no longer a subtree. Maybe like:

    diff --git a/test/lint/test_runner/src/util.rs b/test/lint/test_runner/src/util.rs
index 11e332e..059156f 100644
--- a/test/lint/test_runner/src/util.rs
+++ b/test/lint/test_runner/src/util.rs
@@ -68,3 +68,2 @@ pub fn get_subtrees() -> Vec<&'static str> {
         "src/ipc/libmultiprocess",
-        "src/leveldb",
         "src/minisketch",
@@ -79,2 +78,3 @@ pub fn get_pathspecs_default_excludes() -> Vec<String> {
         .chain(&[
+            "src/leveldb", // No longer a subtree in this release branch, due to direct cherry-picks
             "doc/release-notes/release-notes-*", // archived notes

    (untested, so feel free to ignore)

  30. fanquake force-pushed on Jun 17, 2026
  31. fanquake commented at 6:03 AM on June 17, 2026: member

    Sure. Pushed up a diff that disables the lint for now. If we end up merging https://github.com/bitcoin-core/leveldb-subtree/pull/63, I think we could just do a proper pull of the subtree.

  32. l0rinc commented at 7:58 AM on June 17, 2026: contributor

    I agree that we need https://github.com/bitcoin-core/leveldb-subtree/pull/61 backports - could we consider manual compaction backports as well - #35465 (needs reviewers and reproducers) - without which I personally wouldn't suggest backporting the seek compaction removal. Maybe it would be simpler to backport the change to expand the hourly flush to ~5 hourly and keep the seek compaction for now (or rather to bump the seek compaction pressure and keep hourly flushes). The manual full compaction behaves slightly differently, we're still testing it, I'm not yet comfortable backporting it without testing it for a few weeks/months. What do you think @andrewtoth, @sipa?

  33. DrahtBot removed the label CI failed on Jun 17, 2026
  34. andrewtoth commented at 1:51 PM on June 17, 2026: contributor

    #35465 would be nice to get in for this point release. I don't think we definitely need it though. We can write a release note like we have in 29/30 point releases #35452 (comment).

  35. l0rinc commented at 10:20 PM on June 17, 2026: contributor

    We can write a release note like we have in 29/30 point releases #35452 (comment).

    It states:

    It should be removed on subsequent restarts. Leaving it set will cause significant disk reads and writes every time the node is restarted.

    What about nodes that upgrade and don't touch their nodes for several years? If we're solving compaction by asking users to do it manually (which I don't think is enough), shouldn't we mention that they should compact regularly instead of just after updating?

  36. fanquake force-pushed on Jun 19, 2026
  37. marcofleon commented at 4:34 PM on June 19, 2026: contributor

    ACK 26c6d97e1f074ff1dc359825c984320b774ec62a

    Reviewed each commit, but didn't do any real testing of the private broadcast fix on this branch.

    tiny nit: 7c0fa17a686a4c582de901a368a0de2238a6f194 is missing the Github-Pull and Rebased-From info.

  38. Disable seek compaction
    Seek compaction is causing a cascade effect in the chainstate DB, causing large parts of the database to be rewritten every ~hour.

Every periodic flush writes around 2 MiB. Since this is roughly the `write_buffer_size`, these writes regularly cause the memtable to rotate into a small L0 file. This file has a small seek budget, and with the random UTXO reads done during validation, it can get scheduled for seek compaction quickly.

That seek compaction pushes the small file down to L1. Since most UTXOs are already lower down in L4/L5, many reads that consult this file do not find the key there and continue downward. The bloom filter makes those misses cheap, but LevelDB still decrements the file's seek budget. The file then gets scheduled for another seek compaction, and the same pattern pushes it down through L2 and L3.

The expensive part happens around L3/L4. L4 has many ~32 MiB files holding the bulk of the UTXO set. When LevelDB compacts into L3, it may split the output into many smaller L3 files to limit how much L4 "grandparent" data any one output overlaps. Each of these small L3 files then gets its own small seek budget. Because chainstate keys are hash-random, each small L3 file can still have a broad key range, so many random reads consult it and quickly drain its budget. Once seek-compacted into L4, each tiny L3 file can overlap many L4 files, so compacting a few hundred KiB from L3 can require rewriting hundreds of MiB from L4. Repeating that across many small L3 files can rewrite most of the chainstate.

This is a poor fit for chainstate because UTXO keys are hash-random, the DB is large enough to have many levels, writes are relatively small and periodic, and reads are frequent. The result is that read misses trigger compactions much earlier than size pressure would, and those compactions have very high write amplification.

Disabling seek compaction may leave more files in upper levels for longer, so reads could theoretically consult more files. But Bitcoin Core enables bloom filters for all its LevelDB instances, so these misses are usually cheap in-memory filter checks rather than disk reads.

For the other DBs, the risk is much smaller. They also use bloom filters, and most are smaller and less read-heavy. With fewer levels and less random read pressure, disabling seek compaction should have little effect there.

Co-authored-by: l0rinc <pap.lorinc@gmail.com>

Github-Pull: #35313
Rebased-From: 6bfdb6093bba4710d0f8313ed0113967a8b5176f
    78714f6d4f
  39. ci: switch runners from cirrus to warpbuild
    Github-Pull: #35378
Rebased-From: 4bdd46ace37f02da062a53a2943caeddca4ed8f9
    6caf6de0a1
  40. crypto: disable ASan instrumentation of SSE4 SHA256 for GCC
    The existing Clang-only no_sanitize("address") guard is extended to
also cover GCC.  When GCC compiles this file with -fsanitize=address
in debug builds, the instrumented inline assembly causes a SEGV during
SHA256AutoDetect()'s self-test on CPUs that use the SSE4 code path
(i.e. those without SHA-NI support), regardless of optimization level.

The original Clang code placed the attribute between the function
declarator and the opening brace.  GCC's Attribute Syntax
documentation notes that this position in a function definition
"may, in future, be permitted," so it is not currently supported.
The attribute is moved to the start of the function definition,
which is valid form for both GCC and Clang.

The preprocessor guards are restructured so each compiler branch is
explicit: __clang__ with __has_feature, and __GNUC__ with
__SANITIZE_ADDRESS__.

Github-Pull: #34953
Rebased-From: fedeff7f201df0206eefb744ad125e44a63a3ea0
    8f13bb1ea0
  41. doc: remove reference to cirrus
    Github-Pull: #35408
Rebased-From: 265563bf75c0b8b615e28d47398098020ff0b109
    1f55b3e463
  42. ci: use ubuntu-latest instead of ubuntu-24.04
    To match the usage of -latest for the warp runners.

Github-Pull: #35408
Rebased-From: 5700a61b73342b506b0114b342499da7642c1c10
    39f8e077c8
  43. ci: Add dynamic cache switching to warp cache
    The GHA cache is very slow, taking on the order of minutes to save and
restore from.

Use WarpBuild's cache instead as this is in the same region and much
faster.

WarpBuild cache action does not auto-fallback to GHA if not being run on
Warp. To allow fork runs to fallback to GHA caching, whilst minimising
duplication in the action files, create new "interal" actions which
perform the switching logic, and use these in the (renamed) cache|save
actions.

Without this we would need the `if` logic in our prvious actions, 4
times in each of save and restore.

Plumb the provider through into the action, as a composite action can't
read `env` (`GITHUB_OUTPUT`) from previous steps.

Github-Pull: #35430
Rebased-From: 2ce4ae7d8f006959cb83be93a17b8125e953c30a
    25506ed6d9
  44. ci: use Warp cache for Docker layers
    Speeds of 1MB/s and 15 minute cached docker image pulls during builds
are not uncommon.

Warp runners provide a local GitHub Actions cache protocol proxy for
Docker layer cache traffic. Point BuildKit's gha cache backend at that
proxy on Warp runners so cached image layers do not have to be fetched
from GitHub's slower cache service.

Add a default for provider so other users (e.g. qa-assets) don't have to
update this unless they use custome runners.

Github-Pull: #35447
Rebased-From: 82901981bfbaeb3f41956597e6d90daa59a0c078
    70000a560b
  45. net: use the proxy if overriden when doing v2->v1 reconnections
    `OpenNetworkConnection()` supports overriding the proxy to use for
connecting. However when v2 connection is attempted and it fails a v1
connection is tried without that proxy.

Store the override proxy in `CNode` and pass it to
`CConnman::m_reconnections` to be used for v1 retries.

Github-Pull: #35410
Rebased-From: fd230f942d8377cafcc346f1605d5aa00db2cc40
    2c7986b3ee
  46. net: ensure no direct private broadcast connections
    Private broadcast connections use either Tor or I2P, which require a
proxy intrinsically or IPv4 or IPv6 which must use a proxy in the
context of private broadcast to avoid leaking the originator's IP
address.

Add a safety check to guard against future mistakes.

Co-authored-by: Andrew Toth <andrewstoth@gmail.com>

Github-Pull: #35410
Rebased-From: d01b461f71e255dcb5d7d0ee84a560f26cfe0b6f
    66377c3c84
  47. test: make reusable SOCKS5 server starting
    Extract the part of `p2p_private_broadcast.py` that configures and
starts the SOCKS5 server into a reusable function and put it into
`test_framework/socks5.py`.

Use bind port 0 to let the OS pick an available port instead of
hackishly assuming that `p2p_port(N)` is available where N is more
than the number of the nodes the test uses.

Github-Pull: #35410
Rebased-From: 2ffa81fac40fd4d03bdcfd37479a6f3305afb483
    ef20249568
  48. test: make reusable starting a standalone P2P listener
    Extract the part of `p2p_private_broadcast.py` that starts
listening on a `P2PConnection` object (or its children classes)
and put it into `test_framework/p2p.py`.

Github-Pull: #35410
Rebased-From: 2333be9cbc58dd2a533c9bd604b9db593b1d96a7
    70a8687d9d
  49. test: make reusable filling of a node's addrman
    Extract the part of `p2p_private_broadcast.py` that fills a given node's
addrman and put it into `test_framework/test_framework.py`.

Github-Pull: #35410
Rebased-From: ab35a028eded00008841a3ec8662823090bfc68b
    6c08cb7323
  50. test: add a regression test for private broadcast v1 retries
    Github-Pull: #35410
Rebased-From: 5a3756d150fc095d0038592c6338225db48fd4b5
    13df77d13b
  51. net: un-default the OpenNetworkConnection()'s proxy_override argument
    This way callers will not forget to set it.

Github-Pull: #35410
Rebased-From: bf0d257c11bebc42b88b7e96368ed2be48bc0ad1
    ccb99122f9
  52. lint: disable leveldb subtree check
    This is no-longer a proper subtree, because of direct cherry-picks.
    1ee11d8ba6
  53. util: Check write failures before renaming settings.json
    In WriteSettings(), verify that writing to the stream and closing it
succeeded before returning true. This prevents RenameOver() from replacing
a valid settings.json with a corrupted or zero-byte file when write limits
or a full disk are encountered.

Additionally, update the ReadSettings() parse failure message to mention
power loss, full disk, or storage error as possible causes.

Fixes #35373

Github-Pull: #35384
Rebased-From: 0654511e1b935d22fe6c88e3f5d5c7b336516222
    ca00827fab
  54. fanquake force-pushed on Jun 19, 2026
  55. marcofleon commented at 6:27 PM on June 19, 2026: contributor

    reACK bc371bbc35a588aa462bcacbfb7b9cd200d47109

  56. coins: test chainstate flush baseline
    Add `CDBWrapper::GetProperty()` and expose it through `CCoinsViewDB::GetDBProperty()` so coins tests can inspect LevelDB runtime properties through the coins view.
Use it in a coins DB flush baseline that records the LevelDB layout after flushing while keeping readback coverage for the flushed coin and best block.

Co-authored-by: Andrew Toth <andrewstoth@gmail.com>

Github-Pull: #35465
Rebased-From: b10889d10752c5d5e4954af2959f7bdff47bd67c
    fef6c8a4f2
  57. validation: randomly compact chainstate
    Full chainstate flushes are convenient maintenance points for long-term LevelDB cleanup because the chainstate was just written.
Randomize the trigger so nodes that flush near the same height do not compact together.

Add blocking chainstate compaction through `CCoinsViewDB::CompactFull()` and give each post-IBD full flush on the normal chainstate a 1/320 chance to start compaction.
With hourly flushes this averages roughly every two weeks and makes a six-month miss about one in a million.
This keeps the schedule stateless and leaves last-compaction height or timestamp bookkeeping out of chainstate metadata.

Co-authored-by: Andrew Toth <andrewstoth@gmail.com>

Github-Pull: #35465
Rebased-From: aa021b26f39fd231b2a3aac5780d5113a4aea639
    711065a3b9
  58. coins: compact chainstate in background
    Full chainstate compaction can take minutes on large databases.
Move `CCoinsViewDB::CompactFull()` to a named `utxocompact` one-shot background thread so validation only schedules the work.

When validation selects compaction after a full flush, the chainstate was just written and another write is less likely to be needed immediately.
The coins view destructor waits for completion, and a mutex prevents compaction from using `m_db` while `ResizeCache()` replaces it.

Co-authored-by: Andrew Toth <andrewstoth@gmail.com>

Github-Pull: #35465
Rebased-From: 394e473d42ba1383dfec45a3eafa8a73a09dbe8b
    ea3b318d8d
  59. fanquake force-pushed on Jun 22, 2026
  60. sedited approved
  61. sedited commented at 10:39 AM on June 22, 2026: contributor

    Re-created ACK ea6c629845a894af3e695b9c23c209f13ded4671

    I think Eugene is missing in the credits.

  62. DrahtBot requested review from marcofleon on Jun 22, 2026
  63. fanquake force-pushed on Jun 22, 2026
  64. DrahtBot added the label CI failed on Jun 22, 2026
  65. doc: update release notes for v31.1rc1 d813722ef2
  66. build: bump version to v31.1rc1 1c9d24fddd
  67. doc: update manual pages for v31.1rc1 c058c29831
  68. fanquake force-pushed on Jun 22, 2026
  69. sedited approved
  70. sedited commented at 11:17 AM on June 22, 2026: contributor

    ACK c058c29831a930f966180c38dfeab4635b8b86fc

  71. marcofleon commented at 11:39 AM on June 22, 2026: contributor

    ACK c058c29831a930f966180c38dfeab4635b8b86fc

  72. DrahtBot removed the label CI failed on Jun 22, 2026
  73. sedited merged this on Jun 22, 2026
  74. sedited closed this on Jun 22, 2026
  75. fanquake deleted the branch on Jun 22, 2026
  76. l0rinc commented at 7:49 PM on June 23, 2026: contributor

    ACK c058c29831a930f966180c38dfeab4635b8b86fc

Contributors
fanquakeDrahtBotmaflckol0rincandrewtothmarcofleonsedited


marcofleon

Labels
Backport

Milestone
31.1

Linked (view graph)
#34953 crypto: disable ASan instrumentation of SSE4 SHA256 for GCC (matching Clang)#35228 wallet: use `outpoint` when estimating input size#35279 psbt, test: remove address type restrictions in test#35313 Bump leveldb subtree#35316 musig: Reject empty pubkey list in GetMuSig2KeyAggCache#35348 ci: switch to GitHub cache for all runners#35378 ci: switch to warp runners#35384 util: Check write failures before renaming settings.json#35408 ci: 35378 followups#35410 net: use the proxy if overriden when doing v2->v1 reconnections#35430 ci: use warp caching on warp runners#35447 ci: use warpbuild cache for docker buildkit cache#35465 coins: compact chainstate regularly
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-07-01 06:51 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me