fuzz: add a custom mutator for PSBT #36631

pull brunoerg wants to merge 2 commits into bitcoin:master from brunoerg:2026-10-fuzz-psbt-custom-mutator changing 7 files +573 −3
  1. brunoerg commented at 7:12 PM on October 23, 2026: contributor

    The partially_signed_transaction_deserialize target feeds raw bytes to the PSBT parser. Byte-level mutations rarely produce well-formed key-value maps with valid keydata (pubkeys, xpubs, control blocks, DER signatures, etc), so most inputs are rejected early and the deeper per-field parsing is rarely reached.

    This PR adds a custom mutator for PSBT which is ported from bitcoinfuzz (with some changes). The mutator splits the input into global/input/output maps of key-value records, applies the mutations and serializes the result again. The first commit is a preparation to allow targets to provide a custom mutator by adding an optional mutator field to FuzzTargetOptions.

    note: LLM assisted on porting the custom mutator.

  2. DrahtBot added the label Fuzzing on Oct 23, 2026
  3. DrahtBot commented at 7:12 PM on October 23, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    External sites

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process. A summary of reviews will appear here.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #36743 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36743.svg"></sub> ([RFC] build: build with -Wmissing-declarations/prototypes by fanquake)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  4. in src/test/fuzz/fuzz.cpp:237 in b44c41bd44 outdated
     232 | +}
     233 | +
     234 | +// libFuzzer disables -len_control by default whenever LLVMFuzzerCustomMutator
     235 | +// is linked in, even for targets that don't provide a mutator. Restore
     236 | +// libFuzzer's usual default for those targets, unless the user set it.
     237 | +static void restore_default_len_control(int* argc, char*** argv)
    


    brunoerg commented at 7:14 PM on October 23, 2026:

    note: since we have one binary for everything, a custom mutator would disable -len_control even for targets that do not use it. This function solves it but libFuzzer will still log the disabling.


    ekzyis commented at 1:21 AM on October 30, 2026:

    I think this will break fuzzing with afl++ in OSS-Fuzz. It defines LLVMFuzzerMutate as a symbol here and then crashes here because of argv manipulation. You can reproduce with this:

    $ cmake -B build_fuzz_afl -DBUILD_FOR_FUZZING=ON -DCMAKE_C_COMPILER=afl-clang-lto -DCMAKE_CXX_COMPILER=afl-clang-lto++ -DSANITIZERS=fuzzer
    $ cmake --build build_fuzz_afl -j$(nproc)
    $ FUZZ=process_message ./build_fuzz_afl/bin/fuzz
    ============================== INFO ================================
    This binary is built for afl++.
    To run the target function on individual input(s) execute:
      ./build_fuzz_afl/bin/fuzz INPUT_FILE1 [INPUT_FILE2 ... ]
    To fuzz with afl-fuzz execute:
      afl-fuzz [afl-flags] -- ./build_fuzz_afl/bin/fuzz [-N]
    afl-fuzz will run N iterations before re-spawning the process (default: INT_MAX)
    You can also use AFL_FUZZER_LOOPCOUNT to set N
    For stdin input processing, pass '-' as single command line option.
    For file input processing, pass '@@' as single command line option.
    To use with afl-cmin or afl-cmin.bash pass '-' as single command line option
    ===================================================================
    Running LLVMFuzzerInitialize ...
    continue...
    fuzz: aflpp_driver.c:422: int LLVMFuzzerRunDriver(int *, char ***, int (*)(const uint8_t *, size_t)): Assertion `N > 0' failed.
    Aborted                    (core dumped) FUZZ=process_message ./build_fuzz_afl/bin/fuzz
    

    This doesn't happen on master (6ed5aad061).


    brunoerg commented at 1:24 AM on October 30, 2026:

    Yea, noticed this, I had tested only on libFuzzer. I've moved this to draft while fixing it.


    brunoerg commented at 1:16 PM on November 2, 2026:

    Done.

  5. brunoerg marked this as a draft on Oct 23, 2026
  6. brunoerg force-pushed on Oct 23, 2026
  7. DrahtBot added the label CI failed on Oct 23, 2026
  8. DrahtBot commented at 7:25 PM on October 23, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task NetBSD Cross: https://github.com/bitcoin/bitcoin/actions/runs/40009584622/job/120606204787</sub> <sub>LLM reason (✨ experimental): CI failed during the fuzz-test build because g_custom_mutator in src/test/fuzz/fuzz.cpp is set but unused, triggering -Werror.</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  9. brunoerg force-pushed on Oct 23, 2026
  10. DrahtBot removed the label CI failed on Oct 23, 2026
  11. brunoerg marked this as ready for review on Oct 26, 2026
  12. fametrano referenced this in commit 1465bc70b6 on Oct 27, 2026
  13. fametrano referenced this in commit 4a0f89f4a6 on Oct 27, 2026
  14. fametrano referenced this in commit b34fca6f94 on Oct 27, 2026
  15. fametrano referenced this in commit 9fc770670c on Oct 27, 2026
  16. fametrano referenced this in commit e666d9ea05 on Oct 27, 2026
  17. fametrano referenced this in commit 7de40680c2 on Oct 27, 2026
  18. fametrano referenced this in commit e3f7200e6d on Oct 27, 2026
  19. fametrano referenced this in commit 8d4608d1a9 on Oct 27, 2026
  20. DrahtBot added the label Needs rebase on Oct 28, 2026
  21. brunoerg force-pushed on Oct 28, 2026
  22. brunoerg commented at 6:14 PM on October 28, 2026: contributor

    Rebased

  23. DrahtBot removed the label Needs rebase on Oct 28, 2026
  24. in src/test/fuzz/util/psbt.cpp:429 in 3f6be480ed outdated
     424 | +        Record& rec{map[rng.randrange(map.size())]};
     425 | +        if (rec.key.empty()) break;
     426 | +        // Leave room to grow, so keydata of other lengths can be reached.
     427 | +        const size_t old_size{rec.key.size()};
     428 | +        rec.key.resize(old_size + 16);
     429 | +        rec.key.resize(FuzzDefaultMutate(rec.key.data(), old_size, rec.key.size()));
    


    ekzyis commented at 1:22 AM on October 30, 2026:

    The key gets erased if FuzzDefaultMutate returns 0 (which is the case for LLVMFuzzerMutate in afl++) . Is that expected to happen in this branch for key mutations?


    brunoerg commented at 5:14 PM on October 30, 2026:

    this would only happen with AFL++'s weak stub when PROVIDE_FUZZ_CUSTOM_MUTATOR is defined, isn't it?


    ekzyis commented at 4:38 PM on October 31, 2026:

    Ah yes, that is my understanding as well after seeing #if defined(PROVIDE_FUZZ_CUSTOM_MUTATOR) at fuzz.cpp:224.

  25. brunoerg marked this as a draft on Oct 30, 2026
  26. brunoerg force-pushed on Nov 2, 2026
  27. brunoerg marked this as ready for review on Nov 2, 2026
  28. in src/test/fuzz/fuzz.cpp:243 in 1d9624833b outdated
     238 | +// libFuzzer disables -len_control by default whenever LLVMFuzzerCustomMutator
     239 | +// is linked in, even for targets that don't provide a mutator. Restore
     240 | +// libFuzzer's usual default for those targets, unless the user set it.
     241 | +static void restore_default_len_control(int* argc, char*** argv)
     242 | +{
     243 | +    if (g_custom_mutator) return;
    


    ekzyis commented at 8:20 PM on November 3, 2026:

    nit: could move this check to the call site. I consider it control flow rather than part of this function's logic.


    brunoerg commented at 2:38 PM on November 4, 2026:

    In case I'm going to touch it again, I can address it.

  29. in CMakeLists.txt:449 in 7e7cbbafe2
     444 | @@ -445,6 +445,21 @@ if(BUILD_FUZZ_BINARY)
     445 |      LDFLAGS ${SANITIZER_LDFLAGS} ${fuzz_flag}
     446 |      LINK_LIBRARIES ${FUZZ_LIBS}
     447 |    )
     448 | +  # Custom mutators call back into the engine's LLVMFuzzerMutate, which only
     449 | +  # libFuzzer provides. AFL++'s driver links a weak stub of it, and doesn't
    


    ekzyis commented at 8:53 PM on November 3, 2026:

    nit: "which only libFuzzer provides" is inaccurate. Suggesting "provided by libFuzzer" instead:

    diff --git a/CMakeLists.txt b/CMakeLists.txt
    index 4381f04c3a..5acf868691 100644
    --- a/CMakeLists.txt
    +++ b/CMakeLists.txt
    @@ -445,9 +445,9 @@ if(BUILD_FUZZ_BINARY)
         LDFLAGS ${SANITIZER_LDFLAGS} ${fuzz_flag}
         LINK_LIBRARIES ${FUZZ_LIBS}
       )
    -  # Custom mutators call back into the engine's LLVMFuzzerMutate, which only
    -  # libFuzzer provides. AFL++'s driver links a weak stub of it, and doesn't
    -  # accept libFuzzer flags, so exclude it explicitly.
    +  # Custom mutators call back into the engine's LLVMFuzzerMutate provided by
    +  # libFuzzer. AFL++'s driver links a weak stub of it, and doesn't accept
    +  # libFuzzer flags, so exclude it explicitly.
       check_cxx_source_compiles_with_flags("
           #ifdef __AFL_COMPILER
           #error AFL++ does not support LLVMFuzzerMutate
    

    brunoerg commented at 2:37 PM on November 4, 2026:

    Done.

  30. in CMakeLists.txt:454 in 7e7cbbafe2
     449 | +  # libFuzzer provides. AFL++'s driver links a weak stub of it, and doesn't
     450 | +  # accept libFuzzer flags, so exclude it explicitly.
     451 | +  check_cxx_source_compiles_with_flags("
     452 | +      #ifdef __AFL_COMPILER
     453 | +      #error AFL++ does not support LLVMFuzzerMutate
     454 | +      #endif
    


    ekzyis commented at 8:57 PM on November 3, 2026:

    I don't know how much we care about honggfuzz, but honggfuzz has a similar problem. It also defines LLVMFuzzerMutate and throws a warning because of -len_control=100 (no crash like AFL++). See reproduction which simulates appending -len_control=100 to argv like restore_default_len_control() would do:

    $ hfuzz-clang++ -std=c++20 argv.cpp -o argv
    $ echo hello > seed
    $ ./argv < in/seed
    [2026-11-03T15:07:11-0600][W][31466] HonggfuzzRunFromFile():80 Cannot open '-len_control=100' as input, using stdin: No such file or directory
    Accepting input from '[STDIN]'
    Usage for fuzzing: honggfuzz -P [flags] -- ./argv
    TestOneInput size=6
    
    // argv.cpp
    #include <cstdint>
    #include <cstddef>
    #include <cstdio>
    #include <vector>
    extern "C" int LLVMFuzzerInitialize(int* argc, char*** argv)
    {
        static char len_control_arg[]{"-len_control=100"};
        static std::vector<char*> g_argv;
        g_argv.assign(*argv, *argv + *argc);
        g_argv.insert(g_argv.begin() + 1, len_control_arg);
        g_argv.push_back(nullptr);
        *argc = g_argv.size() - 1;
        *argv = g_argv.data();
        return 0;
    }
    extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
    {
        printf("TestOneInput size=%zu\n", size);
        return 0;
    }
    

    Claude suggested a configure-time opt-in. Then we wouldn't need to detect the engine and rewrite arguments, but the custom mutator would be off by default and I think if there's a custom mutator available, we never want to run without it. Sounds like a good trade-off to me, though.


    brunoerg commented at 2:27 PM on November 4, 2026:

    Well noticed, I don't use honggfuzz but since we support it, I should consider it. I would not go with a configure-time opt-in because it would require changes on some infra, including OSS-Fuzz.


    brunoerg commented at 2:38 PM on November 4, 2026:

    Done. I just changed it to detect the real libFuzzer runtime instead of LLVMFuzzerMutate.

  31. ekzyis commented at 9:27 PM on November 3, 2026: contributor

    Partial review. Left some nits and a remark wrt honggfuzz + suggestion from Claude. Will look into mutator effectiveness next.

  32. fuzz: allow targets to provide a libFuzzer custom mutator
    Add an optional `mutator` field to FuzzTargetOptions. When the fuzz
    binary is linked against libFuzzer, LLVMFuzzerCustomMutator dispatches
    to the selected target's mutator, falling back to LLVMFuzzerMutate for
    targets that don't set one.
    
    Custom mutators can call FuzzDefaultMutate() to apply libFuzzer's
    default mutations to parts of the input.
    
    libFuzzer disables -len_control by default whenever
    LLVMFuzzerCustomMutator is linked in, which would otherwise affect every
    target. Targets without a custom mutator get -len_control=100 restored,
    unless it was set explicitly.
    
    This is only enabled when the real libFuzzer runtime is linked, detected
    at configure time, so other engines are unaffected.
    8c7e8786f6
  33. fuzz: add structure-aware PSBT mutator
    Add a custom mutator for raw PSBTs, ported from bitcoinfuzz, and use it
    for the partially_signed_transaction_deserialize target.
    93dc3cfe09
  34. brunoerg force-pushed on Nov 4, 2026
  35. brunoerg commented at 2:40 PM on November 4, 2026: contributor

    Force-pushed:

  36. fametrano referenced this in commit 4e577b6e10 on Nov 7, 2026
  37. fametrano referenced this in commit f803ec8bd2 on Nov 7, 2026
Labels

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-11-21 10:51 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me