Minimize with honggfuzz #301

pull erickcestari wants to merge 2 commits into bitcoin-core:main from erickcestari:minimize-with-honggfuzz changing 2 files +297 −47
  1. erickcestari commented at 9:14 PM on November 11, 2026: contributor

    Engines measure coverage differently, so minimizing without honggfuzz drops inputs only it needs. Prefixes honggfuzz keeps are mapped back to the inputs they come from, and minimizer timeouts match libFuzzer's 1200s default.

    An input stays in the new corpus if afl++, hongfuzz or libfuzzer finds that it reaches coverage the other kept inputs don't. It's dropped only when none of them do.

    Closes #267

  2. delete-nonreduced-fuzz-inputs: Use libFuzzer's timeout in afl-cmin
    A timed out input is only judged on its partial coverage, so afl-cmin gets
    the same 1200s per-input timeout as the libFuzzer merge.
    422da42ccc
  3. delete-nonreduced-fuzz-inputs: Minimize with honggfuzz
    Engines measure coverage differently, so minimizing without honggfuzz drops
    inputs only it needs. Prefixes honggfuzz keeps are mapped back to the inputs
    they come from.
    
    Closes #267
    c3d50257d0
  4. erickcestari commented at 9:27 PM on November 11, 2026: contributor

    Ran the full minimizer from this branch inside a ubuntu docker container on a 16-thread laptop, against Bitcoin Core master at 6f1365e. It took about 4.5h:

    Step Duration Files changed
    afl-cmin 13 min 82,439
    honggfuzz 2h10 42,787
    libFuzzer fuzzer 20 min 5,384
    libFuzzer fuzzer,address,undefined,integer 1h46 26,335

    Result: 218,114 inputs (3.5G) went down to 156,945 (2.6G).


    I replayed the old and new corpus for every target with the fuzzer,address,undefined,integer build (-runs=0) and compared the INITED cov numbers:

    • No crashes or timeouts on either corpus.
    • cov is the same on 137 targets, higher on the new corpus on 41 and lower on 59.
    • policy_estimator lost all its inputs, which is expected since it was renamed to block_policy_estimator on master.

    Many targets aren't deterministic, so I reran the 12 targets with the biggest drops two more times on each corpus (3 runs each):

    Target Old cov New cov
    txgraph 12860 to 12917 12823 to 12824 lower
    p2p_headers_presync 7584 to 7594 7553 to 7561 lower
    spkm_migration 18659 to 18683 18644 to 18657 lower
    dbwrapper 9303 to 9340 9290 to 9299 lower
    coinscache_sim, coins_view_overlay, dbwrapper_threaded, load_external_block_file, p2p_private_broadcast, threadpool, tx_pool, utxo_snapshot ranges overlap, noise

    The last libFuzzer pass uses the same build as this check, so it should already have kept any input needed for these edges. That means the four real drops come from coverage that isn't deterministic, or that depends on state built up across inputs in one process. They aren't caused by how the engines are combined. They are small, about 10 to 90 edges on targets with 7k to 19k.

    Obs: honggfuzz is the slowest step because its build uses sanitizers (address,undefined), like the second libFuzzer pass, which is about as slow. honggfuzz --minimize is also single-threaded, so targets run in parallel one per core and the step only ends when the slowest target does. Starting the targets with the most inputs first (e.g. descriptor_parse, wallet_create_transaction, tx_package_eval) would let them overlap with the rest instead of running alone at the end, which shortens the tail without changing any coverage.

    <details> <summary>Full per-target results</summary>

    targets done: 238
    identical: 137  higher: 41  lower: 59  errors/missing: 1
    LOWER txgraph: cov 12917->12823 (-94) ft 86800->86343 (-457)
    LOWER p2p_headers_presync: cov 7591->7553 (-38) ft 27247->27181 (-66)
    LOWER descriptor_parse: cov 26302->26271 (-31) ft 182877->182722 (-155)
    LOWER threadpool: cov 741->712 (-29) ft 3481->3406 (-75)
    LOWER load_external_block_file: cov 2672->2653 (-19) ft 17506->17455 (-51)
    LOWER coinscache_sim: cov 4519->4502 (-17) ft 31201->31045 (-156)
    LOWER dbwrapper_threaded: cov 9700->9685 (-15) ft 63116->62907 (-209)
    LOWER spkm_migration: cov 18659->18644 (-15) ft 109858->109778 (-80)
    LOWER coins_view_overlay: cov 11630->11615 (-15) ft 64107->64049 (-58)
    LOWER tx_package_eval: cov 19434->19420 (-14) ft 133258->133146 (-112)
    LOWER dbwrapper: cov 9303->9290 (-13) ft 60652->60577 (-75)
    LOWER p2p_private_broadcast: cov 18269->18257 (-12) ft 92392->92370 (-22)
    LOWER tx_pool: cov 31558->31548 (-10) ft 209898->209772 (-126)
    LOWER utxo_snapshot: cov 13680->13670 (-10) ft 39494->39408 (-86)
    LOWER connman: cov 22868->22860 (-8) ft 160095->160256 (+161)
    LOWER process_messages: cov 30530->30523 (-7) ft 131608->131577 (-31)
    LOWER txdownloadman: cov 8598->8591 (-7) ft 58408->58389 (-19)
    LOWER coins_view_db: cov 15996->15991 (-5) ft 101981->101679 (-302)
    LOWER p2p_handshake: cov 7318->7314 (-4) ft 28398->28362 (-36)
    LOWER connect_block: cov 17636->17632 (-4) ft 83760->83757 (-3)
    LOWER parse_univalue: cov 20033->20030 (-3) ft 126324->126323 (-1)
    LOWER transaction: cov 9911->9909 (-2) ft 50059->50000 (-59)
    LOWER process_message: cov 27723->27721 (-2) ft 84332->84322 (-10)
    LOWER timeoffsets: cov 773->771 (-2) ft 5504->5502 (-2)
    LOWER netaddr_deserialize: cov 615->613 (-2) ft 1024->1023 (-1)
    LOWER mini_miner: cov 8879->8878 (-1) ft 58285->58257 (-28)
    LOWER net: cov 2190->2189 (-1) ft 12133->12121 (-12)
    LOWER addrman: cov 6232->6231 (-1) ft 35326->35322 (-4)
    LOWER addrman_serdeser: cov 5708->5707 (-1) ft 33821->33818 (-3)
    LOWER signet: cov 11371->11370 (-1) ft 71235->71234 (-1)
    LOWER wallet_bdb_parser: cov 1177->1176 (-1) ft 1742->1741 (-1)
    LOWER ephemeral_package_eval: cov 18102->18102 (+0) ft 122521->122444 (-77)
    LOWER wallet_fees: cov 5339->5339 (+0) ft 7543->7466 (-77)
    LOWER txdownloadman_impl: cov 9242->9242 (+0) ft 62595->62573 (-22)
    LOWER policy_estimator_io: cov 1161->1161 (+0) ft 3505->3493 (-12)
    LOWER p2p_transport_bidirectional_v2: cov 15504->15504 (+0) ft 61767->61757 (-10)
    LOWER block_index_tree: cov 2430->2430 (+0) ft 16043->16035 (-8)
    LOWER data_stream_addr_man: cov 4691->4691 (+0) ft 18469->18462 (-7)
    LOWER p2p_transport_bidirectional_v1v2: cov 8899->8899 (+0) ft 35101->35095 (-6)
    LOWER p2p_transport_serialization: cov 1409->1409 (+0) ft 9651->9645 (-6)
    LOWER versionbits: cov 704->704 (+0) ft 2116->2111 (-5)
    LOWER secp256k1_ecdsa_signature_parse_der_lax: cov 6227->6227 (+0) ft 6616->6612 (-4)
    LOWER i2p: cov 3536->3536 (+0) ft 10219->10216 (-3)
    LOWER miniscript_smart: cov 14724->14724 (+0) ft 78024->78021 (-3)
    LOWER vecdeque: cov 7024->7024 (+0) ft 51765->51762 (-3)
    LOWER golomb_rice: cov 2704->2704 (+0) ft 6991->6989 (-2)
    LOWER base58_encode_decode: cov 307->307 (+0) ft 996->995 (-1)
    LOWER base64_encode_decode: cov 143->143 (+0) ft 360->359 (-1)
    LOWER bech32_random_decode: cov 324->324 (+0) ft 1053->1052 (-1)
    LOWER block_filter_deserialize: cov 216->216 (+0) ft 461->460 (-1)
    LOWER blockfilter: cov 3538->3538 (+0) ft 11459->11458 (-1)
    LOWER minisketch: cov 1501->1501 (+0) ft 8262->8261 (-1)
    LOWER netaddress: cov 4434->4434 (+0) ft 8221->8220 (-1)
    LOWER prefilled_transaction_deserialize: cov 600->600 (+0) ft 2611->2610 (-1)
    LOWER script_ops: cov 752->752 (+0) ft 4635->4634 (-1)
    LOWER script_sigcache: cov 10768->10768 (+0) ft 25083->25082 (-1)
    LOWER str_printf: cov 961->961 (+0) ft 3358->3357 (-1)
    LOWER string: cov 1941->1941 (+0) ft 7402->7401 (-1)
    LOWER txrequest: cov 2976->2976 (+0) ft 21887->21886 (-1)
    HIGHER bnb_finds_min_waste: cov 1704->1704 (+0) ft 8172->8173 (+1)
    HIGHER clusterlin_chunking: cov 491->491 (+0) ft 2379->2380 (+1)
    HIGHER coinselection_bnb: cov 4252->4252 (+0) ft 21727->21728 (+1)
    HIGHER headers_sync_state: cov 3124->3124 (+0) ft 10239->10240 (+1)
    HIGHER key_origin_info_deserialize: cov 213->213 (+0) ft 582->583 (+1)
    HIGHER local_address: cov 2048->2048 (+0) ft 9500->9501 (+1)
    HIGHER tx_in: cov 415->415 (+0) ft 743->744 (+1)
    HIGHER block_deserialize: cov 642->642 (+0) ft 3759->3761 (+2)
    HIGHER coinselection_srd: cov 4255->4255 (+0) ft 21819->21821 (+2)
    HIGHER crypter: cov 12979->12979 (+0) ft 90386->90388 (+2)
    HIGHER key: cov 15904->15904 (+0) ft 38341->38343 (+2)
    HIGHER pool_resource: cov 1661->1661 (+0) ft 9764->9769 (+5)
    HIGHER p2p_transport_bidirectional: cov 1798->1798 (+0) ft 11837->11903 (+66)
    HIGHER script_sign: cov 20010->20011 (+1) ft 144454->144436 (-18)
    HIGHER rolling_bloom_filter: cov 2528->2529 (+1) ft 16819->16807 (-12)
    HIGHER rbf: cov 7643->7644 (+1) ft 52512->52510 (-2)
    HIGHER addr_info_deserialize: cov 523->524 (+1) ft 870->871 (+1)
    HIGHER partially_signed_transaction_deserialize: cov 7935->7936 (+1) ft 48382->48383 (+1)
    HIGHER psbt_output_deserialize: cov 3657->3658 (+1) ft 22257->22258 (+1)
    HIGHER system: cov 2183->2184 (+1) ft 13920->13921 (+1)
    HIGHER txorphan_protected: cov 5155->5156 (+1) ft 25723->25726 (+3)
    HIGHER psbt: cov 30446->30447 (+1) ft 165287->165291 (+4)
    HIGHER rpc: cov 56630->56632 (+2) ft 225190->224954 (-236)
    HIGHER miniscript_stable: cov 14776->14778 (+2) ft 79407->79409 (+2)
    HIGHER socks5: cov 766->768 (+2) ft 2434->2436 (+2)
    HIGHER block_policy_estimator: cov 2436->2438 (+2) ft 12385->12388 (+3)
    HIGHER script: cov 8501->8506 (+5) ft 31579->31585 (+6)
    HIGHER partially_downloaded_block: cov 8145->8150 (+5) ft 53932->53964 (+32)
    HIGHER validation_load_mempool: cov 7265->7271 (+6) ft 42604->42643 (+39)
    HIGHER utxo_snapshot_invalid: cov 10278->10285 (+7) ft 30809->30727 (-82)
    HIGHER package_rbf: cov 8810->8819 (+9) ft 60262->60238 (-24)
    HIGHER ipc: cov 6382->6391 (+9) ft 44067->44140 (+73)
    HIGHER block_index: cov 4949->4961 (+12) ft 12336->12327 (-9)
    HIGHER cmpctblock: cov 32504->32519 (+15) ft 164587->164733 (+146)
    HIGHER mocked_descriptor_parse: cov 26395->26410 (+15) ft 189483->189632 (+149)
    HIGHER scriptpubkeyman: cov 38824->38844 (+20) ft 260778->260795 (+17)
    HIGHER tx_pool_standard: cov 20329->20367 (+38) ft 129303->129452 (+149)
    HIGHER wallet_create_transaction: cov 28339->28378 (+39) ft 186474->185942 (-532)
    HIGHER coins_view_stacked: cov 18543->18585 (+42) ft 110010->109877 (-133)
    HIGHER dbwrapper_concurrent_reads: cov 9305->9352 (+47) ft 49388->49388 (+0)
    HIGHER utxo_total_supply: cov 25675->25728 (+53) ft 107911->108094 (+183)
    ISSUE policy_estimator: old=('-', '-', '1') new=('-', '-', 'missing')
    

    </details>


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin-core/qa-assets. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-11-20 20:25 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me