Pruning large/slow inputs?

fanquake commented at 1:44 pm on April 19, 2023: member

Some targets currently run for > half an hour, i.e banman or addrman_deserialize, with their slowest inputs running for a minute or more. Is it worth trying to prune these slow inputs, assuming no, or a negligable decrease in coverage, if runtime can be improved by a decent percentage?

https://cirrus-ci.com/task/6079733960540160?logs=ci#L1068:

 0Run addrman_deserialize with args ['valgrind', '--quiet', '--error-exitcode=1', '/tmp/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/fuzz/fuzz', '-runs=1', '/tmp/bitcoin-core/ci/scratch/qa-assets/fuzz_seed_corpus/addrman_deserialize']INFO: Running with entropic power schedule (0xFF, 100).
 1INFO: Seed: 3456924522
 2INFO: Loaded 1 modules   (243081 inline 8-bit counters): 243081 [0x2736df8, 0x2772381), 
 3INFO: Loaded 1 PC tables (243081 PCs): 243081 [0x2772388,0x2b27c18), 
 4INFO:     3096 files found in /tmp/bitcoin-core/ci/scratch/qa-assets/fuzz_seed_corpus/addrman_deserialize
 5INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
 6INFO: seed corpus: files: 3096 min: 1b max: 1048576b total: 128207363b rss: 251Mb
 7[#64](/bitcoin-core-qa-assets/64/)	pulse  cov: 1321 ft: 1729 corp: 31/980b exec/s: 21 rss: 272Mb
 8[#128](/bitcoin-core-qa-assets/128/)	pulse  cov: 1323 ft: 1754 corp: 33/1080b exec/s: 18 rss: 273Mb
 9[#256](/bitcoin-core-qa-assets/256/)	pulse  cov: 1324 ft: 1755 corp: 34/1133b exec/s: 15 rss: 273Mb
10[#512](/bitcoin-core-qa-assets/512/)	pulse  cov: 1493 ft: 2023 corp: 68/3165b exec/s: 21 rss: 274Mb
11[#1024](/bitcoin-core-qa-assets/1024/)	pulse  cov: 2713 ft: 4888 corp: 135/9095b exec/s: 35 rss: 282Mb
12[#2048](/bitcoin-core-qa-assets/2048/)	pulse  cov: 2941 ft: 12633 corp: 547/156Kb exec/s: 23 rss: 284Mb
13Slowest unit: 12 s:
14artifact_prefix='./'; Test unit written to ./slow-unit-fa63ff08d2d73478c8e4dfb1f6d10c314c51bda2
15Slowest unit: 20 s:
16artifact_prefix='./'; Test unit written to ./slow-unit-2cf6dcdc90d27082d6f06c1bbcf2c9fb617c4665
17Slowest unit: 29 s:
18artifact_prefix='./'; Test unit written to ./slow-unit-c7043fea9ac0a00b8421f0c66b91141a47e2ce4a
19Slowest unit: 77 s:
20artifact_prefix='./'; Test unit written to ./slow-unit-c2c4bf148a02aebee1179dbb17b0f1c4ce69d949
21[#3097](/bitcoin-core-qa-assets/3097/)	INITED cov: 2960 ft: 15207 corp: 945/20Mb exec/s: 1 rss: 416Mb
22[#3097](/bitcoin-core-qa-assets/3097/)	DONE   cov: 2960 ft: 15207 corp: 945/20Mb lim: 760806 exec/s: 1 rss: 416Mb
23Done 3097 runs in 2437 second(s)

https://cirrus-ci.com/task/6079733960540160?logs=ci#L943:

 0Run banman with args ['valgrind', '--quiet', '--error-exitcode=1', '/tmp/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/fuzz/fuzz', '-runs=1', '/tmp/bitcoin-core/ci/scratch/qa-assets/fuzz_seed_corpus/banman']INFO: Running with entropic power schedule (0xFF, 100).
 1INFO: Seed: 3396028099
 2INFO: Loaded 1 modules   (243081 inline 8-bit counters): 243081 [0x2736df8, 0x2772381), 
 3INFO: Loaded 1 PC tables (243081 PCs): 243081 [0x2772388,0x2b27c18), 
 4INFO:     4783 files found in /tmp/bitcoin-core/ci/scratch/qa-assets/fuzz_seed_corpus/banman
 5INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
 6INFO: seed corpus: files: 4783 min: 1b max: 4194310b total: 221535216b rss: 254Mb
 7[#128](/bitcoin-core-qa-assets/128/)	pulse  cov: 1334 ft: 2275 corp: 46/332b exec/s: 64 rss: 276Mb
 8[#256](/bitcoin-core-qa-assets/256/)	pulse  cov: 1409 ft: 2700 corp: 84/735b exec/s: 64 rss: 276Mb
 9[#512](/bitcoin-core-qa-assets/512/)	pulse  cov: 2378 ft: 4873 corp: 157/1826b exec/s: 56 rss: 279Mb
10[#1024](/bitcoin-core-qa-assets/1024/)	pulse  cov: 2631 ft: 7553 corp: 306/5510b exec/s: 56 rss: 280Mb
11[#2048](/bitcoin-core-qa-assets/2048/)	pulse  cov: 2844 ft: 10686 corp: 585/21Kb exec/s: 51 rss: 280Mb
12Slowest unit: 10 s:
13artifact_prefix='./'; Test unit written to ./slow-unit-f80fda481ebe048a46dc7d56cac745553ac86462
14Slowest unit: 12 s:
15artifact_prefix='./'; Test unit written to ./slow-unit-3866f35faef1aa8c88ce6a52ba599c96b9773452
16Slowest unit: 15 s:
17artifact_prefix='./'; Test unit written to ./slow-unit-b4b646db3c8b4309d520a929dd9969cd3d266026
18Slowest unit: 17 s:
19artifact_prefix='./'; Test unit written to ./slow-unit-92ff80cf45bd551bcace289e5fa514f411b614cf
20Slowest unit: 20 s:
21artifact_prefix='./'; Test unit written to ./slow-unit-efd82da2184653f7522296ad76c71eb62d226205
22Slowest unit: 26 s:
23artifact_prefix='./'; Test unit written to ./slow-unit-94cebd976b987de8e532cc562064831761aa3d8d
24[#4096](/bitcoin-core-qa-assets/4096/)	pulse  cov: 2896 ft: 17823 corp: 1307/792Kb exec/s: 4 rss: 309Mb
25Slowest unit: 32 s:
26artifact_prefix='./'; Test unit written to ./slow-unit-e0627c9a132a8619dd91150e67aa5142ef7f1db0
27Slowest unit: 43 s:
28artifact_prefix='./'; Test unit written to ./slow-unit-79d9d700db64f0f307875e04b72d9c71e3fce554
29Slowest unit: 59 s:
30artifact_prefix='./'; Test unit written to ./slow-unit-329c339c4dbe81a59cd0167e0ed96c579a71144d
31[#4784](/bitcoin-core-qa-assets/4784/)	INITED cov: 2896 ft: 17955 corp: 1346/2709Kb exec/s: 2 rss: 392Mb
32[#4784](/bitcoin-core-qa-assets/4784/)	DONE   cov: 2896 ft: 17955 corp: 1346/2709Kb lim: 621890 exec/s: 2 rss: 392Mb
33Done 4784 runs in 2100 second(s)

maflcko commented at 2:05 pm on April 19, 2023: contributor

Yes, this is done at or around the branch-off point for every release. See pulls

maflcko commented at 2:14 pm on April 19, 2023: contributor

(The reason this is done at around the branch-off point is that the then resulting fuzz inputs will be “valid” for the longest duration for the latest release branch)

maflcko commented at 8:40 am on April 21, 2023: contributor

@dergoegge Is someone working on this? If not, I’ll do it.

dergoegge commented at 8:49 am on April 21, 2023: member

@MarcoFalke Not working on this currently. I’ve also never done this before so you doing it is probably best (unless there is an easy command/script to run?).

I’ve been meaning to write some docs for the usual processes in this repo (merging, pruning, maybe linking to the Bitcoin Core fuzzing docs?). Will follow up with that separately.

maflcko commented at 9:00 am on April 21, 2023: contributor

Yeah, it is really simple. See #97

Make sure to set up a fresh Jammy VM for this, as documented in the bash script.

maflcko commented at 9:04 am on April 21, 2023: contributor

Ok, I’ll run the script and create the pull today. Anyone is then invited to review, if they want.

maflcko commented at 12:22 pm on April 21, 2023: contributor

See #119, I think this can be closed now?

fanquake commented at 3:00 pm on April 29, 2023: member

Think we can add some additional notes/here (@dergoegge may follow up), but I’ll close this for now.

fanquake closed this on Apr 29, 2023

Pruning large/slow inputs? #118