[fuzzer] generate new seeds in an infinite loop with random parameters #20752

1. 
   pstratem commented at 4:45 am on December 23, 2020: contributor

   this vastly simplifies running all of the fuzzing targets in parallel to expand the corpus

   depends on #20751

2. fanquake added the label Tests on Dec 23, 2020
3. pstratem force-pushed on Dec 23, 2020
4. pstratem force-pushed on Dec 23, 2020
5. Use subprocess.Popen call directly, remove use of thread pool. ea36284d17
6. pstratem force-pushed on Dec 23, 2020
7. pstratem force-pushed on Dec 23, 2020
8. pstratem force-pushed on Dec 23, 2020
9. pstratem force-pushed on Dec 23, 2020
10. in test/fuzz/test_runner.py:204 in ff804c0392 outdated

    206 
    207     running_jobs = set()
    208-    while len(jobs) > 0 or len(running_jobs) > 0:
    209-        if len(running_jobs) < max_jobs and len(jobs) > 0:
    210-            target, args = jobs.pop()
    211+    while True:
    

    ---

    ---

    

    MarcoFalke commented at 7:35 am on December 23, 2020:

    It was nice to be able to put the while true outside the script. The benefit was that the outside could update this script, the code repo, as well as the seeds repo. This allows continuous fuzzing of the master branch with the latest code and seeds while atomically updating the build system and this script.

    ---

    

    sipa commented at 11:17 pm on December 27, 2020:

    @MarcoFalke I don’t think there is a problem with that really, as no progress is lost by interrupting. Just stop the test_runner at any point, update/recompile, and run again.

    ---

    

    MarcoFalke commented at 7:44 am on December 28, 2020:

    That would complicate the outside script because it had to poll for changes in the repo, while running the fuzzer script. As soon as there are changes, interrupt and rerun.

    ---

    

    MarcoFalke commented at 7:45 am on December 28, 2020:

    If y’all like the while True, maybe expose it via some kind of option?

    ---

    

    sipa commented at 7:49 am on December 28, 2020:

    It’s not so much the while True I like, but the fact that this doesn’t leave cores unused after most of the jobs are done. Doing it randomly also more easily accommodates adding more configurations (which presumably would hurt your workflow equally much, as every new combination would multiply the time between checks by some factor).

    What about adding an option to make this automatically exit after some specified wall clock time?

11. pstratem force-pushed on Dec 23, 2020
12. Infinite loop generating random seeds 1ff0dc525f
13. pstratem force-pushed on Dec 23, 2020
14. pstratem marked this as a draft on Dec 23, 2020
15. 
    DrahtBot commented at 11:18 am on December 24, 2020: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #20759 (doc: [test] Remove outdated comment in fuzz runner by MarcoFalke)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

16. 
    practicalswift commented at 11:11 pm on December 27, 2020: contributor

    Concept ACK
17. DrahtBot added the label Needs rebase on Dec 13, 2021
18. 
    DrahtBot commented at 11:18 pm on December 13, 2021: member

    🐙 This pull request conflicts with the target branch and needs rebase.

    Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a “draft”.

19. 
    fanquake commented at 12:49 pm on April 26, 2022: member

    depends on #20751

    Given this depends on a PR that has since been closed un-merged, I’m going to close for now. Can be re-opened if the dependency changes etc.

20. fanquake closed this on Apr 26, 2022

    ---

21. DrahtBot locked this on Apr 26, 2023

Contributors
pstratem MarcoFalke sipa DrahtBot practicalswift fanquake

Labels
Tests Needs rebase