ci: Where to run heavy and high-maintenance CI tasks? #33668

1. 
   maflcko commented at 2:42 pm on October 21, 2025: member

   There are some heavy, and some possibly high-maintenance CI configs, which may not be productive to run on every pull request. They could be run on just the master branch for every merge?

   Come up in the context of:

   • #33509 (comment)
   • #33411 (comment)
   • #33461 (comment)
   • #33626
   • … ?

   And possibly useful for #22064 (comment)

2. maflcko added the label Brainstorming on Oct 21, 2025
3. maflcko added the label Tests on Oct 21, 2025
4. maflcko added the label CI failed on Oct 21, 2025
5. maflcko removed the label CI failed on Oct 21, 2025
6. 
   fanquake commented at 2:47 pm on October 21, 2025: member

   I don’t really think “heavy” should be as much of a blocker as fragile here. There should be scope to expand the amount of CPU dedicated to the CI.
7. 
   maflcko commented at 3:14 pm on October 21, 2025: member

   Good point that if the only issue is the CPU usage, it could be fixed by more CPU. Though, stuff like msan on -O0 is so heavy that it doesn’t pass at all on the GHA provided VM after 6 hours, so it will likely be slow enough to not make sense on pull requests at all.
8. 
   m3dwards commented at 3:56 pm on October 28, 2025: contributor

   They could be run on just the master branch for every merge

   I like this idea for things that are expensive (beyond what can be solved with more CPU / Memory). As for fragile, I’m not sure I like the idea of just running fragile things less over running them regularly to get more data on why they are fragile and hopefully fixing that. That said, fixing fragile CI jobs often takes significant development effort and you can often be left wondering if it was worth the investment over disabling the job or accepting a certain amount of flake.

9. maflcko renamed this:
   ci: Where to run heavy and fragile CI tasks?
   ci: Where to run heavy and high-maintenance CI tasks?
   on Oct 28, 2025
10. 
    maflcko commented at 5:17 pm on October 28, 2025: member

    As for fragile, I’m not sure I like the idea of just running fragile things less over running them regularly to get more data on why they are fragile and hopefully fixing that. That said, fixing fragile CI jobs often takes significant development effort and you can often be left wondering if it was worth the investment over disabling the job or accepting a certain amount of flake.

    Ah, I think “fragile” may not have been the right word. I was not trying to refer to flaky CI tasks with non-deterministic intermittent failures, which happen for all tasks. Instead, I was more referring to tasks that are deterministically failing, albeit with false-positive warnings, or otherwise hard-to-understand and hard-to-act-on error messages, blocking pull requests for unrelated reasons.

    I’ve replace the word to say “high-maintenance” instead.

11. 
    m3dwards commented at 5:51 pm on October 28, 2025: contributor

    I was more referring to tasks that are deterministically failing, albeit with false-positive warnings, or otherwise hard-to-understand and hard-to-act-on error messages, blocking pull requests for unrelated reasons

    Ah, in that case, sounds good to run them less frequently 👍

12. 
    maflcko commented at 6:59 pm on October 28, 2025: member

    I went ahead and pushed the msan -O0, and the s390x task to https://github.com/maflcko/bitcoin-core-nightly/actions/runs/18885767497/job/53900576177. However, they are so slow on GHA, that they time out after 6 hours.

    Possibly with a CTest Dashboard to upload results, those tasks could be run on faster or at least dedicated hardware without timeouts.

Contributors
maflcko fanquake m3dwards

Labels
Brainstorming Tests