ci: Cache more stuff in the ci images: msan, iwyu, pip, sdks #27028

1. 
   maflcko commented at 4:21 pm on February 2, 2023: member

   Now that apt packages are cached in the ci images, it makes sense to think about caching all other packages as well.
2. 
   DrahtBot commented at 4:21 pm on February 2, 2023: contributor

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Reviews

   See the guideline for information on the review process.

   Type	Reviewers
   ACK	TheCharlatan

   If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

   Conflicts

   Reviewers, this pull request conflicts with the following ones:

   • #27090 (ci: Use dedicated root path for ci by MarcoFalke)

   If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

3. DrahtBot renamed this:
   ci: Cache more stuff in the ci images: msan, iwyu, pip, sdks
   ci: Cache more stuff in the ci images: msan, iwyu, pip, sdks
   on Feb 2, 2023
4. DrahtBot added the label Tests on Feb 2, 2023
5. maflcko force-pushed on Feb 2, 2023
6. maflcko force-pushed on Feb 2, 2023
7. maflcko force-pushed on Feb 2, 2023
8. maflcko force-pushed on Feb 15, 2023
9. maflcko force-pushed on Feb 15, 2023
10. in ci/test/01_base_install.sh:13 in fabcf65e46 outdated

     6@@ -7,7 +7,9 @@
     7 export LC_ALL=C.UTF-8
     8 
     9 CI_EXEC_ROOT () { bash -c "$*"; }
    10+CI_EXEC()       { bash -c "$*"; }
    

    ---

    ---

    

    fanquake commented at 5:37 pm on February 20, 2023:

    Are these (doing) the same thing? I see they are used in the same context, immediately after one another. Why the new func? Maybe a comment to explain the difference?

    ---

    

    maflcko commented at 9:14 am on February 21, 2023:

    Yes, they do the same thing, which is the same as bash -c "..." does. I think they should be removed eventually, but for now they aid reviewability (diff moved dimmed zebra)

    ---

    

    maflcko commented at 9:21 am on February 21, 2023:

    Added comment
11. maflcko force-pushed on Feb 21, 2023
12. maflcko force-pushed on Feb 21, 2023
13. maflcko force-pushed on Mar 6, 2023
14. 
    maflcko commented at 2:14 pm on March 6, 2023: member

    Rebased. For review I suggest --color-moved=dimmed-zebra.
15. maflcko force-pushed on Mar 7, 2023
16. 
    TheCharlatan commented at 10:29 pm on March 10, 2023: contributor

    On master I can successfully run MAKEJOBS="-j12" FILE_ENV="./ci/test/00_setup_env_native_tidy.sh" ./ci/test_run_all.sh. When I do this now (with all prior images removed) I get:

    0python3: can't open file '/home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use/iwyu_tool.py': [Errno 2] No such file or directory
    1python3: can't open file '/home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use/fix_includes.py': [Errno 2] No such file or directory
    

17. maflcko force-pushed on Mar 13, 2023
18. 
    maflcko commented at 4:12 pm on March 13, 2023: member

    Thanks, fixed
19. 
    maflcko commented at 3:19 pm on March 20, 2023: member

    Would be good to make progress here. This will allow Cirrus CI to use docker images with a cache and thus avoid all 3rd party network errors like (https://github.com/bitcoin/bitcoin/pull/27277#issuecomment-1475912615) , unless they happen inside of Cirrus CI itself, or GitHub.
20. 
    TheCharlatan commented at 4:51 pm on March 20, 2023: contributor

    Light code review ACK 5555a336019848d2e19e4533b3d2f34e438e6367

    Confirmed that it indeed caches these additional packages. That said, I am not familiar with the build flow through the various scripts and cannot really comment if the code was moved to the correct place, or if conventions were followed.

    Would be good to make progress here.

    Yes, I need to get a better scheduling workflow for PR’s requiring interaction.

21. 
    maflcko commented at 5:10 pm on March 20, 2023: member

    Yes, I need to get a better scheduling workflow for PR’s requiring interaction.

    I was planning to put it up for hi-prio, but I keep missing the meeting :sweat_smile:

22. maflcko requested review from fanquake on Mar 21, 2023
23. DrahtBot added the label Needs rebase on Mar 23, 2023
24. ci: Cache more stuff in the ci images: msan, iwyu, pip, sdks faa0839837
25. maflcko force-pushed on Mar 24, 2023
26. DrahtBot removed the label Needs rebase on Mar 24, 2023
27. TheCharlatan approved
28. 
    TheCharlatan commented at 10:30 am on March 24, 2023: contributor

    re-ACK faa0839837e6d2ba5d0ab08fddd497a257d7e075
29. fanquake merged this on Mar 24, 2023
30. fanquake closed this on Mar 24, 2023

    ---

31. maflcko deleted the branch on Mar 24, 2023
32. sidhujag referenced this in commit cd3e257000 on Mar 24, 2023
33. hebasto referenced this in commit 20d7a890d3 on Aug 14, 2023
34. hebasto referenced this in commit c2a87bd302 on Aug 14, 2023
35. fanquake referenced this in commit 85e672ab3d on Aug 15, 2023
36. sidhujag referenced this in commit 36f4ab1b1f on Aug 17, 2023
37. janus referenced this in commit b2224a047b on Sep 11, 2023
38. Retropex referenced this in commit 43e58c5087 on Oct 4, 2023
39. Retropex referenced this in commit 5f62ed90f6 on Oct 4, 2023
40. bitcoin locked this on Mar 23, 2024

Contributors
maflcko DrahtBot fanquake TheCharlatan

Review Requested
fanquake

Labels
Tests