ci: Use hard-coded root path for CI containers (bugfix) #28185

1. 
   MarcoFalke commented at 9:56 am on July 30, 2023: member

   Currently the CI system will fail if the git folder that holds the Bitcoin Core source is moved from one location to another.

   Fix this by using a single hard-coded root path inside the CI system containers.

   Steps to test:

   • Run the CI system: MAKEJOBS="-j$(nproc)" FILE_ENV="./ci/test/00_setup_env_win64.sh" ./ci/test_run_all.sh
   • Move the git folder: pwd && cd .. && mv bitcoin_core_folder_1 bitcoin_core_folder_2 && cd ./bitcoin_core_folder_2 && pwd
   • Run the CI system again: (same cmd as above)

   On master (error):

   0STRIPPROG="x86_64-w64-mingw32-strip" /bin/bash /bitcoin_core_folder_2/ci/scratch/build/bitcoin-x86_64-w64-mingw32/build-aux/install-sh -c -s ./src/qt/bitcoin-qt.exe ./release
   1/bitcoin_core_folder_2/ci/scratch/build/bitcoin-x86_64-w64-mingw32/build-aux/install-sh: ./src/qt/bitcoin-qt.exe does not exist.
   2make: *** [Makefile:1258: bitcoin-25.99.0-win64-setup.exe] Error 1
   

   On this pull: (pass).

2. 
   DrahtBot commented at 9:56 am on July 30, 2023: contributor

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Reviews

   See the guideline for information on the review process.

   Type	Reviewers
   ACK	fanquake
   Concept ACK	hebasto

   If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

   Conflicts

   No conflicts as of last run.

3. DrahtBot renamed this:
   ci: Use hard-coded root path for CI containers (bugfix)
   ci: Use hard-coded root path for CI containers (bugfix)
   on Jul 30, 2023
4. DrahtBot added the label Tests on Jul 30, 2023
5. MarcoFalke force-pushed on Jul 30, 2023
6. MarcoFalke marked this as a draft on Jul 30, 2023
7. DrahtBot added the label CI failed on Jul 30, 2023
8. MarcoFalke force-pushed on Jul 31, 2023
9. 
   MarcoFalke commented at 1:05 pm on July 31, 2023: member

   I rebased this on a commit by @fanquake to fix another bug, which I am not sure how it happened.

   • Usually the CI system should be cloning iwyu into /ci_base_install/. See https://cirrus-ci.com/task/4887069785325568?logs=build#L2243:

   0Cloning into '/ci_base_install/ci/scratch/iwyu/include-what-you-use'...
   

   • However, it may or may not (?) copy it for some reason from /ci_base_install/. See https://cirrus-ci.com/task/4634154260758528?logs=ci#L117:

   0++ CI_EXEC rsync --archive --stats --human-readable /ci_base_install/ /tmp/cirrus-ci-build
   1sent 10.29K bytes  received 88 bytes  20.76K bytes/sec
   

   • Then, it seemingly fails to pick up the python script: See https://cirrus-ci.com/task/4634154260758528?logs=ci#L3121:

   0python3: can't open file '/tmp/cirrus-ci-build/ci/scratch/iwyu/include-what-you-use/iwyu_tool.py': [Errno 2] No such file or directory
   

   The same seemingly happened with msan, so instead of debugging further, I just did the same changes to msan in faa31866bebb2a25de44af13562b434e22a0d6e4 to work around the bugs and clarify the code in any case.

10. MarcoFalke force-pushed on Jul 31, 2023
11. MarcoFalke force-pushed on Jul 31, 2023
12. MarcoFalke force-pushed on Jul 31, 2023
13. MarcoFalke marked this as ready for review on Aug 9, 2023
14. ci: Drop BASE_SCRATCH_DIR from LIBCXX_DIR

    Using a hard-coded path avoids non-determinism issues and improves CI
    UX.

    fab27127f4
15. ci: Only create folders when needed

    Now that container volumes are used, the folders are no longer mounted.
    They are only needed when running without a container engine (docker,
    podman).

    fa084f5ba5
16. ci: Use hard-coded root path for CI containers fafa17c00b
17. MarcoFalke force-pushed on Aug 9, 2023
18. 
    MarcoFalke commented at 10:33 am on August 9, 2023: member

    Rebased on master to drop already merged bugfix commit d86a83d6b8587b0971e66c6910af23dd8c042969.
19. DrahtBot removed the label CI failed on Aug 9, 2023
20. MarcoFalke requested review from hebasto on Aug 14, 2023
21. 
    MarcoFalke commented at 9:55 am on August 14, 2023: member

    I used the commands in the pull request description to re-test this is still a bugfix on current master.
22. MarcoFalke added this to the milestone 26.0 on Aug 14, 2023
23. 
    hebasto commented at 2:43 pm on August 14, 2023: member

    Currently the CI system will fail if the git folder that holds the Bitcoin Core source is moved from one location to another.

    Are there any real-life use cases for this scenario?

24. 
    MarcoFalke commented at 2:49 pm on August 14, 2023: member

    Yes, Cirrus CI persistent workers, and potentially GitHub (or other’s) persistent workers (haven’t checked).

    Example: https://cirrus-ci.com/task/5714733521698816?logs=ci#L4307

25. 
    hebasto commented at 2:51 pm on August 14, 2023: member

    Concept ACK.
26. 
    MarcoFalke commented at 2:54 pm on August 14, 2023: member

    Also, a developer may be moving the git folder at any time.
27. fanquake approved
28. 
    fanquake commented at 10:17 am on August 15, 2023: member

    ACK https://github.com/bitcoin/bitcoin/pull/28185/commits/fafa17c00b87c685200198979ed13018e55e474a - somewhat tested. MSAN changes are the same as what we did for tidy.
29. fanquake merged this on Aug 15, 2023
30. fanquake closed this on Aug 15, 2023

    ---

31. MarcoFalke deleted the branch on Aug 15, 2023

Contributors
MarcoFalke DrahtBot hebasto fanquake

Review Requested
hebasto

Labels
Tests

Milestone
26.0