ci: use LLVM/Clang 17 in tidy job #28482

fanquake commented at 3:05 PM on September 14, 2023: member

Also update MSAN to use 17.0.2.

Related to #28465.

DrahtBot commented at 3:05 PM on September 14, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	maflcko

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

No conflicts as of last run.

DrahtBot added the label Tests on Sep 14, 2023

maflcko commented at 5:37 PM on September 14, 2023: member

For reference, I cancelled the tasks here, except for tidy, because they are not needed on this draft-pull and there was a backlog of 20 pull requests * 12 tasks (=240 tasks) on https://cirrus-ci.com/github/bitcoin/bitcoin

maflcko commented at 5:43 PM on September 14, 2023: member

(tidy passed, so I guess this is good to go once the iwyu branch is available)

in ci/test/01_base_install.sh:78 in fb3a37a6a3 outdated

  73 | @@ -74,8 +74,8 @@ if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
  74 |  fi
  75 |  
  76 |  if [[ "${RUN_TIDY}" == "true" ]]; then
  77 | -  git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_16 /include-what-you-use
  78 | -  cmake -B /iwyu-build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-16 -S /include-what-you-use
  79 | +  git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b master /include-what-you-use
  80 | +  cmake -B /iwyu-build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-17 -S /include-what-you-use

maflcko commented at 6:03 AM on September 15, 2023:

Given that for this task we want the version number to be fixed and given that it spans over multiple files, I wonder if it makes sense to add an export TIDY_LLVM_V="17" to the env file?

This should also avoid invalidating the docker image for all other (unrelated) tasks, due to touching this 01_install file.

fanquake commented at 9:44 AM on September 15, 2023:

Pushed up a change to do this.

fanquake force-pushed on Sep 15, 2023

DrahtBot added the label CI failed on Sep 15, 2023

fanquake force-pushed on Sep 20, 2023

maflcko added the label DrahtBot Guix build requested on Sep 20, 2023

DrahtBot added the label Needs rebase on Sep 20, 2023

fanquake force-pushed on Sep 20, 2023

DrahtBot removed the label Needs rebase on Sep 20, 2023

DrahtBot commented at 10:00 PM on September 20, 2023: contributor

Guix builds (on x86_64)

File	commit ff564c75e751db6cfaf2a5f1b8a3b471f510976f<br>(master)	commit 9cc7949836a7f696373c552510d1337983ee0d58<br>(master and this pull)
SHA256SUMS.part	`981398bf96618280...`	`e148dcb81fab8ffb...`
*-aarch64-linux-gnu-debug.tar.gz	`b5ba491f19f81d99...`	`c5f7b6dcd678a7df...`
*-aarch64-linux-gnu.tar.gz	`d8f956f48dcc4b7a...`	`04de76c319fd77e8...`
*-arm-linux-gnueabihf-debug.tar.gz	`157cbc7a443c454d...`	`35440bf337bc38fd...`
*-arm-linux-gnueabihf.tar.gz	`4039ab72018bbc73...`	`87640b014b2967c4...`
*-arm64-apple-darwin-unsigned.tar.gz	`85f1ef47f3e4a06c...`	`5f634f0497a22ba2...`
*-arm64-apple-darwin-unsigned.zip	`43a97e811d5c8f57...`	`bf03dd99af24253c...`
*-arm64-apple-darwin.tar.gz	`962ae228ba52fc7f...`	`c377d3db9cd2b103...`
*-powerpc64-linux-gnu-debug.tar.gz	`2cec20c7fa9b4ea3...`	`530b21fa8fef36c0...`
*-powerpc64-linux-gnu.tar.gz	`5535ae062729336a...`	`18b02ba2e29dba46...`
*-powerpc64le-linux-gnu-debug.tar.gz	`264510179fe26b1e...`	`daa8d80454f046a8...`
*-powerpc64le-linux-gnu.tar.gz	`447224e806b89ec0...`	`d4665aa1146e0deb...`
*-riscv64-linux-gnu-debug.tar.gz	`8b62022ab6e5f337...`	`d20162d7b35e131e...`
*-riscv64-linux-gnu.tar.gz	`77ce785aa1b72983...`	`23cc03931664eba9...`
*-x86_64-apple-darwin-unsigned.tar.gz	`78a8fa924cdf73f8...`	`f5002e8a6b9d2fd3...`
*-x86_64-apple-darwin-unsigned.zip	`7d5d75ff080c0137...`	`530b179e840217ee...`
*-x86_64-apple-darwin.tar.gz	`ed8e677f06d86030...`	`cfcef9a53cead33c...`
*-x86_64-linux-gnu-debug.tar.gz	`2c9e947b15a34679...`	`b593b7c4bc728bbf...`
*-x86_64-linux-gnu.tar.gz	`b48c19cfba93d938...`	`dd9de75ac6f4c3ec...`
*.tar.gz	`d7c0c117bd75c33d...`	`b3cbc30a75dac68b...`
guix_build.log	`f260bbca51ee3402...`	`67727e17b88aa5b6...`
guix_build.log.diff		`df60ba7db3e7bf7d...`

DrahtBot removed the label DrahtBot Guix build requested on Sep 20, 2023

DrahtBot removed the label CI failed on Sep 20, 2023

DrahtBot added the label CI failed on Oct 1, 2023

DrahtBot removed the label CI failed on Oct 2, 2023

fanquake force-pushed on Oct 3, 2023

fanquake force-pushed on Oct 6, 2023

DrahtBot added the label CI failed on Oct 6, 2023

fanquake commented at 4:03 PM on October 9, 2023: member

Given the discussion in https://github.com/include-what-you-use/include-what-you-use/issues/1276, it seems there might not actually be a new release of IWYU until a breaking change in LLVM (the master branch "seems" to be working ok here, and with LLVM 17). I wonder if we should just pick a commit from the repo, to move things forward.

in ci/test/01_base_install.sh:45 in c414453af6 outdated

  41 | @@ -42,7 +42,7 @@ if [ -n "$PIP_PACKAGES" ]; then
  42 |  fi
  43 |  
  44 |  if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
  45 | -  git clone --depth=1 https://github.com/llvm/llvm-project -b "llvmorg-17.0.0-rc4" /msan/llvm-project
  46 | +  git clone --depth=1 https://github.com/llvm/llvm-project -b llvmorg-17.0.2 /msan/llvm-project

maflcko commented at 6:49 AM on October 10, 2023:

Could add ${CI_RETRY_EXE} while touching this?

fanquake commented at 9:40 AM on October 10, 2023:

Added

in ci/test/01_base_install.sh:76 in c414453af6 outdated

  72 | @@ -73,8 +73,8 @@ if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
  73 |  fi
  74 |  
  75 |  if [[ "${RUN_TIDY}" == "true" ]]; then
  76 | -  git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_16 /include-what-you-use
  77 | -  cmake -B /iwyu-build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-16 -S /include-what-you-use
  78 | +  git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b master /include-what-you-use

maflcko commented at 6:49 AM on October 10, 2023:

Same?

fanquake commented at 9:40 AM on October 10, 2023:

Added

maflcko commented at 6:50 AM on October 10, 2023: member

Yes, seems fine to hard-code the current master commit

ci: use LLVM 17.0.2 in MSAN jobs ce46b68941

fanquake force-pushed on Oct 10, 2023

fanquake marked this as ready for review on Oct 10, 2023

in ci/test/01_base_install.sh:77 in 1a658ab1a2 outdated

  72 | @@ -73,8 +73,9 @@ if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
  73 |  fi
  74 |  
  75 |  if [[ "${RUN_TIDY}" == "true" ]]; then
  76 | -  git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_16 /include-what-you-use
  77 | -  cmake -B /iwyu-build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-16 -S /include-what-you-use
  78 | +  ${CI_RETRY_EXE} git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b master /include-what-you-use
  79 | +  git -C /include-what-you-use checkout a138eaac254e5a472464e31d5ec418fe6e6f1fc7

maflcko commented at 10:00 AM on October 10, 2023:

You'll have to download the commit before checking it out.

master may be a different commit than this one.

fanquake commented at 10:01 AM on October 10, 2023:

Right. I'll just drop depth=1

ci: use LLVM/Clang 17 in tidy job 8735e2c136

fanquake force-pushed on Oct 10, 2023

maflcko commented at 10:04 AM on October 10, 2023: member

lgtm ACK 8735e2c1365cba9246918ea167a40ffe44beb3ba

fanquake commented at 12:11 PM on October 10, 2023: member

Tidy is green. MSAN failure is an out-of-space https://cirrus-ci.com/task/4727416686379008?logs=ci#L8613:

COMMIT ci_native_msan
Error: committing container for step {Env:[FILE_ENV=./ci/test/00_setup_env_native_msan.sh PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[bash -c cd /ci_container_base/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh] Flags:[] Attrs:map[json:true] Message:RUN bash -c cd /ci_container_base/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh Original:RUN ["bash", "-c", "cd /ci_container_base/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]}: copying layers and metadata for container "9078ecd7c02b7fbc1c0de1d5d4534f78e13de447a360657750598e2941b10ce8": writing blob: adding layer with blob "sha256:4292df0af79f12a96574004b5501db4696d6f32549f7fdf6cf97f02981331925": processing tar file(write /msan/llvm-project/llvm/test/CodeGen/AMDGPU/twoaddr-regsequence.mir: no space left on device): exit status 1

maflcko commented at 1:13 PM on October 10, 2023: member

For now, you can re-run. The machines have 80GB, which should be enough for all tasks to store their images:

podman image ls
REPOSITORY                                     TAG         IMAGE ID      CREATED       SIZE
localhost/ci_native_qt5                        latest      e8fdb0e1b968  2 days ago    1.01 GB
localhost/ci_win64                             latest      0edf4d450648  5 days ago    2.13 GB
localhost/ci_native_nowallet_libbitcoinkernel  latest      c3d13f300783  2 weeks ago   722 MB
localhost/ci_macos_cross                       latest      6af49a13445b  2 weeks ago   1.07 GB
localhost/ci_native_msan                       latest      de994cb050d8  2 weeks ago   5.7 GB
localhost/ci_i686_centos                       latest      1bc6df012537  2 weeks ago   656 MB
docker.io/library/debian                       buster      4ad95f81332e  2 weeks ago   119 MB
docker.io/library/debian                       bullseye    909f4919a453  2 weeks ago   129 MB
<none>                                         <none>      9f25d95a5e92  3 weeks ago   80.4 MB
<none>                                         <none>      5846a450a8d0  6 weeks ago   80.4 MB
<none>                                         <none>      c8cf6c4e81f8  6 weeks ago   80.4 MB
quay.io/centos/amd64                           stream9     75bbb0f90548  7 weeks ago   160 MB
docker.io/amd64/ubuntu                         23.10       58f19d7ce16d  7 weeks ago   73.3 MB
<none>                                         <none>      3945dca58267  7 weeks ago   80.4 MB
docker.io/amd64/debian                         bookworm    3a37950934ff  7 weeks ago   121 MB
docker.io/library/ubuntu                       22.04       01f29b872827  2 months ago  80.4 MB
docker.io/amd64/ubuntu                         22.04       01f29b872827  2 months ago  80.4 MB
docker.io/library/ubuntu                       20.04       6df894023726  2 months ago  75.2 MB

Maybe using a depends_sources volume that is shared across all tasks can help reduce the disk usage?

DrahtBot removed the label CI failed on Oct 10, 2023

fanquake merged this on Oct 11, 2023

fanquake closed this on Oct 11, 2023

fanquake deleted the branch on Oct 11, 2023

Frank-GER referenced this in commit 2d2b1e5bdc on Oct 13, 2023

bitcoin locked this on Oct 10, 2024