ci: Add missing set -e to 01_base_install.sh #27739

Otherwise errors are silently ignored

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--021abf342d371248e50ceaed478a90ca-->

Reviews

See the guideline for information on the review process.

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Looks like it is working: https://cirrus-ci.com/task/4766589006905344?logs=build#L2788

Added another unrelated commit to re-trigger CI and to give more yummy review to reviewers.

Still not sure what is going on here:

MAKEJOBS="-j12" FILE_ENV="./ci/test/00_setup_env_native_tidy.sh" ./ci/test_run_all.sh
> ...
> fatal: destination path '/home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use' already exists and is not an empty directory.
ls home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use
> ls: cannot access 'home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use': No such file or directory

Command runs fine on master, so likely something that is now surfaced with set -e.

Can you replace set -e with set -ex and show the surrounding line?

I can't see how this can happen. The error doesn't happen in base_install, from what I can tell, so it seems unrelated to the changes here. However, it passing on master makes me puzzle.

Re #27739 (comment)

Can you replace set -e with set -ex and show the surrounding line?

0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
+ '[' -n '' ']'
+ [[ '' == \t\r\u\e ]]
+ [[ true == \t\r\u\e ]]
+ git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_16 /home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use
fatal: destination path '/home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use' already exists and is not an empty directory.

Does it reproduce on your system? I tried removing any remnants of the image, but the problem persists.

Can you share the output where it should say "Skip base install"? Because it should never be called twice

+ [[ true == \t\r\u\e ]]

Unrelated: Anyone know why it prints "true" as "\t\r\u\e"? Maybe we should remove bash, but I am not sure if there is an alternative.

Re #27739 (comment)

Can you share the output where it should say "Skip base install"? Because it should never be called twice

<details><summary>Details</summary> <p>

ci_native_tidy_ccache ci_native_tidy_depends ci_native_tidy_previous_releases

Number of files: 979 (reg: 918, dir: 61) Number of created files: 973 (reg: 918, dir: 55) Number of deleted files: 0 Number of regular files transferred: 918 Total file size: 74.35M bytes Total transferred file size: 74.35M bytes Literal data: 74.35M bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 74.44M Total bytes received: 17.79K

sent 74.44M bytes received 17.79K bytes 148.91M bytes/sec total size is 74.35M speedup is 1.00

• CFG_DONE=ci.base-install-done ++ git config --global ci.base-install-done
• '[' '' == true ']'
• '[' -n '' ']'
• [[ ubuntu:lunar == centos ]]
• '[' '' '!=' no ']'
• [[ -n '' ]]
• retry -- apt-get update Get:1 http://security.ubuntu.com/ubuntu lunar-security InRelease [109 kB] Hit:2 http://archive.ubuntu.com/ubuntu lunar InRelease Get:3 http://archive.ubuntu.com/ubuntu lunar-updates InRelease [109 kB] Get:4 http://archive.ubuntu.com/ubuntu lunar-backports InRelease [99.8 kB] Fetched 317 kB in 3s (123 kB/s) Reading package lists...
• retry -- bash -c 'apt-get install --no-install-recommends --no-upgrade -y clang-16 libclang-16-dev llvm-16-dev libomp-16-dev clang-tidy-16 jq bear cmake libevent-dev libboost-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev systemtap-sdt-dev libqt5gui5 libqt5core5a libqt5dbus5 qttools5-dev qttools5-dev-tools libqrencode-dev libsqlite3-dev libdb++-dev build-essential libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates ccache python3 rsync git procps bison' Reading package lists... Building dependency tree... Reading state information... Skipping clang-16, it is already installed and upgrade is not set. Skipping libclang-16-dev, it is already installed and upgrade is not set. Skipping llvm-16-dev, it is already installed and upgrade is not set. Skipping libomp-16-dev, it is already installed and upgrade is not set. Skipping clang-tidy-16, it is already installed and upgrade is not set. Skipping jq, it is already installed and upgrade is not set. Skipping bear, it is already installed and upgrade is not set. Skipping cmake, it is already installed and upgrade is not set. Skipping libevent-dev, it is already installed and upgrade is not set. Skipping libboost-dev, it is already installed and upgrade is not set. Skipping libminiupnpc-dev, it is already installed and upgrade is not set. Skipping libnatpmp-dev, it is already installed and upgrade is not set. Skipping libzmq3-dev, it is already installed and upgrade is not set. Skipping systemtap-sdt-dev, it is already installed and upgrade is not set. Skipping libqt5gui5, it is already installed and upgrade is not set. Skipping libqt5core5a, it is already installed and upgrade is not set. Skipping libqt5dbus5, it is already installed and upgrade is not set. Skipping qttools5-dev, it is already installed and upgrade is not set. Skipping qttools5-dev-tools, it is already installed and upgrade is not set. Skipping libqrencode-dev, it is already installed and upgrade is not set. Skipping libsqlite3-dev, it is already installed and upgrade is not set. Skipping libdb++-dev, it is already installed and upgrade is not set. Skipping build-essential, it is already installed and upgrade is not set. Skipping libtool, it is already installed and upgrade is not set. Skipping autotools-dev, it is already installed and upgrade is not set. Skipping automake, it is already installed and upgrade is not set. Skipping pkg-config, it is already installed and upgrade is not set. Skipping bsdmainutils, it is already installed and upgrade is not set. Skipping curl, it is already installed and upgrade is not set. Skipping ca-certificates, it is already installed and upgrade is not set. Skipping ccache, it is already installed and upgrade is not set. Skipping python3, it is already installed and upgrade is not set. Skipping rsync, it is already installed and upgrade is not set. Skipping git, it is already installed and upgrade is not set. Skipping procps, it is already installed and upgrade is not set. Skipping bison, it is already installed and upgrade is not set. 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
• '[' -n '' ']'
• [[ '' == \t\r\u\e ]]
• [[ true == \t\r\u\e ]]
• git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_16 /home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use fatal: destination path '/home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use' already exists and is not an empty directory.

</p> </details>

Ok so git config --global ci.base-install-done evaluates to the empty string? Can you share the details where it is first set?

https://github.com/bitcoin/bitcoin/blob/fa534d780e0bb811b7b762cbd430e30f92dd7d7e/ci/test/01_base_install.sh#L81

(This should happen when building the image)

Re #27739 (comment)

Can you share the details where it is first set?

Here is the entire output I am willing to share (the rest above prints my env, which I don't want to paste here)

<p> <details> Creating ubuntu:lunar container to run in [+] Building 1.0s (9/9) FINISHED => [internal] load build definition from test_imagefile 0.0s => => transferring dockerfile: 42B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/ubuntu:lunar 0.9s => [internal] load build context 0.0s => => transferring context: 257B 0.0s => [1/4] FROM docker.io/library/ubuntu:lunar@sha256:db2764b64a490a9ce44a7ce0c7ba7cc6d0472c8c44684960dbca7a06525eb597 0.0s => CACHED [2/4] COPY ./ci/retry/retry /usr/bin/retry 0.0s => CACHED [3/4] COPY ./ci/test/00_setup_env.sh ././ci/test/00_setup_env_native_tidy.sh ./ci/test/01_base_install.sh /ci_base_install/ci/test/ 0.0s => CACHED [4/4] RUN ["bash", "-c", "cd /ci_base_install/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"] 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:e30eaeeb008ee7d59ed41ca8ddbd92bc95d93aadfc64b90daff6e00198c85d99 0.0s => => naming to docker.io/library/ci_native_tidy 0.0s ci_native_tidy_ccache ci_native_tidy_depends ci_native_tidy_previous_releases

Number of files: 979 (reg: 918, dir: 61) Number of created files: 973 (reg: 918, dir: 55) Number of deleted files: 0 Number of regular files transferred: 918 Total file size: 74.35M bytes Total transferred file size: 74.35M bytes Literal data: 74.35M bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 74.44M Total bytes received: 17.78K

sent 74.44M bytes received 17.78K bytes 148.91M bytes/sec total size is 74.35M speedup is 1.00 Hit:1 http://archive.ubuntu.com/ubuntu lunar InRelease Get:2 http://archive.ubuntu.com/ubuntu lunar-updates InRelease [109 kB] Get:3 http://archive.ubuntu.com/ubuntu lunar-backports InRelease [99.8 kB] Get:4 http://security.ubuntu.com/ubuntu lunar-security InRelease [109 kB] Fetched 317 kB in 1s (528 kB/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information... Skipping clang-16, it is already installed and upgrade is not set. Skipping libclang-16-dev, it is already installed and upgrade is not set. Skipping llvm-16-dev, it is already installed and upgrade is not set. Skipping libomp-16-dev, it is already installed and upgrade is not set. Skipping clang-tidy-16, it is already installed and upgrade is not set. Skipping jq, it is already installed and upgrade is not set. Skipping bear, it is already installed and upgrade is not set. Skipping cmake, it is already installed and upgrade is not set. Skipping libevent-dev, it is already installed and upgrade is not set. Skipping libboost-dev, it is already installed and upgrade is not set. Skipping libminiupnpc-dev, it is already installed and upgrade is not set. Skipping libnatpmp-dev, it is already installed and upgrade is not set. Skipping libzmq3-dev, it is already installed and upgrade is not set. Skipping systemtap-sdt-dev, it is already installed and upgrade is not set. Skipping libqt5gui5, it is already installed and upgrade is not set. Skipping libqt5core5a, it is already installed and upgrade is not set. Skipping libqt5dbus5, it is already installed and upgrade is not set. Skipping qttools5-dev, it is already installed and upgrade is not set. Skipping qttools5-dev-tools, it is already installed and upgrade is not set. Skipping libqrencode-dev, it is already installed and upgrade is not set. Skipping libsqlite3-dev, it is already installed and upgrade is not set. Skipping libdb++-dev, it is already installed and upgrade is not set. Skipping build-essential, it is already installed and upgrade is not set. Skipping libtool, it is already installed and upgrade is not set. Skipping autotools-dev, it is already installed and upgrade is not set. Skipping automake, it is already installed and upgrade is not set. Skipping pkg-config, it is already installed and upgrade is not set. Skipping bsdmainutils, it is already installed and upgrade is not set. Skipping curl, it is already installed and upgrade is not set. Skipping ca-certificates, it is already installed and upgrade is not set. Skipping ccache, it is already installed and upgrade is not set. Skipping python3, it is already installed and upgrade is not set. Skipping rsync, it is already installed and upgrade is not set. Skipping git, it is already installed and upgrade is not set. Skipping procps, it is already installed and upgrade is not set. Skipping bison, it is already installed and upgrade is not set. 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. fatal: destination path '/home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use' already exists and is not an empty directory.

</p> </details>

From what I am seeing this is in the image building stage.

Well, it isn't building the image, just printing CACHED. I'd need the output from when it was built of the step that ran base_install:

CACHED [4/4] RUN ["bash", "-c", "cd /ci_base_install/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]

the rest above prints my env, which I don't want to paste here

If you can reproduce on a fresh install of your OS, that'd help too

Re #27739 (comment)

If you can reproduce on a fresh install of your OS, that'd help too

Not a fresh install, but another OS where I have not run this before:

<p> <details> MAKEJOBS="-j12" FILE_ENV="./ci/test/00_setup_env_native_tidy.sh" ./ci/test_run_all.sh Setting specific values in env Fallback to default values in env (if not yet set) ... CONTAINER_NAME=ci_native_tidy DEPENDS_DIR=/home/drgrid/bitcoin/depends TERM=xterm-256color SHELL=/usr/bin/zsh BITCOIN_CONFIG=CC=clang-16 CXX=clang++-16 --with-incompatible-bdb --disable-hardening CFLAGS='-O0 -g0' CXXFLAGS='-O0 -g0 -I/usr/lib/llvm-16/lib/clang/16/include' SDK_URL=https://bitcoincore.org/depends-sources/sdks SHLVL=2 LC_TELEPHONE=de_CH.UTF-8 LOGNAME=drgrid RUN_TIDY=true XDG_RUNTIME_DIR=/run/user/1000 CI_IMAGE_NAME_TAG=ubuntu:lunar PATH=/home/drgrid/bitcoin/ci/retry:/home/drgrid/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/drgrid/.rvm/bin LC_IDENTIFICATION=de_CH.UTF-8 BOOST_TEST_RANDOM=1 CCACHE_TEMPDIR=/tmp/.ccache-temp TEST_RUNNER_ENV= BASE_OUTDIR=/home/drgrid/bitcoin/ci/scratch/out/x86_64-pc-linux-gnu PREVIOUS_RELEASES_DIR=/home/drgrid/bitcoin/releases/x86_64-pc-linux-gnu NO_DEPENDS=1 _=/usr/bin/python3 Creating ubuntu:lunar container to run in [+] Building 130.9s (9/9) FINISHED => [internal] load build definition from test_imagefile 0.1s => => transferring dockerfile: 404B 0.0s => [internal] load .dockerignore 0.1s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/ubuntu:lunar 1.5s => [1/4] FROM docker.io/library/ubuntu:lunar@sha256:db2764b64a490a9ce44a7ce0c7ba7cc6d0472c8c44684960dbca7a06525eb597 1.3s => => resolve docker.io/library/ubuntu:lunar@sha256:db2764b64a490a9ce44a7ce0c7ba7cc6d0472c8c44684960dbca7a06525eb597 0.0s => => sha256:db2764b64a490a9ce44a7ce0c7ba7cc6d0472c8c44684960dbca7a06525eb597 1.13kB / 1.13kB 0.0s => => sha256:ce7f6664be1081be78dfcf319cb11d7bceeef17b32df373282a66fe940e47f6d 424B / 424B 0.0s => => sha256:0120ea4307c597784895e9f539a67d88239d2c71ebfa830195fa2a650e4f3b5d 2.30kB / 2.30kB 0.0s => => sha256:cc1fe050fbd8eee8b0e2ef684c783d2c269fb7f61a0df0dee4639340bf612925 26.83MB / 26.83MB 0.7s => => extracting sha256:cc1fe050fbd8eee8b0e2ef684c783d2c269fb7f61a0df0dee4639340bf612925 0.5s => [internal] load build context 0.0s => => transferring context: 12.78kB 0.0s => [2/4] COPY ./ci/retry/retry /usr/bin/retry 0.1s => [3/4] COPY ./ci/test/00_setup_env.sh ././ci/test/00_setup_env_native_tidy.sh ./ci/test/01_base_install.sh /ci_base_install/ci/test/ 0.2s => [4/4] RUN ["bash", "-c", "cd /ci_base_install/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"] 117.2s => exporting to image 10.4s => => exporting layers 10.4s => => writing image sha256:345b719a8a0223b71ba2bb051960ff42e89017ae86b39f4791ebdb771611a3a6 0.0s => => naming to docker.io/library/ci_native_tidy 0.0s ci_native_tidy_ccache ci_native_tidy_depends ci_native_tidy_previous_releases

Number of files: 979 (reg: 918, dir: 61) Number of created files: 975 (reg: 918, dir: 57) Number of deleted files: 0 Number of regular files transferred: 918 Total file size: 74.35M bytes Total transferred file size: 74.35M bytes Literal data: 74.35M bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 74.44M Total bytes received: 17.79K

sent 74.44M bytes received 17.79K bytes 148.91M bytes/sec total size is 74.35M speedup is 1.00 Hit:1 http://security.ubuntu.com/ubuntu lunar-security InRelease Hit:2 http://archive.ubuntu.com/ubuntu lunar InRelease Hit:3 http://archive.ubuntu.com/ubuntu lunar-updates InRelease Hit:4 http://archive.ubuntu.com/ubuntu lunar-backports InRelease Reading package lists... Reading package lists... Building dependency tree... Reading state information... Skipping clang-16, it is already installed and upgrade is not set. Skipping libclang-16-dev, it is already installed and upgrade is not set. Skipping llvm-16-dev, it is already installed and upgrade is not set. Skipping libomp-16-dev, it is already installed and upgrade is not set. Skipping clang-tidy-16, it is already installed and upgrade is not set. Skipping jq, it is already installed and upgrade is not set. Skipping bear, it is already installed and upgrade is not set. Skipping cmake, it is already installed and upgrade is not set. Skipping libevent-dev, it is already installed and upgrade is not set. Skipping libboost-dev, it is already installed and upgrade is not set. Skipping libminiupnpc-dev, it is already installed and upgrade is not set. Skipping libnatpmp-dev, it is already installed and upgrade is not set. Skipping libzmq3-dev, it is already installed and upgrade is not set. Skipping systemtap-sdt-dev, it is already installed and upgrade is not set. Skipping libqt5gui5, it is already installed and upgrade is not set. Skipping libqt5core5a, it is already installed and upgrade is not set. Skipping libqt5dbus5, it is already installed and upgrade is not set. Skipping qttools5-dev, it is already installed and upgrade is not set. Skipping qttools5-dev-tools, it is already installed and upgrade is not set. Skipping libqrencode-dev, it is already installed and upgrade is not set. Skipping libsqlite3-dev, it is already installed and upgrade is not set. Skipping libdb++-dev, it is already installed and upgrade is not set. Skipping build-essential, it is already installed and upgrade is not set. Skipping libtool, it is already installed and upgrade is not set. Skipping autotools-dev, it is already installed and upgrade is not set. Skipping automake, it is already installed and upgrade is not set. Skipping pkg-config, it is already installed and upgrade is not set. Skipping bsdmainutils, it is already installed and upgrade is not set. Skipping curl, it is already installed and upgrade is not set. Skipping ca-certificates, it is already installed and upgrade is not set. Skipping ccache, it is already installed and upgrade is not set. Skipping python3, it is already installed and upgrade is not set. Skipping rsync, it is already installed and upgrade is not set. Skipping git, it is already installed and upgrade is not set. Skipping procps, it is already installed and upgrade is not set. Skipping bison, it is already installed and upgrade is not set. 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. fatal: destination path '/home/drgrid/bitcoin/ci/scratch/iwyu/include-what-you-use' already exists and is not an empty directory.

</p> </details>

Thanks, the last commit should fix your bug.

I wonder if env's like USER(NAME) and PATH should also be excluded, but this can be done later?

Otherwise errors are silently ignored

Should we do the same in ci/test/00_setup_env.sh?

https://cirrus-ci.com/task/5206141060251648:

./ci/test/00_setup_env.sh: line 33: /ci_base_install/depends/config.guess: No such file or directory

ACK fa12558d21aa03c22407a1458a0345d8a7ab6a4b

Re #27739 (comment)

Thanks, the last commit should fix your bug.

Thank you for digging into this, seems like everything is working as it should now. ACK fa12558

+ [[ true == \t\r\u\e ]]

Unrelated: Anyone know why it prints "true" as "\t\r\u\e"? Maybe we should remove bash, but I am not sure if there is an alternative.

There are two options:

• remove quoting:

--- a/ci/test/01_base_install.sh
+++ b/ci/test/01_base_install.sh
@@ -70,7 +70,7 @@ if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
   ninja -C "${BASE_SCRATCH_DIR}"/msan/cxx_build/ "$MAKEJOBS"
 fi
 
-if [[ "${RUN_TIDY}" == "true" ]]; then
+if [[ $RUN_TIDY == true ]]; then
   git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_16 "${DIR_IWYU}"/include-what-you-use
   cmake -B "${DIR_IWYU}"/build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-16 -S "${DIR_IWYU}"/include-what-you-use
   make -C "${DIR_IWYU}"/build/ install "$MAKEJOBS"

or

• use Bash builtin [ command:

--- a/ci/test/01_base_install.sh
+++ b/ci/test/01_base_install.sh
@@ -70,7 +70,7 @@ if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
   ninja -C "${BASE_SCRATCH_DIR}"/msan/cxx_build/ "$MAKEJOBS"
 fi
 
-if [[ "${RUN_TIDY}" == "true" ]]; then
+if [ "${RUN_TIDY}" = "true" ]; then
   git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_16 "${DIR_IWYU}"/include-what-you-use
   cmake -B "${DIR_IWYU}"/build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-16 -S "${DIR_IWYU}"/include-what-you-use
   make -C "${DIR_IWYU}"/build/ install "$MAKEJOBS"

I'd prefer the former option because set -x apply additional quoting for [ and ], which makes the output less readable.

Anyway, the current usage of [...] and [[...]] is quite inconsistent in the CI scripts.