ci: Work around podman stop intermittent failure #28547

maflcko commented at 9:03 AM on September 28, 2023: member

Sometimes, it seems that podman stop does not work. Presumably, it falls back to podman kill, which is async.

Try to work around this intermittent issue by using the rm --force over stop.

Example failing log https://cirrus-ci.com/task/4549784611061760?logs=ci#L238:

Restart docker before run to stop and clear all containers started with --rm
++ podman container stop --all
e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009
++ echo 'Prune all dangling images'
Prune all dangling images
++ docker image prune --force
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
+++ docker run --cap-add LINUX_IMMUTABLE --rm --interactive --detach --tty --mount type=bind,src=/tmp/cirrus-build-1970593815,dst=/tmp/cirrus-build-1970593815,readonly --mount type=volume,src=ci_macos_cross_ccache,dst=/tmp/ccache_dir --mount type=volume,src=ci_macos_cross_depends,dst=/ci_container_base/depends --mount type=volume,src=ci_macos_cross_previous_releases,dst=/ci_container_base/prev_releases --env-file /tmp/env --name ci_macos_cross ci_macos_cross
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
time="2023-09-27T20:55:39Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Error: creating container storage: the container name "ci_macos_cross" is already in use by e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009. You have to remove that container to be able to reuse that name: that name is already in use

DrahtBot commented at 9:03 AM on September 28, 2023: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	hebasto

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#27793 (ci: label docker images and prune dangling images selectively by stickies-v)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot renamed this:
~~ci: Work around podman stop bug~~
ci: Work around podman stop bug
on Sep 28, 2023

DrahtBot added the label Tests on Sep 28, 2023

maflcko renamed this:
~~ci: Work around podman stop bug~~
ci: Work around podman stop intermittent failure
on Sep 28, 2023

DrahtBot added the label CI failed on Sep 28, 2023

DrahtBot removed the label CI failed on Sep 28, 2023

maflcko commented at 10:59 AM on September 28, 2023: member

Looks like this is working, except that the container id contains the name when the issue happens https://cirrus-ci.com/task/6582091933024256?logs=ci#L250:

Restart docker before run to stop and clear all containers started with --rm
++ podman container stop --all
e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009
++ export MAYBE_REPLACE=--replace
++ MAYBE_REPLACE=--replace
++ echo 'Prune all dangling images'
Prune all dangling images
++ docker image prune --force
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
+++ docker run --cap-add LINUX_IMMUTABLE --rm --interactive --detach --tty --replace --mount type=bind,src=/tmp/cirrus-build-2529187836,dst=/tmp/cirrus-build-2529187836,readonly --mount type=volume,src=ci_macos_cross_ccache,dst=/tmp/ccache_dir --mount type=volume,src=ci_macos_cross_depends,dst=/ci_container_base/depends --mount type=volume,src=ci_macos_cross_previous_releases,dst=/ci_container_base/prev_releases --env-file /tmp/env --name ci_macos_cross ci_macos_cross
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
time="2023-09-28T10:48:22Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
++ CI_CONTAINER_ID='ci_macos_cross
be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265'
++ export CI_CONTAINER_ID
++ export 'CI_EXEC_CMD_PREFIX=docker exec ci_macos_cross
be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265'
++ CI_EXEC_CMD_PREFIX='docker exec ci_macos_cross
be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265'
++ export -f CI_EXEC
++ CI_EXEC rsync --archive --stats --human-readable /tmp/cirrus-build-2529187836/ /ci_container_base
++ docker exec ci_macos_cross be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265 bash -c 'export PATH=/ci_container_base/ci/scratch/bins/:/ci_container_base/ci/retry:$PATH && cd "/ci_container_base" && rsync --archive --stats --human-readable /tmp/cirrus-build-2529187836/ /ci_container_base'
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: crun: executable file `be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
++ echo 'Nothing to copy from /tmp/cirrus-build-2529187836/'
Nothing to copy from /tmp/cirrus-build-2529187836/
++ CI_EXEC /ci_container_base/ci/test/01_base_install.sh
++ docker exec ci_macos_cross be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265 bash -c 'export PATH=/ci_container_base/ci/scratch/bins/:/ci_container_base/ci/retry:$PATH && cd "/ci_container_base" && /ci_container_base/ci/test/01_base_install.sh'
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: crun: executable file `be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
Exit status: 127

DrahtBot added the label CI failed on Oct 4, 2023

maflcko force-pushed on Oct 4, 2023

maflcko commented at 3:32 PM on October 4, 2023: member

ok, pushed the complete set of changes (please refer to the commit messages for details)

maflcko force-pushed on Oct 5, 2023

DrahtBot removed the label CI failed on Oct 5, 2023

maflcko added this to the milestone 26.0 on Oct 5, 2023

DrahtBot added the label Needs rebase on Oct 9, 2023

ci: Rename 04_install to 02_run_container

This reflects what the script does (docker run ...).

fac9abbf47

ci: Add set -ex to 02_run_container.sh

The same is done by the 06 script.

fa09a031c1

ci: Work around podman stop bug

Force remove any containers, pontentially leaving dangling processes,
which should be fine.

fa695b4df0

ci: move-only CI_CONTAINER_ID to 02_run_container.sh

This limits the scope of the CI_CONTAINER_ID symbol.

Can be reviewed with --color-moved=dimmed-zebra

fa2c894cbb

maflcko force-pushed on Oct 9, 2023

DrahtBot removed the label Needs rebase on Oct 9, 2023

maflcko requested review from hebasto on Oct 11, 2023

hebasto approved

hebasto commented at 9:44 AM on October 12, 2023: member

ACK fa2c894cbb41a64371717139fb3c3ddfb9bb8b19, I have reviewed the code and tested it locally.

fanquake merged this on Oct 13, 2023

fanquake closed this on Oct 13, 2023

maflcko deleted the branch on Oct 13, 2023

Frank-GER referenced this in commit 7859bdc824 on Oct 13, 2023

maflcko commented at 10:59 AM on October 23, 2023: member

If the issue persists, I guess cherry-picking https://github.com/bitcoin/bitcoin/commit/4444a114456fdf75b25ebf98c4e169853b670d85 can be another attempt.

bitcoin locked this on Dec 3, 2024