ci: Enable pipefail in 03_test

maflcko commented at 12:45 PM on March 11, 2026: member

The CI script is problematic, because it is written in Bash, without pipefail enabled. Thus, some failures are silently ignored.

Enabling pipefail is a bit tedious, because:

The IWYU task has no (--verbose) ccache output, so the pipe fails after grep [1]. Also, right now on master, the if silently skips: ci/test/03_test_script.sh: line 122: [: : integer expression expected.
The Alpine task has Hits: twice in the output, so the pipe fails after head -1 [2]

Not sure what the easiest way to fix this would be. Some options are:

Just use tail -1 and 0 as fallback: hit_rate=$(ccache --show-stats | grep "Hits:" | tail -1 | sed 's/.*($.*$%).*/\1/' || echo "0")
Properly parse, using Python and --print-stats (this pull)

[1]

+ ccache --version
+ head -n 1
ccache version 4.11.2
+ ccache --show-stats --verbose
Cache directory:    /home/admin/actions-runner/_work/_temp/ccache_dir
Config file:        /home/admin/actions-runner/_work/_temp/ccache_dir/ccache.conf
System config file: /etc/ccache.conf
Stats updated:      Tue Feb 17 08:40:20 2026
Local storage:
  Cache size (GB):  0.0 / 2.0 ( 0.00%)
  Files:              0
  Hits:               0
  Misses:             0
  Reads:              0
  Writes:             0
++ ccache --show-stats
++ grep Hits:
++ head -1
++ sed 's/.*(\(.*\)%).*/\1/'
+ hit_rate=
Command '['docker', 'exec', '--env', 'DANGER_RUN_CI_ON_HOST=1', 'f5e8f319c22101ada5be9d4c5fd7d883ce37b830e86ec64627cb7d2b96749053', '/home/admin/actions-runner/_work/_temp/ci/test/03_test_script.sh']' returned non-zero exit status 1.
Error: Process completed with exit code 1.

[2]

+ ccache --version
+ head -n 1
ccache version 4.12.1
+ ccache --show-stats --verbose
Cache directory:    /home/admin/actions-runner/_work/_temp/ccache_dir
Config file:        /home/admin/actions-runner/_work/_temp/ccache_dir/ccache.conf
System config file: /etc/ccache.conf
Stats updated:      Tue Feb 17 08:40:35 2026
Cacheable calls:     873 / 873 (100.0%)
  Hits:              846 / 873 (96.91%)
    Direct:          822 / 846 (97.16%)
    Preprocessed:     24 / 846 ( 2.84%)
  Misses:             27 / 873 ( 3.09%)
Successful lookups:
  Direct:            822 / 873 (94.16%)
  Preprocessed:       24 /  51 (47.06%)
Local storage:
  Cache size (GB):   2.0 / 2.0 (99.95%)
  Files:            2580
  Cleanups:           13
  Hits:              846 / 873 (96.91%)
  Misses:             27 / 873 ( 3.09%)
  Reads:            1772
  Writes:             52
++ ccache --show-stats
++ grep Hits:
++ head -1
++ sed 's/.*(\(.*\)%).*/\1/'
+ hit_rate=96.91
Command '['docker', 'exec', '--env', 'DANGER_RUN_CI_ON_HOST=1', '272a66a48206f1f6096612e196127ce46ea4dbff5dc14be3a4a20c4ee523956f', '/home/admin/actions-runner/_work/_temp/ci/test/03_test_script.sh']' returned non-zero exit status 141.
Error: Process completed with exit code 1.

DrahtBot added the label Tests on Mar 11, 2026

DrahtBot commented at 12:45 PM on March 11, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/34801.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	willcl-ark
Concept ACK	fanquake
Approach ACK	BrandonOdiwuor

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

maflcko commented at 12:46 PM on March 11, 2026: member

Draft for now, because CI fails here IIRC.

DrahtBot added the label CI failed on Mar 11, 2026

willcl-ark commented at 8:21 PM on March 11, 2026: member

How about:

ccache --version
ccache --show-stats --verbose
ccache --print-stats | python3 -c '
import os
import sys

hits = 0
miss = 0

for line in sys.stdin:
    key, value = line.split("\t", 1)
    if key in ("local_storage_hit", "primary_storage_hit"):
        hits = int(value)
    elif key in ("local_storage_miss", "primary_storage_miss"):
        miss = int(value)

calls = hits + miss
rate = (hits / calls * 100) if calls else 0
print(f"{rate:.2f}")
if rate < 75:
    container = os.environ["CONTAINER_NAME"]
    print(
        "::notice title=low ccache hitrate::"
        f"Ccache hit-rate in {container} was {rate:.2f}%"
    )
'

No possible pipe failures in this non-critical section, I think :)

(slightly more verbose logs, but I think that's fine?)

BrandonOdiwuor commented at 5:55 AM on March 27, 2026: contributor

Approach ACK 0509ac7743b8234964db52ae31464edf2f614ae1

Enabling pipefail in ci/test/03_test_script.sh and replacing the fragile ccache hit-rate shell parsing with a clean Python script (using --print-stats) is a nice cleanup.

fanquake commented at 2:09 PM on May 20, 2026: member

Concept ACK on enabling -pipefail

maflcko force-pushed on May 21, 2026

maflcko removed the label CI failed on May 21, 2026

ci: Enable pipefail in 03_test_script.sh

This requires re-writing the --show-stats parsing to use --print-stats
parsing, to avoid pipefail failures.

fa99a3ccac

maflcko marked this as ready for review on May 21, 2026

maflcko force-pushed on May 21, 2026

DrahtBot added the label CI failed on May 21, 2026

maflcko commented at 4:43 PM on May 21, 2026: member

Replaced key == ... with key in ... and added || true after iwyu.

DrahtBot removed the label CI failed on May 21, 2026

maflcko requested review from willcl-ark on May 22, 2026

in ci/test/03_test_script.sh:142 in fa99a3ccac

 141 | -  echo "::notice title=low ccache hitrate::Ccache hit-rate in $CONTAINER_NAME was $hit_rate%"
 142 | -fi
 143 | +ccache --print-stats | python3 -c '
 144 | +import os
 145 | +import sys
 146 | +

willcl-ark commented at 8:21 AM on May 22, 2026:

Might get some extra safety against future ccache output changes if we init to zero:

hits = 0
miss = 0

maflcko commented at 8:28 AM on May 22, 2026:

I've considered it, but I think we want to know about those failures, instead of silently failing, like before

willcl-ark commented at 8:30 AM on May 22, 2026:

Sounds fine to me

willcl-ark approved

willcl-ark commented at 8:24 AM on May 22, 2026: member

ACK fa99a3ccacead02c26ea74c0e12cd845c4dafb91

Sorry I forgot to circle back for so long.

LGTM. CI output appears as expected, left one comment about future-proofing more, but fixing it if it ever breaks seems fine too.

DrahtBot requested review from fanquake on May 22, 2026

DrahtBot requested review from BrandonOdiwuor on May 22, 2026

fanquake merged this on May 22, 2026

fanquake closed this on May 22, 2026

maflcko deleted the branch on May 22, 2026

maflcko commented at 10:19 AM on May 22, 2026: member

On a general note, it is pretty ugly that ci/test/03_test_script.sh is written in Bash, and not Python (or so). So it is a bit unclean having to call python -c "...." to use clean and simple Python. Though, rewriting this whole script in Python may be non-trivial to review, so I haven't attempted it yet...

ci: Enable pipefail in 03_test_script.sh #34801

Code Coverage & Benchmarks

Reviews