ci: Print ccache stats, add pip cache, and cleanups #18581

hebasto commented at 12:55 AM on April 10, 2020: member

The Travis pip cache is free and saves a dozen of seconds :)

Here are some excerpts from the Travis logs with ccache statistics (I found useful):

Arm64

ccache version 3.6
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr 10 18:21:06 2020
stats zeroed                        Fri Apr 10 18:03:04 2020
cache hit (direct)                   156
cache hit (preprocessed)             198
cache miss                           143
cache hit rate                     71.23 %
called for link                        8
cleanups performed                     9
files in cache                      1255
cache size                          80.6 MB
max cache size                     100.0 MB

s390x

ccache version 3.4.1
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats zero time                     Fri Apr 10 17:15:35 2020
cache hit (direct)                   115
cache hit (preprocessed)             163
cache miss                            94
cache hit rate                     74.73 %
called for link                        6
cleanups performed                     0
files in cache                       804
cache size                         304.8 MB
max cache size                       5.0 GB

Win64

ccache version 3.4.1
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats zero time                     Fri Apr 10 17:38:07 2020
cache hit (direct)                   147
cache hit (preprocessed)             199
cache miss                           138
cache hit rate                     71.49 %
called for link                        7
cleanups performed                     7
files in cache                      1242
cache size                          87.0 MB
max cache size                     100.0 MB

CentOS 7

ccache version 3.7.7
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr 10 17:45:59 2020
stats zeroed                        Fri Apr 10 17:34:27 2020
cache hit (direct)                   152
cache hit (preprocessed)             200
cache miss                           144
cache hit rate                     70.97 %
called for link                        8
cleanups performed                     4
files in cache                      1352
cache size                          86.1 MB
max cache size                     100.0 MB

bionic

ccache version 3.4.1
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats zero time                     Fri Apr 10 17:21:10 2020
cache hit (direct)                   136
cache hit (preprocessed)              35
cache miss                           330
cache hit rate                     34.13 %
called for link                        8
cleanups performed                    18
files in cache                      1302
cache size                          83.8 MB
max cache size                     100.0 MB

xenial

ccache version 3.2.4
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
cache hit (direct)                   161
cache hit (preprocessed)             174
cache miss                            95
called for link                        7
files in cache                      3411
cache size                           1.1 GB
max cache size                       5.0 GB

focal

ccache version 3.7.7
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Fri Apr 10 17:35:57 2020
stats zeroed                        Fri Apr 10 17:21:32 2020
cache hit (direct)                    78
cache hit (preprocessed)             125
cache miss                           107
cache hit rate                     65.48 %
called for link                      120
cleanups performed                     0
files in cache                      6218
cache size                           1.8 GB
max cache size                       5.0 GB

macOS 10.12

ccache version 3.4.1
cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats zero time                     Fri Apr 10 17:47:19 2020
cache hit (direct)                    28
cache hit (preprocessed)               1
cache miss                           469
cache hit rate                      5.82 %
called for link                        8
cleanups performed                    17
files in cache                      1946
cache size                          82.6 MB
max cache size                     100.0 MB

macOS 10.14 native

ccache version 3.7.8
cache directory                     /Users/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
primary config                      /Users/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
secondary config      (readonly)    /usr/local/Cellar/ccache/3.7.8/etc/ccache.conf
stats updated                       Fri Apr 10 17:38:44 2020
stats zeroed                        Fri Apr 10 17:22:36 2020
cache hit (direct)                   213
cache hit (preprocessed)             293
cache miss                           144
cache hit rate                     77.85 %
called for link                       11
called for preprocessing              56
compile failed                        30
preprocessor error                    56
bad compiler arguments                14
autoconf compile/link                 77
no input file                         66
cleanups performed                     0
files in cache                      1564
cache size                         284.6 MB
max cache size                       5.0 GB

fanquake added the label Tests on Apr 10, 2020

promag commented at 11:33 AM on April 10, 2020: member

ACK, looks good to me.

in ci/test/00_setup_env.sh:56 in b1f7d35add outdated

  52 | @@ -53,7 +53,6 @@ export DEPENDS_DIR=${DEPENDS_DIR:-$BASE_ROOT_DIR/depends}
  53 |  export BASE_OUTDIR=${BASE_OUTDIR:-$BASE_SCRATCH_DIR/out/$HOST}
  54 |  export PREVIOUS_RELEASES_DIR=${PREVIOUS_RELEASES_DIR:-$BASE_ROOT_DIR/releases/$HOST}
  55 |  export SDK_URL=${SDK_URL:-https://bitcoincore.org/depends-sources/sdks}
  56 | -export WINEDEBUG=${WINEDEBUG:-fixme-all}

MarcoFalke commented at 12:05 PM on April 10, 2020:

Another use:

ci/test/04_install.sh:env | grep -E '^(BITCOIN_CONFIG|BASE_|QEMU_|CCACHE_|WINEDEBUG|LC_ALL|BOOST_TEST_RANDOM|CONFIG_SHELL|(ASAN|LSAN|TSAN|UBSAN)_OPTIONS|TEST_PREVIOUS_RELEASES|PREVIOUS_RELEASES_DIR)' | tee /tmp/env

hebasto commented at 6:29 PM on April 10, 2020:

Fixed.

in ci/test/06_script_a.sh:10 in b1f7d35add outdated

   6 | @@ -7,6 +7,7 @@
   7 |  export LC_ALL=C.UTF-8
   8 |  
   9 |  BITCOIN_CONFIG_ALL="--disable-dependency-tracking --prefix=$DEPENDS_DIR/$HOST --bindir=$BASE_OUTDIR/bin --libdir=$BASE_OUTDIR/lib"
  10 | +DOCKER_EXEC command -v ccache > /dev/null && ccache --zero-stats

MarcoFalke commented at 12:10 PM on April 10, 2020:

DOCKER_EXEC command -v ccache \> /dev/null && ccache --zero-stats

MarcoFalke commented at 12:10 PM on April 10, 2020: member

Telling from the version, it looks like you are executing ccache on the host, not in the docker

ci: Drop unused WINEDEBUG variable a6eea715ff

ci: Print ccache statistics summary 6d480667b2

ci: Add pip cache 73f8303629

hebasto force-pushed on Apr 10, 2020

MarcoFalke commented at 5:17 PM on April 10, 2020: member

Could update OP with the new ccache stats, since they are now stale?

hebasto commented at 5:21 PM on April 10, 2020: member

Could update OP with the new ccache stats, since they are now stale?

Sure. Waiting for the Travis build in the bitcoin repo :)

hebasto commented at 6:28 PM on April 10, 2020: member

Updated b1f7d35addccbf2ea5ae37fc162e1973e3969fa3 -> 73f83036298e4932772704b2d5359c78d81bd7cc (pr18581.01 -> pr18581.02, diff):

addressed @MarcoFalke's comment

Another use:

ci/test/04_install.sh:env | grep -E '^(BITCOIN_CONFIG|BASE_|QEMU_|CCACHE_|WINEDEBUG|LC_ALL|BOOST_TEST_RANDOM|CONFIG_SHELL|(ASAN|LSAN|TSAN|UBSAN)_OPTIONS|TEST_PREVIOUS_RELEASES|PREVIOUS_RELEASES_DIR)' | tee /tmp/env

and this one:

Telling from the version, it looks like you are executing ccache on the host, not in the docker

hebasto commented at 6:29 PM on April 10, 2020: member

Could update OP with the new ccache stats, since they are now stale?

Updated.

MarcoFalke merged this on Apr 13, 2020

MarcoFalke closed this on Apr 13, 2020

sidhujag referenced this in commit a0875dcb38 on Apr 13, 2020

hebasto deleted the branch on Apr 14, 2020

DrahtBot locked this on Feb 15, 2022