ci: Increase CCACHE_SIZE in some builds on Travis #19519

hebasto commented at 5:44 PM on July 14, 2020: member

This PR will decrease build time for the longest Travis builds. Up to ~20 min improvement with warmed up cache is expected.

ci: Increase CCACHE_SIZE in some builds on Travis fdf697fe75

DrahtBot commented at 7:01 PM on July 14, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#16546 (External signer support - Wallet Box edition by Sjors)
#15382 (util: add RunCommandParseJSON by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Tests on Jul 14, 2020

hebasto commented at 9:37 PM on July 15, 2020: member

@MarcoFalke Mind looking into this?

MarcoFalke commented at 6:19 AM on July 16, 2020: member

why is tsan and asan not bumped?

hebasto commented at 6:59 AM on July 16, 2020: member

why is tsan and msan not bumped?

TSan build is made on Cirrus CI which I did not touch in this PR. Or did you mean something else?

fjahr commented at 8:05 PM on July 21, 2020: member

Concept ACK, speeding up Travis is always good :)

Could you give a little more context on how you ended up with 200M? I guess you tested different values? Are there any downsides to setting them to 300M like on the macOS build? I am not familiar with all the limitations of the Travis environment.

hebasto commented at 2:49 AM on July 22, 2020: member

@fjahr

Could you give a little more context on how you ended up with 200M? I guess you tested different values? Are there any downsides to setting them to 300M like on the macOS build? I am not familiar with all the limitations of the Travis environment.

ccache statistics has "cache hit rate" and "cleanups performed". The idea is to make the former >90%, and make the latter about zero for warmed up cache.

fanquake commented at 8:04 AM on July 28, 2020: member

@MarcoFalke can you weigh in here again?

in ci/test/00_setup_env_native_fuzz.sh:18 in fdf697fe75

  14 | @@ -15,3 +15,4 @@ export RUN_FUNCTIONAL_TESTS=false
  15 |  export RUN_FUZZ_TESTS=true
  16 |  export GOAL="install"
  17 |  export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,address,undefined CC=clang CXX=clang++"
  18 | +export CCACHE_SIZE=200M

MarcoFalke commented at 8:11 AM on July 28, 2020:

Effective size is now 114 MB:

https://travis-ci.org/github/bitcoin/bitcoin/jobs/708074351#L2863

MarcoFalke commented at 11:53 AM on July 28, 2020:

Still 120: https://travis-ci.org/github/bitcoin/bitcoin/jobs/712518048#L2866

in ci/test/00_setup_env_native_fuzz_with_valgrind.sh:19 in fdf697fe75

  15 | @@ -16,3 +16,4 @@ export RUN_FUZZ_TESTS=true
  16 |  export FUZZ_TESTS_CONFIG="--valgrind"
  17 |  export GOAL="install"
  18 |  export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer CC=clang CXX=clang++"
  19 | +export CCACHE_SIZE=200M

MarcoFalke commented at 8:13 AM on July 28, 2020:

Effecitve size is 93 MB:

https://travis-ci.org/github/bitcoin/bitcoin/jobs/708074353#L2875

How is it possible that this gives a speedup, when the default size is 100 MB?

MarcoFalke commented at 11:53 AM on July 28, 2020:

Still 95 after two runs: https://travis-ci.org/github/bitcoin/bitcoin/jobs/712518050#L2879

in ci/test/00_setup_env_native_msan.sh:23 in fdf697fe75

  19 | @@ -20,3 +20,4 @@ export GOAL="install"
  20 |  export BITCOIN_CONFIG="--enable-wallet --with-sanitizers=memory --with-asm=no --prefix=${BASE_ROOT_DIR}/depends/x86_64-pc-linux-gnu/ CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}' BDB_LIBS='-L${BDB_PREFIX}/lib -ldb_cxx-4.8' BDB_CFLAGS='-I${BDB_PREFIX}/include'"
  21 |  export USE_MEMORY_SANITIZER="true"
  22 |  export RUN_FUNCTIONAL_TESTS="false"
  23 | +export CCACHE_SIZE=250M

MarcoFalke commented at 8:13 AM on July 28, 2020:

Effective size is now 150MB:

https://travis-ci.org/github/bitcoin/bitcoin/jobs/708074350#L3511

MarcoFalke commented at 11:54 AM on July 28, 2020:

Still 150 after two runs: https://travis-ci.org/github/bitcoin/bitcoin/jobs/712518047#L3500

MarcoFalke approved

MarcoFalke commented at 8:13 AM on July 28, 2020: member

No objection ACK

hebasto commented at 8:20 AM on July 28, 2020: member

@MarcoFalke

The data about cache size are for not fully warmed up cache. The next commit on top of this branch will increase cache usage.

Also please see my #19519 (comment)

MarcoFalke commented at 8:24 AM on July 28, 2020: member

Ok, I will monitor how the cache size evolves. We can always revert if needed

MarcoFalke merged this on Jul 28, 2020

MarcoFalke closed this on Jul 28, 2020

hebasto deleted the branch on Jul 28, 2020

MarcoFalke commented at 12:19 PM on July 28, 2020: member

The data about cache size are for not fully warmed up cache

The cache should be warm after the first compile. Bitcoin Core isn't written in LaTeX, luckily ;)

sidhujag referenced this in commit 071d9f8b3e on Jul 28, 2020

hebasto commented at 7:36 AM on July 29, 2020: member

@MarcoFalke

From my experience the build time depends on the "cleanups performed" value from ccache statistics. Each time the zero value of this parameter had been achieved the difference between "cache size" and "max cache size" becomes more or less substantial. It seems related to ccache internals.

DrahtBot locked this on Feb 15, 2022