depends cache is not invalidated when the compiler changes

MarcoFalke commented at 11:41 am on October 25, 2019: member

The depends cache has no mechanism of being invalidated when the cpu architecture, the operating system, or even just the compiler changes from down under it.

I think long term we should be moving to proper compiler caches like ccache (#17103) or guix

MarcoFalke added the label Bug on Oct 25, 2019

MarcoFalke added the label Build system on Oct 25, 2019

MarcoFalke commented at 11:43 am on October 25, 2019: member

For example, just one of the bugs I ran into due to this: #17233 (comment)

MarcoFalke commented at 11:43 am on October 25, 2019: member

@dongcarl @theuni Any thoughts or objections?

elichai commented at 10:20 am on December 5, 2019: contributor

How much time penalty we get by removing these two lines? https://github.com/bitcoin/bitcoin/blob/master/.travis.yml#L35:L36

We could also mount the ccache dir to the dockers and install+run ccache in the dockers. that way we only cache the ccache dir

laanwj commented at 10:28 am on December 5, 2019: member

How much time penalty we get by removing these two lines? https://github.com/bitcoin/bitcoin/blob/master/.travis.yml#L35:L36

What would that do? If it causes a depends rebuild on every travis run, especially with the runs with Qt included, you’re looking at hour(s).

elichai commented at 11:05 am on December 5, 2019: contributor

How much time penalty we get by removing these two lines? https://github.com/bitcoin/bitcoin/blob/master/.travis.yml#L35:L36

What would that do? If it causes a depends rebuild on every travis run, especially with the runs with Qt included, you’re looking at hour(s).

even with ccache?

laanwj commented at 11:36 am on December 5, 2019: member

I don’t think ccache really helps that much with building qt. Mind that the qt build first builds a build system (qmake) for the host, then uses that to build the library for the target. Boost has a similar weird build setup. Even getting it to use ccache consistently will be a challenge, and after that you’re still left with enough overhead.

MarcoFalke referenced this in commit 447f8676b2 on Apr 16, 2020

sidhujag referenced this in commit 966c2e74c0 on Apr 17, 2020

MarcoFalke commented at 3:38 pm on May 30, 2020: member

FYI. On the DrahtBot guix build, I am caching depends under a key that is equal to the hash of the whole guix tree. The same can probably not be done for depends and only to a lesser extent to gitian.

dongcarl commented at 5:46 pm on February 16, 2021: member

Following up on this after the merge of #20629

For Guix builds

5200929bfe26c549d7da92c0adf8adf61e143416 ensures that the GUIX_ENVIRONMENT env var is inclueded in the id string used for cache invalidation. This means that any change in the definition of our Guix packages will appropriately invalidate the depends cache

For non-Guix builds

4c7d41858821e4fecf7cb0cec3fcad002365e6c9 adds a lot of toolchain information to the id string in order to trigger cache invalidation should any of it changes. Not as perfect as the situation with Guix, but I believe it’s a good enough approximation for most use-cases.

MarcoFalke commented at 9:18 am on February 17, 2021: member

kewl, thx. Closing for now

MarcoFalke closed this on Feb 17, 2021

DrahtBot locked this on Aug 18, 2022

depends cache is not invalidated when the compiler changes #17248

For Guix builds

For non-Guix builds