ci: Define cases when invalidate vcpkg binary cache #23329

hebasto commented at 7:30 PM on October 20, 2021: member

On master (c8bae2be341c921823eee95a9eec7e2b74f2f0ae), the size of the vcpkg binary cache is potentially unbounded.

The reason of such behavior is the internal caching logic, following which, any change in the set of parameters defined by the vcpkg implementation will cause compiling of a new binaries following by adding them to the current cache.

This PR defines two obvious cases when the vcpkg binary cache will be invalidated.

ci, refactor: Rename VCPKG_TAG variable and vcpkg_cache script

The VCPKG_TAG variable renamed to CI_VCPKG_TAG to prevent any possible
name clash with vcpkg-specific variables.

The vcpkg_cache script renamed into more meaningful one.

b00646bc77

ci: Improve vcpkg binary cache settings

This change comes with two improvements:
1) using the VCPKG_DEFAULT_BINARY_CACHE variable drops dependency on
vcpkg default cached archives location, and improves readability
2) two obvious cases when binary cache is invalidated are defined, that
guaranties it won't grow boundlessly when, for example, vcpkg has being
updated.

e8692cf2c1

hebasto added the label Tests on Oct 20, 2021

hebasto added the label Windows on Oct 20, 2021

fanquake commented at 11:22 PM on October 20, 2021: member

Isn't this almost the same as #23215? Why can't it be done in that PR? It's nearly impossible to keep track of the constant changes for these Windows CIs.

hebasto commented at 6:30 AM on October 21, 2021: member

It's nearly impossible to keep track of the constant changes for these Windows CIs.

Sorry about that.

Isn't this almost the same as #23215?

Yes, this PR is split from #23215.

Why can't it be done in that PR?

That was my initial idea when I opened #23215. But it created a confusion (at least, I have such an impression), that the goal of such combined PR was introducing a vcpkg binary caching. It is not the case. So I decided to leave #23215 solely on purpose of introducing of vcpkg downloads cache which has completely different goal in comparison to binary cache.

Therefore, this PR is focused on an improvement of already used vcpkg binary caching feature.

And #23215 will be focused on an introducing a new vcpkg downloads cache.

hebasto commented at 6:30 AM on October 21, 2021: member

Friendly ping @sipsorcery :tiger2:

sipsorcery commented at 5:52 PM on October 21, 2021: member

@hebasto are you sure that you're caching the correct vcpkg directory?

It's my understanding that the C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives will hold the zipped source files that will then need to be built.

On my local machine the vcpkg manifest mechanism puts the built binaries in build_msvc/vcpkg_installed.

hebasto commented at 8:24 PM on October 21, 2021: member

@sipsorcery

@hebasto are you sure that you're caching the correct vcpkg directory?

Yes, I am :tiger2:

One could track any dependency package (berkeleydb, for example):

https://github.com/bitcoin/bitcoin/commit/16df28c3e3617814cd48c05120a4d766f1480c74, the binary cache is unpopulated, i.e., C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives is empty:

In the build log one can see:

...
  The following packages will be built and installed:
      berkeleydb[core]:x64-windows-static -> 4.8.30#5
...
  Could not locate cached archive: C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives\62\62c79c2f0c0d9d79208917f62e9cc30952a57354.zip
...
  Starting package 1/80: berkeleydb:x64-windows-static
  Building package berkeleydb[core]:x64-windows-static...
  -- Downloading http://download.oracle.com/berkeley-db/db-4.8.30.NC.zip -> db-4.8.30.NC.zip...
  -- Extracting source C:/Users/ContainerAdministrator/AppData/Local/Temp/vcpkg/downloads/db-4.8.30.NC.zip
  -- Applying patch fix-conflict-macro.patch
  -- Using source at C:/Users/ContainerAdministrator/AppData/Local/Temp/vcpkg/buildtrees/berkeleydb/src/db-4-87fc9a5730.clean
  -- Found external ninja('1.10.2').
  -- Configuring x64-windows-static
  -- Building x64-windows-static-rel
  -- Installing: C:/Users/ContainerAdministrator/AppData/Local/Temp/vcpkg/packages/berkeleydb_x64-windows-static/share/berkeleydb/copyright
  -- Performing post-build validation
  -- Performing post-build validation done
  Stored binary cache: C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives\62\62c79c2f0c0d9d79208917f62e9cc30952a57354.zip
  Building package berkeleydb[core]:x64-windows-static... done
  Installing package berkeleydb[core]:x64-windows-static...
  Installing package berkeleydb[core]:x64-windows-static... done
  Elapsed time for package berkeleydb:x64-windows-static: 52.01 s
...

https://github.com/bitcoin/bitcoin/commit/a7f28af4372a9d6ebe1c596ec455cbeac703f69d, now the binary cache is fully populated, and in the build log:

...
  The following packages will be built and installed:
      berkeleydb[core]:x64-windows-static -> 4.8.30#5
...
  Using cached binary package: C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives\62\62c79c2f0c0d9d79208917f62e9cc30952a57354.zip
...
  Starting package 1/80: berkeleydb:x64-windows-static
  Building package berkeleydb[core]:x64-windows-static...
  Building package berkeleydb[core]:x64-windows-static... done
  Installing package berkeleydb[core]:x64-windows-static...
  Installing package berkeleydb[core]:x64-windows-static... done
  Elapsed time for package berkeleydb:x64-windows-static: 27.04 ms
...

It's my understanding that the C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives will hold the zipped source files that will then need to be built.

Actually, the C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives\62\62c79c2f0c0d9d79208917f62e9cc30952a57354.zip archive also contains a compiled lib\libdb48.lib.

On my local machine the vcpkg manifest mechanism puts the built binaries in build_msvc/vcpkg_installed.

Correct. But this action seems orthogonal to the vcpkg "Binary Caching" mechanism.

sipsorcery commented at 8:52 PM on October 21, 2021: member

@hebasto are you sure that you're caching the correct vcpkg directory?

Yes, I am 🐅

I stand corrected. I did a build, deleted the build_msvc/vcpkg_installed directory, built again and the binary dependencies were copied from C:\Users\aaron\AppData\Local\vcpkg\archives.

Am I correct that the CI snippet below deletes and re-creates the %VCPKG_DEFAULT_BINARY_CACHE% directory is the msbuild version changes? And that's the mechanism to stop it growing when the vcpkg dependencies need to get rebuilt?

  vcpkg_binary_cache:
    folder: '%VCPKG_DEFAULT_BINARY_CACHE%'
    reupload_on_changes: true
    fingerprint_script:
      - echo %CI_VCPKG_TAG%
      - msbuild -version
    populate_script:
      - mkdir %VCPKG_DEFAULT_BINARY_CACHE%

hebasto commented at 9:16 PM on October 21, 2021: member

@sipsorcery

Am I correct that the CI snippet below deletes and re-creates the %VCPKG_DEFAULT_BINARY_CACHE% directory is the msbuild version changes?

To be precise, an old cache is not loaded, rather "deleted". Cases for such cache invalidation are any of the following: CI_VCPKG_TAG is changed and/or msbuild -version is changed.

And that's the mechanism to stop it growing when the vcpkg dependencies need to get rebuilt?

Yes, it is. But only in two explicit cases: CI_VCPKG_TAG is changed and/or msbuild -version is changed.

FWIW, usual binary cache expected size is 83 MB, on master (8a083bc5b57ff98f2e3c550f1b10dba202e3aa79) it is 334 MB now.

Just in case: https://cirrus-ci.org/guide/writing-tasks/#cache-instruction

sipsorcery commented at 9:56 PM on October 21, 2021: member

ACK e8692cf2c151d2abc206ca699e04ae05d4c31dcd.

fanquake merged this on Oct 22, 2021

fanquake closed this on Oct 22, 2021

hebasto deleted the branch on Oct 22, 2021

fanquake referenced this in commit 04437ee721 on Oct 23, 2021

DrahtBot locked this on Oct 30, 2022