ci: Invalidate depends caches when sources have been changed #22710

1. 
   hebasto commented at 8:46 pm on August 15, 2021: member

   On master (502d22ceed1f90ed41336260f8eb428d3acaf514) the Cirrus CI do not invalidate depends caches when their sources has been changed, i.e. source version updates, a new dependency etc.

   This behavior causes:

   • breaking CI build for Android APK (see #22708)
   • the cache sizes growing

   It is assumed that caches could be invalidated via Cirrus CI API/GUI, but it is inconvenient and it does not work as expected in all cases.

   The common part of fingerprint_keys for both depends_sources_cache and depends_built_cache is the recent commit that changes sources in the src/depends sub-directory.

   For depends_built_cache additionally CIRRUS_TASK_NAME is used to avoid sharing of the same cache between tasks with different OSes and different DEP_OPTS.

   ---

   The depends_sources cache:

   • in master (502d22ceed1f90ed41336260f8eb428d3acaf514)

   

   • with this PR

   

2. DrahtBot added the label Tests on Aug 15, 2021
3. hebasto force-pushed on Aug 16, 2021
4. hebasto renamed this:
   [WIP] ci: Invalidate depends sources cache
   ci: Invalidate depends caches when sources have been changed
   on Aug 16, 2021
5. hebasto marked this as ready for review on Aug 16, 2021
6. in .cirrus.yml:218 in d6fb7768e4 outdated

   214@@ -213,6 +215,7 @@ task:
   215   << : *DEPENDS_SDK_CACHE_TEMPLATE
   216   depends_sources_cache:
   217     folder: "depends/sources"
   218+    fingerprint_script: echo $SOURCES_CACHE_FINGERPRINT_KEY
   

   ---

   ---

   

   MarcoFalke commented at 8:23 am on August 16, 2021:

   why not git log -1 ./depends?

   ---

   

   hebasto commented at 8:25 am on August 16, 2021:

   Should we invoke it within a Docker image, i.e., with DOCKER_EXEC?

   ---

   

   MarcoFalke commented at 8:29 am on August 16, 2021:

   It should work as long as BASE_TEMPLATE is run before git is called

   ---

   

   hebasto commented at 8:47 am on August 16, 2021:

   Thanks! Done in the recent push.

   ---

   

   hebasto commented at 1:37 pm on August 16, 2021:

   @MarcoFalke

   why not git log -1 ./depends?

   It works fine for depends_sources_cache, but not for depends_built_cache as the the actual content of the latter depends on the DEP_OPTS variable.

   ---

   

   hebasto commented at 3:43 pm on August 16, 2021:

   It works fine for depends_sources_cache, but not for depends_built_cache as the the actual content of the latter depends on the DEP_OPTS variable.

   Addressed in the recent push. OP has been updated.

7. hebasto force-pushed on Aug 16, 2021
8. 
   jonatack commented at 9:06 am on August 16, 2021: member

   Concept ACK
9. ci: Reorder scripts to make git available before depends_sources_cache 939640f87e
10. hebasto force-pushed on Aug 16, 2021
11. hebasto force-pushed on Aug 16, 2021
12. 
    Zero-1729 commented at 4:11 pm on August 16, 2021: contributor

    Approach ACK
13. 
    hebasto commented at 6:25 pm on August 16, 2021: member

    CI is green, and this PR is ready for reviewing :)
14. 
    DrahtBot commented at 6:41 pm on August 16, 2021: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #22708 ([PoC] build, qt: Add Wayland support for Linux builds with depends by hebasto)
    • #21652 ([WIP NOMERGE DRAFT] ci: Switch more tasks to self-hosted by MarcoFalke)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

15. ci: Invalidate depends caches when sources have been changed f52a72af56
16. in .cirrus.yml:43 in 93f3427a47 outdated

    39@@ -41,9 +40,14 @@ global_task_template: &GLOBAL_TASK_TEMPLATE
    40     folder: "/tmp/ccache_dir"
    41   depends_built_cache:
    42     folder: "depends/built"
    43+    fingerprint_script: echo $CIRRUS_TASK_NAME $(git log -1 --oneline ./depends)
    

    ---

    ---

    

    laanwj commented at 12:55 pm on August 18, 2021:

    Might use git rev-list -1 HEAD ./depends here; this prints only the (full, independent of core.abbrev) commit hash.

    ---

    

    hebasto commented at 4:36 pm on August 18, 2021:

    Thanks! Updated.
17. hebasto force-pushed on Aug 18, 2021
18. 
    hebasto commented at 4:35 pm on August 18, 2021: member

    Updated 93f3427a47bf1761e4c66f9f6c5a0c09728e3a0e -> f52a72af569526f319d9f1480eb473b9570125c3 (pr22710.04 -> pr22710.05, diff):

    • addressed @laanwj’s comment
19. 
    MarcoFalke commented at 12:08 pm on August 19, 2021: member

    cr ACK f52a72af569526f319d9f1480eb473b9570125c3
20. 
    Zero-1729 commented at 12:58 pm on August 19, 2021: contributor

    crACK f52a72af569526f319d9f1480eb473b9570125c3
21. MarcoFalke merged this on Aug 19, 2021
22. MarcoFalke closed this on Aug 19, 2021

    ---

23. hebasto deleted the branch on Aug 19, 2021
24. MarcoFalke referenced this in commit 0263583ad9 on Aug 20, 2021
25. sidhujag referenced this in commit 9a20bb94a9 on Aug 20, 2021
26. sidhujag referenced this in commit 1215cacb4f on Aug 20, 2021
27. DrahtBot locked this on Aug 19, 2022

Contributors
hebasto MarcoFalke jonatack Zero-1729 DrahtBot laanwj

Labels
Tests