depends: `local_dir` cache invalidation can miss source changes #35764

Is there an existing issue for this?

• I have searched the existing issues

Current behaviour

Depends caches packages using $(package)_local_dir as a tarball in SOURCES_PATH. The tarball is currently recreated only when find -newer reports a path newer than the cached archive.

This does not reliably identify directory contents:

• Two worktrees sharing SOURCES_PATH can contain different source trees.
• A tarball created by one worktree may be newer than every path in another.
• Deleting a file leaves no path whose mtime can be checked.
• Changed or restored files may have older or equal mtimes.
• Timestamp granularity and clock differences make ordering unreliable.

native_libmultiprocess is currently the only package using local_dir.

A shared SOURCES_PATH may contain e.g. src-ipc-libmultiprocess.tar generated from another worktree. If none of the current worktree's paths are newer, Depends reuses that archive.

The fetched stamp then correctly hashes the stale archive, and its checksum correctly affects the package build ID. However, the package is built from the wrong explicit source input.

Clearing BASE_CACHE or disabling Guix substitutes does not help because the stale input remains in SOURCES_PATH.

Expected behaviour

Local source identity should change when any of the following changes:

• A file is added, removed, or modified.
• File permissions, including the executable bit, change.
• A symlink target changes.
• An empty directory is added or removed.

Filesystem timestamps, ownership paths, and absolute worktree locations should not determine the identity. Identical trees in separate worktrees should produce the same archive checksum.

Different trees using the same shared SOURCES_PATH should be able to coexist.

Steps to reproduce

Tricky to define clearly here, but I'll try:

1. Have two Bitcoin Core worktrees on different branches.
2. Both worktrees use the same Depends SOURCES_PATH.
3. The branches contain different versions of src/ipc/libmultiprocess.
4. Both worktrees already exist before either build starts, so their source-file mtimes are older than the next archive created.
5. Build Depends in worktree A. This creates src-ipc-libmultiprocess.tar in the shared source cache.
6. Then build Depends in worktree B.
7. The cached tarball is newer than every path in worktree B, so find -newer reports no changes.
8. Depends reuses worktree A’s tarball even though worktree B has different contents.
9. The checksum and build ID are calculated correctly—but for A’s stale archive.
10. Worktree B therefore reproducibly builds the wrong libmultiprocess source.

The same problem can happen without two worktrees when a file is deleted or restored with an older mtime: the directory contents changed, but find -newer cannot detect it.

Relevant log output

No response

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

master branch, pr-35261

Operating system and version

NixOS 26.05

Machine specifications

No response

cc @ryanofsky

Thanks I wasn't aware of SOURCES_PATH when I implemented this so it makes sense to handle it better.

A consideration here is that the tarball is refreshed while parsing the makefile every time make is run, even if make is only invoked to do something simple like print a variable (make print-build_CC). So the point of the find -newer check is to be fast, while still detecting changes as well as normal make commands using mtimes. I think the implementation handles this pretty well, because all the changes listed above in the "when any of the following changes" list will cause mtimes to be updated except file permissions changes, which won't trigger updates in normal make commands either.

But the current implementation doesn't handle SOURCES_PATH well at all, and a logical extension of the original approach that handles it better would be to use different tarballs for different worktrees. Following patch hashes worktree paths and adds them to tarball filenames (can test with make print-native_libmultiprocess_file_name) to implement this:

<details><summary>diff</summary> <p>

--- a/depends/funcs.mk
+++ b/depends/funcs.mk
@@ -43,7 +43,10 @@ endef
 # Shell script to create a source tarball in $(1)_source from local directory
 # $(1)_local_dir instead of downloading remote sources. Tarball is recreated if
 # any paths in the local directory have a newer mtime, and checksum of the
-# tarball is saved to $(1)_fetched and returned as output.
+# tarball is saved to $(1)_fetched and returned as output. The tarball filename
+# includes a hash of the local directory's absolute path (see
+# int_get_build_properties) so different worktrees using a shared SOURCES_PATH
+# get distinct tarballs and do not interfere with each other.
 define fetch_local_dir_sha256
     if ! [ -f $($(1)_source) ] || [ -n "$$(find $($(1)_local_dir) -newer $($(1)_source) | head -n1)" ]; then \
         mkdir -p $(dir $($(1)_source)) && \
@@ -102,6 +105,11 @@ $(1)_source_dir:=$(SOURCES_PATH)
 # If $(1)_file_name is empty and $(1)_local_dir is nonempty, set file name to a
 # .tar file with a friendly filename named after the directory path.
 $(if $($(1)_file_name),,$(if $($(1)_local_dir),$(eval $(1)_file_name:=$(call int_friendly_file_name,$($(1)_local_dir)).tar)))
+# Append a hash of the local directory's absolute path so that different
+# worktrees sharing a SOURCES_PATH get distinct tarballs. find -newer still
+# detects source changes within a worktree.
+$(if $($(1)_local_dir),$(eval $(1)_local_dir_hash:=$(shell (cd $($(1)_local_dir) && pwd) | $(build_SHA256SUM) | cut -c1-$(HASH_LENGTH))))
+$(if $($(1)_local_dir_hash),$(eval $(1)_file_name:=$(basename $($(1)_file_name))-$($(1)_local_dir_hash).tar))
 $(1)_source:=$$($(1)_source_dir)/$($(1)_file_name)
 $(1)_download_dir:=$(base_download_dir)/$(1)-$($(1)_version)
 $(1)_prefixbin:=$($($(1)_type)_prefix)/bin/

</p> </details>

#35765 could also be an good approach if it's ok to hash the whole subtree each time make is invoked, which could be the case.

Thanks, hashing the worktree path does indeed seem a reasonable (smaller) fix for the SOURCES_PATH collision.

My preference for content addressing (approach in #35765) is that even #35618 plus a path hash still retains mtime-based invalidation, which misses mode-only changes, modifies checkout mtimes, and gives identical trees at different paths separate source archives. It also doesn't make things totally atomic.

The current local tree is a only few hundred KB, and creating the normalized archive takes about 30 ms (for me, locally), so the stronger content-based identity seems pretty affordable to me?

That said, the path hash is a fine fallback if avoiding parse-time I/O, or simply having the smallest LoC change possible. I'm ~ happy with either.

TBH, I'm pretty surprised I didn't run into this sooner with how many guix builds I do with a single shared SOURCES_PATH...