Remove .NOTPARALLEL from depends/Makefile #22694

.NOTPARALLEL prevents make from making independent targets in parallel. This restriction applies only to the work done by the parent make.

Without .NOTPARALLEL running make in the depends directory with -j will allow make to download, extract, build, etc packages in parallel.

This issue is to discuss if .NOTPARALLEL is really required in depends/Makefile. Hopefully, we can remove it.

Tagging @theuni @hebasto @dongcarl since we discussed this a bit

I missed the discussion so am probably missing some context. One potential issue could be that depends build logs become a bit less usable, if they end up being a mashup of overlapping build jobs, rather than packages being built sequentially?

Are there more benefits / depends improvements you have in mind, other than just faster builds, which are dependent on this change? I think the fact that we currently have (slower) sequential builds is somewhat offset by heavy caching, and that packages are updated infrequently. A builder is going to see the most benefit from parallelism when doing an initial build (and download) from scratch.

I missed the discussion so am probably missing some context. One potential issue could be that depends build logs become a bit less usable, if they end up being a mashup of overlapping build jobs, rather than packages being built sequentially?

make has a command line option to synchronize its output. See https://www.gnu.org/software/make/manual/make.html#Terminal-Output.

Are there more benefits / depends improvements you have in mind, other than just faster builds, which are dependent on this change?

No, just performance.

I think the fact that we currently have (slower) sequential builds is somewhat offset by heavy caching, and that packages are updated infrequently. A builder is going to see the most benefit from parallelism when doing an initial build (and download) from scratch.

Agree.

Just removing .NOTPARALLEL

--- a/depends/Makefile
+++ b/depends/Makefile
@@ -1,5 +1,3 @@
-.NOTPARALLEL :
-
 # Pattern rule to print variables, e.g. make print-top_srcdir
 print-%: FORCE [@echo](/bitcoin-bitcoin/contributor/echo/) '$*'='$($*)'

is not working as expected.

With make -j 8 -C depends many packages have not being built at all.

make has a command line option to synchronize its output.

As long as this is integrated into depends, so that users get useful make output by default.

is not working as expected. With make -j 8 -C depends many packages have not being built at all.

I was seeing issues too, maybe the original reason that .NOTPARALLEL was added, and it looked like miniupnpc might be the problem, however I can build with .NOTPARALLEL removed, using gmake -C depends -j9 --output-sync --no-print-directory and it's building all packages.

With everything downloaded, a full depends build actually seems to be slower for me, with .NOTPARALLEL removed. Will perform better benchmarks soon.

Just removing .NOTPARALLEL is not working as expected. With make -j 8 -C depends many packages have not being built at all.

Some of the packages depend on other packages. E.g. fontconfig depends on freetype. i suspect rules are missing which would tell make about this dependency. It will take a few days before i can properly look at it and provide a more precise answer.

make has a command line option to synchronize its output. As long as this is integrated into depends, so that users get useful make output by default.

There is another pr which adds -r to MAKEFLAGS. #22126. We could as well add -O. Another option is to leave this to the user to specify this option.

The reason of the failure without .NOTPARALLEL is a race for $(host_prefix) directory.

With the following diff:

diff --git a/depends/Makefile b/depends/Makefile
index a3b9cd209..72198f71e 100644
--- a/depends/Makefile
+++ b/depends/Makefile
@@ -1,5 +1,3 @@
-.NOTPARALLEL :
-
 # Pattern rule to print variables, e.g. make print-top_srcdir
 print-%: FORCE [@echo](/bitcoin-bitcoin/contributor/echo/) '$*'='$($*)'
diff --git a/depends/funcs.mk b/depends/funcs.mk
index 34a030fab..975d5d29e 100644
--- a/depends/funcs.mk
+++ b/depends/funcs.mk
@@ -207,7 +207,7 @@ $($(1)_preprocessed): | $($(1)_extracted)
        $(AT)touch $$@
 $($(1)_configured): | $($(1)_dependencies) $($(1)_preprocessed)
        $(AT)echo Configuring $(1)...
-       $(AT)rm -rf $(host_prefix); mkdir -p $(host_prefix)/lib; cd $(host_prefix); $(foreach package,$($(1)_all_dependencies), tar --no-same-owner -xf $($(package)_cached); )
+       $(AT)mkdir -p $(host_prefix)/lib; cd $(host_prefix); $(foreach package,$($(1)_all_dependencies), tar --no-same-owner -xf $($(package)_cached); )
        $(AT)mkdir -p $$(@D)
        $(AT)+cd $$(@D); $($(1)_config_env) $(call $(1)_config_cmds, $(1))
        $(AT)touch $$@

I've got

$ make -C depends clean
$ time make -j 9 -C depends
...

real	4m23,409s
user	25m18,663s
sys	1m58,159s

On master (b6a8e68b4e11be3aa5ceff8c483594f8e576faaf):

$ make -C depends clean
$ time make -j 9 -C depends
...

real	6m13,353s
user	24m18,275s
sys	2m5,152s

Is rm -rf $(host_prefix) intended to ensure that host_prefix contains freshly built packages only? This removal of host_prefix is quite unpleasant. When one package needs to reconfigure this rule removes the shared directory where all the other packages are installed.

4m23 vs 6m13 is 30% performance improvement. When Makefile contains .NOTPARALLEL it is only the top make that runs sequentially. It prevents building multiple packages in parallel. Each individual package is still built in parallel. Some of these packages contain a lot more files than 9. In this situation -j with a value comparable to the number of source files in a package should make a bigger difference. Provided the hardware can run this many processes.

I wouldn't be against such a change, but it appears that accommodating for .NOTPARALLEL would require significant changes to our build system. This is also a part of our codebase that attracts few contributors; as such, I'm not sure that this 30% improvement is worth the current burden and maintenance.

#22694 (comment) is indeed the reason for .NOTPARALLEL.

For each build, all necessary deps are extracted to a single prefix dir, and the package builds against that prefix. This ensures that the dependencies available to each package are deterministic and minimal. If parallel builds were allowed, there may be 2 or more packages extracting their deps to the host prefix simultaneously. This would mean that builds at different -j levels might see different headers and libs "installed".

The alternative would be to have a unique prefix for each package to use, but that would require post-processing each package to set a canonical prefix after the fact in order to work with autoconf's notion of using a single prefix. IMO that's not worth the trouble.

tl;dr: Removing .NOTPARALLEL safely would require a significant rework of depends.

following on the @theuni reasoning, I think this issue can be closed for now.

i'll close this. Thanks, everybody, for comments. regards, Dmitry