Makefile: build precomputed_ecmult generators using build-system toolchain to support cross-compiling

whitslack commented at 8:53 am on November 20, 2022: contributor

When cross-compiling libsecp256k1, if the precomputed_ecmult*.c source files need to be regenerated, then the generators need to be built for the build system, not for the host system. Autoconf supports this fairly cleanly via the AX_PROG_CC_FOR_BUILD macro (from Autoconf Archive), but Automake requires some hackery. When building the generators, we override the CC, CFLAGS, CPPFLAGS, and LDFLAGS variables to their build-system counterparts, whose names are suffixed with _FOR_BUILD and whose values are populated by the aforementioned Autoconf macro and may be overridden on the make command line. Since Automake lacks support for overriding EXEEXT on a per-program basis, we define a recipe that builds the generator binaries with names suffixed with $(EXEEXT) and then renames them suffixed with $(BUILD_EXEEXT).

Makefile: build precomp generators using build-system toolchain

When cross-compiling libsecp256k1, if the `precomputed_ecmult*.c` source
files need to be regenerated, then the generators need to be built for
the *build* system, not for the *host* system. Autoconf supports this
fairly cleanly via the `AX_PROG_CC_FOR_BUILD` macro (from Autoconf
Archive), but Automake requires some hackery. When building the
generators, we override the `CC`, `CFLAGS`, `CPPFLAGS`, and `LDFLAGS`
variables to their build-system counterparts, whose names are suffixed
with `_FOR_BUILD` and whose values are populated by the aforementioned
Autoconf macro and may be overridden on the `make` command line. Since
Automake lacks support for overriding `EXEEXT` on a per-program basis,
we define a recipe that builds the generator binaries with names
suffixed with `$(EXEEXT)` and then renames them suffixed with
`$(BUILD_EXEEXT)`.

772e747bd9

whitslack commented at 8:57 am on November 20, 2022: contributor

Note: This PR introduces a dependency on Autoconf Archive. If this is undesirable, then the ax_prog_cc_for_build.m4 file can be extracted from Autoconf Archive and added to the build-aux/m4 subdirectory of this project.

Addendum: It looks like the automatic tests are failing because the build environment lacks Autoconf Archive. I don’t know enough about Cirrus to fix that. Help?

sipa commented at 1:52 pm on November 20, 2022: contributor

The precomputed_* files don’t need to be regenerated. They’re independent of the host or target build environment, so they can be created on any system, and compiled on any system.

whitslack commented at 4:26 pm on November 20, 2022: contributor

@sipa: The precomputed files need to be regenerated if you specify non-default values for --with-ecmult-window= and --with-ecmult-gen-precision= options to configure. If you’re building using a cross-compiler, then the regeneration will fail because the generators will not be executable on the system executing the build. Standard practice is to compile build-time tools using a toolchain that targets the build system.

whitslack commented at 4:40 pm on November 20, 2022: contributor

An example illustrating the failure:

 0$ git checkout origin/master
 1$ git clean -fdx
 2$ ./autogen.sh
 3$ ./configure --build=x86_64-pc-linux-gnu --host=armv6j-unknown-linux-gnueabihf --with-ecmult-window=2 --with-ecmult-gen-precision=2
 4$ make clean-precomp
 5$ make
 6make  precompute_ecmult_gen
 7make[1]: Entering directory '/example'
 8  CC       src/precompute_ecmult_gen-precompute_ecmult_gen.o
 9  CCLD     precompute_ecmult_gen
10make[1]: Leaving directory '/example'
11./precompute_ecmult_gen
12./precompute_ecmult_gen: 1: Syntax error: word unexpected (expecting ")")
13make: *** [Makefile:1979: src/precomputed_ecmult_gen.c] Error 2

The same example after applying this PR:

 0$ git fetch origin pull/1159/head
 1$ git checkout FETCH_HEAD
 2$ git clean -fdx
 3$ ./autogen.sh
 4$ ./configure --build=x86_64-pc-linux-gnu --host=armv6j-unknown-linux-gnueabihf --with-ecmult-window=2 --with-ecmult-gen-precision=2
 5$ make clean-precomp
 6$ make
 7make  precompute_ecmult_gen
 8make[1]: Entering directory '/example'
 9  CC       src/precompute_ecmult_gen-precompute_ecmult_gen.o
10  CCLD     precompute_ecmult_gen
11make[1]: Leaving directory '/example'
12./precompute_ecmult_gen
13make  precompute_ecmult
14make[1]: Entering directory '/example'
15  CC       src/precompute_ecmult-precompute_ecmult.o
16  CCLD     precompute_ecmult
17make[1]: Leaving directory '/example'
18./precompute_ecmult
19make  all-am
20make[1]: Entering directory '/example'
21  CC       src/bench.o
22  CC       src/libsecp256k1_la-secp256k1.lo
23  CC       src/libsecp256k1_precomputed_la-precomputed_ecmult.lo
24  CC       src/libsecp256k1_precomputed_la-precomputed_ecmult_gen.lo
25  CCLD     libsecp256k1_precomputed.la
26  CCLD     libsecp256k1.la
27  CCLD     bench
28  CC       src/bench_internal-bench_internal.o
29  CCLD     bench_internal
30  CC       src/bench_ecmult-bench_ecmult.o
31  CCLD     bench_ecmult
32  CC       src/tests-tests.o
33  CCLD     tests
34  CC       src/exhaustive_tests-tests_exhaustive.o
35  CCLD     exhaustive_tests
36make[1]: Leaving directory '/example'

real-or-random commented at 11:46 am on November 21, 2022: contributor

This has a bit of a history. We used to support AX_PROG_CC_FOR_BUILD for years but then removed it (https://github.com/bitcoin-core/secp256k1/pull/988/commits/d94a37a20c3b5b44f1bcf60d309ffc50727e18e4) because our experience was that it results in a lot of issues on various platforms and rare configs. The macro itself and the autoconf code we’ll need is a little bit hackish, plus users often didn’t understand how building is supposed to work.

The current resolution (as of #988) is to have the precomputed files in the repo. There should be only two reasons to rebuild them:

As a developer, if you changed the programs that generate them.
As a user, if you set --with-ecmult-window=SIZE for SIZE > 15. (But that’s slower on “normal” machines.)

See the docs:

 0  --with-ecmult-window=SIZE|auto
 1                          window size for ecmult precomputation for
 2                          verification, specified as integer in range [2..24].
 3                          Larger values result in possibly better performance
 4                          at the cost of an exponentially larger precomputed
 5                          table. The table will store 2^(SIZE-1) * 64 bytes of
 6                          data but can be larger in memory due to
 7                          platform-specific padding and alignment. A window
 8                          size larger than 15 will require you delete the
 9                          prebuilt precomputed_ecmult.c file so that it can be
10                          rebuilt. For very large window sizes, use "make -j
11                          1" to reduce memory use during compilation. "auto"
12                          is a reasonable setting for desktop machines
13                          (currently 15). [default=auto]

I see in your example that you want a SIZE of 2. And I see that you run make clean-precomp. Maybe this not entirely clear from the docs you’re never supposed to run make clean-precomp as a user. Just checkout the repo, configure with --ecmult-window-size=2, and make. That should work. Can you try?

(And maybe try a SIZE of 4. It should be much faster than 2 but still need only a negligible amount of memory.)

(And even if you want > 15, just delete precomputed_ecmult.c as the docs say and automake should figure out how to rebuild it – if you’re not cross-compiling.)

whitslack commented at 0:55 am on November 22, 2022: contributor

@real-or-random: Thank you for the history review. It’s a shame that a few edge cases had to ruin it for everyone. I do have to wonder if the issues you mentioned were due to Autotools not being used correctly, but I guess that ship has sailed.

My interest comes not as a user but as a packager. Gentoo takes a purist approach to open-source software: anything than can be rebuilt from sources ought to be. Standard operating procedure is to discard pregenerated assets and rebuild them from sources wherever possible. This is what the ebuilds for libsecp256k1 do with the precomputed ecmult tables. We have had to employ a hack to support cross-compiling, whereas most software packages that use Autotools and that compile tools to be executed during the build apparently have no trouble using AX_PROG_CC_FOR_BUILD.

I’ll withdraw my request for proper cross-compilation support and continue carrying my patch downstream. Thank you for your time and explanation.

whitslack closed this on Nov 22, 2022

real-or-random commented at 11:28 am on November 22, 2022: contributor

@whitslack

Thanks for your understanding. And thanks for reaching out! We don’t typically get feedback from packagers, and I see people applying (sometimes weird) patches without having talked to us.

My interest comes not as a user but as a packager. Gentoo takes a purist approach to open-source software: anything than can be rebuilt from sources ought to be. Standard operating procedure is to discard pregenerated assets and rebuild them from sources wherever possible.

I see. I was a Gentoo user in the past, but that’s maybe a little bit too purist for my taste (though there’s apparently a “wherever possible” exception). I’m not sure if this helps you, but maybe a simpler patch is to invoke a compiler on the build system manually. We try to make sure that the library is very easy to build manually, even though that’s currently not documented.

I had a look at your ebuild at https://gitlab.com/bitcoin/gentoo/-/blob/d783d4d95cc1e7befeecec30a2d1f9e0d3a5ad72/dev-libs/libsecp256k1/libsecp256k1-0.1.0_pre20220803.ebuild#L73-74

0		$(usex lowmem '--with-ecmult-window=2 --with-ecmult-gen-precision=2' '')
1		$(usex precompute-ecmult '--with-ecmult-window=24 --with-ecmult-gen-precision=8' '')

This should really be

0		$(usex lowmem '--with-ecmult-window=4 --with-ecmult-gen-precision=2' '')

(We should really just have a --with-low-mem option…)

And precompute-ecmult should probably be removed. I don’t see what the purpose of this flag is. We always have precomputed ecmult tables and the only thing the USE flag currently does is increase these tables to a huge size, so huge that performance will suffer for sure (because the tables won’t fit in the CPU cache).

The defaults are 15 and 4, and they’ve been selected using careful benchmarks for desktop machines. Maybe you can squeeze out a few percents on your specific machine with something like (14,4) or (16,4) or similar if you really do the benchmarks. But (24,8) is insanely high and has no benefits.

whitslack commented at 11:35 am on November 22, 2022: contributor

And precompute-ecmult should probably be removed. @real-or-random: Thanks. I’ll make those changes. Those flags originated before I got involved.

whitslack commented at 0:48 am on November 23, 2022: contributor

I’ll make those changes.

Done.

gentoo-repo-qa-bot referenced this in commit 6e39601a74 on Nov 23, 2022

whitslack cross-referenced this on Jan 6, 2023 from issue dev-libs/libsecp256k1 ebuild updates by whitslack

whitslack referenced this in commit aa83d52772 on Jan 6, 2023

whitslack referenced this in commit b1b09655f3 on Jan 6, 2023

whitslack referenced this in commit 863272f631 on Jan 6, 2023

whitslack referenced this in commit f99a2e5aea on Jan 6, 2023

whitslack referenced this in commit 5feffcdabc on Jan 7, 2023

whitslack referenced this in commit 798a0aa948 on Jan 7, 2023

gentoo-bot referenced this in commit 92347c4b7a on Jan 7, 2023

gentoo-bot referenced this in commit 691874242c on Jan 7, 2023

Makefile: build precomputed_ecmult generators using build-system toolchain to support cross-compiling #1159