ci: Switch all Linux builds to Debian and more improvements

real-or-random commented at 0:01 am on March 2, 2021: contributor

Best reviewed commit by commit

real-or-random force-pushed on Mar 2, 2021

in .cirrus.yml:114 in 658faf66f7 outdated

112@@ -101,14 +113,15 @@ task:
113     - env:
114         CC: gcc

jonasnick commented at 10:23 pm on March 2, 2021:

Hm, your new file invocations shows that this actually builds 64-bit binaries https://cirrus-ci.com/task/5529748383203328?command=test#L212. Not sure if this is missing the -m32 flag or if the automake --host option should have set that.

real-or-random commented at 10:55 pm on March 2, 2021:

Hm damn, I played a lot with this, and in the end I thought it works without the -m32 flag. I’ll have a look…

By the way, a difference to the previous approach on Debian (what we had on Travis) is that this here doesn’t use gcc-multilib because this is not compatible with installing the other s390x compiler (okay we could have two images) and I generally considered it to be more consistent: “multilib” is a thing we anyway can do only on x86(_64) and not for s390x (and arm and more archs, which we hopefully can add now by basically copy and paste).

real-or-random commented at 11:40 pm on March 2, 2021:

Ok, the issue was indeed that we set CC explicitly to gcc. When it’s not set, configure finds the right i686-linux-gn-gcc. I’ve set it explicitly to i686-linux-gnu-gcc now.

jonasnick commented at 2:46 pm on March 3, 2021:

Yup, thanks that appears to work.

jonasnick commented at 10:48 pm on March 2, 2021: contributor

658faf66f757105280b021d0d0aba697bfce4f73 looks good to me mod comment

in ci/cirrus.sh:34 in 658faf66f7 outdated

28+file ./bench_* || true
29+
30 if [ -n "$BUILD" ]
31 then
32-    make -j2 "$BUILD"
33+    make "$BUILD"

jonasnick commented at 10:51 pm on March 2, 2021:

Why remove -j2?

real-or-random commented at 10:59 pm on March 2, 2021:

Oh yeah, so the call to make doing the actual compilation now is a few linesabove.

But yeah, $BUILD is usually check here and this can run in parallel too: https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html So we should probably do this. In fact, we should have maybe have -j13 or probably set it cleverly because on Linux we ask for 1 CPU but on MacOS we always get 12 CPUs.

real-or-random commented at 0:25 am on March 3, 2021:

I changed the existing commit not to remove -j2 and I added a commit that improves parallelism on macOS. This can be improved further when we overhaul the CI script.

ci: Switch all Linux builds to Debian

The experiment of using Nix Shell was not really successful. Most
notably, Nix uses a bunch of wrapper scripts around compilers, which
make the build much less "pure". This may be useful but it's exactly
not what we want for CI. In particular, this resulted in gcc being used
for the "clang" builds because a wrapper script redefined the CC env
variable.

This now builds a single docker image (Debian) for all architectures
that we test in CI on Linux.

f24e122d13

real-or-random force-pushed on Mar 2, 2021

ci: Print information about binaries using "file" b994a8be3c

ci: Run PRs on merge result instead of on the source branch

This is taken from Bitcoin Core's .cirrus.yml

c7f754fe4d

ci: Split output of logs into multiple sections 28eccdf806

real-or-random force-pushed on Mar 3, 2021

in .cirrus.yml:19 in 9e5029e8dc outdated

15@@ -16,26 +16,38 @@ env:
16   CTIMETEST: yes
17   BENCH: yes
18   ITERS: 2
19-  # We only need the top commit
20-  CIRRUS_CLONE_DEPTH: 1
21+  MAKEFLAGS: -j2

jonasnick commented at 2:48 pm on March 3, 2021:

What does this do?

real-or-random commented at 2:56 pm on March 3, 2021:

MAKEFLAGS is a way to set flags for make globally. make will pick this up. See in 5.7.3 ian the manual (https://www.gnu.org/software/make/manual/make.html#index-MAKEFLAGS) or in Appendix A:

The MAKEFLAGS variable can also be useful if you want to have certain options, such as ‘-k’ (see Summary of Options), set each time you run make. You simply put a value for MAKEFLAGS in your environment.

real-or-random commented at 2:58 pm on March 3, 2021:

ugh but now I need to remove -j2 from the invocations in the script… Sorry, I screwed up with too many fixups and squashes

jonasnick commented at 3:14 pm on March 3, 2021:

Ah appendix A is helpful. The first sentence in the manual is “Flags […] are passed automatically to the sub-make through the variable MAKEFLAGS.” which would make this use of MAKEFLAGS rather esoteric.

real-or-random commented at 3:58 pm on March 3, 2021:

Yeah, I admit the docs are somewhat obscure. (Maybe this was introduced as a way to pass stuff to sub-makes but it’s widely used to specify -j2 for example, and there’s nothing wrong with that.) See also POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html

I’ll update once again to remove the explicit flags from the command line.

real-or-random commented at 5:20 pm on March 3, 2021:

fixed

ci: Select number of parallel make jobs depending on CI environment

This should improve compilation times on macOS. Things can certainly
be improved further, e.g., by running the benchmarks in parallel.

9361f360bb

real-or-random force-pushed on Mar 3, 2021

jonasnick approved

jonasnick commented at 7:01 pm on March 3, 2021: contributor

ACK 9361f360bb04156c7a0fa8f2664680b74d463ed5

in ci/cirrus.sh:27 in b994a8be3c outdated

20@@ -21,22 +21,27 @@ valgrind --version || true
21     --with-valgrind="$WITH_VALGRIND" \
22     --host="$HOST" $EXTRAFLAGS
23 
24+make -j2
25+
26+# Print information about binaries so that we can see that the architecture is correct
27+file *tests || true

sipa commented at 7:03 pm on March 7, 2021:

Why the “|| true” ?

real-or-random commented at 7:36 pm on March 7, 2021:

Otherwise the statement will fail if the wildcard does not match anything, and the files may not exist in every job. This is a little bit more meaningful in the line below when checking for benchmarks (because we have jobs with benchmarks disabled). But I just thought it’s more consistent to apply the || true in every line. In the future we may have special jobs without tests enabled (cttimetests only etc.)

sipa approved

sipa commented at 7:05 pm on March 7, 2021: contributor

utACK 9361f360bb04156c7a0fa8f2664680b74d463ed5

real-or-random merged this on Mar 7, 2021

real-or-random closed this on Mar 7, 2021

sipa cross-referenced this on Mar 8, 2021 from issue Safegcd inverses, drop Jacobi symbols, remove libgmp by sipa

Fabcien referenced this in commit bbaaa10666 on Apr 13, 2021

deadalnix referenced this in commit 73fca9b82e on Apr 14, 2021

ci: Switch all Linux builds to Debian and more improvements #901