ci: Switch all Linux builds to Debian and more improvements #901
pull real-or-random wants to merge 5 commits into bitcoin-core:master from real-or-random:202103-ci-debian changing 8 files +54 â59-
real-or-random commented at 0:01 am on March 2, 2021: contributorBest reviewed commit by commit
-
real-or-random force-pushed on Mar 2, 2021
-
real-or-random force-pushed on Mar 2, 2021
-
in .cirrus.yml:114 in 658faf66f7 outdated
112@@ -101,14 +113,15 @@ task: 113 - env: 114 CC: gcc
jonasnick commented at 10:23 pm on March 2, 2021:Hm, your newfile
invocations shows that this actually builds 64-bit binaries https://cirrus-ci.com/task/5529748383203328?command=test#L212. Not sure if this is missing the-m32
flag or if the automake--host
option should have set that.
real-or-random commented at 10:55 pm on March 2, 2021:Hm damn, I played a lot with this, and in the end I thought it works without the
-m32
flag. I’ll have a look…By the way, a difference to the previous approach on Debian (what we had on Travis) is that this here doesn’t use gcc-multilib because this is not compatible with installing the other s390x compiler (okay we could have two images) and I generally considered it to be more consistent: “multilib” is a thing we anyway can do only on x86(_64) and not for s390x (and arm and more archs, which we hopefully can add now by basically copy and paste).
real-or-random commented at 11:40 pm on March 2, 2021:Ok, the issue was indeed that we setCC
explicitly togcc
. When it’s not set, configure finds the righti686-linux-gn-gcc
. I’ve set it explicitly toi686-linux-gnu-gcc
now.
jonasnick commented at 2:46 pm on March 3, 2021:Yup, thanks that appears to work.jonasnick commented at 10:48 pm on March 2, 2021: contributor658faf66f757105280b021d0d0aba697bfce4f73 looks good to me mod commentin ci/cirrus.sh:34 in 658faf66f7 outdated
28+file ./bench_* || true 29+ 30 if [ -n "$BUILD" ] 31 then 32- make -j2 "$BUILD" 33+ make "$BUILD"
jonasnick commented at 10:51 pm on March 2, 2021:Why remove-j2
?
real-or-random commented at 10:59 pm on March 2, 2021:Oh yeah, so the call to
make
doing the actual compilation now is a few linesabove.But yeah,
$BUILD
is usuallycheck
here and this can run in parallel too: https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html So we should probably do this. In fact, we should have maybe have -j13 or probably set it cleverly because on Linux we ask for 1 CPU but on MacOS we always get 12 CPUs.
real-or-random commented at 0:25 am on March 3, 2021:I changed the existing commit not to remove -j2 and I added a commit that improves parallelism on macOS. This can be improved further when we overhaul the CI script.ci: Switch all Linux builds to Debian
The experiment of using Nix Shell was not really successful. Most notably, Nix uses a bunch of wrapper scripts around compilers, which make the build much less "pure". This may be useful but it's exactly not what we want for CI. In particular, this resulted in gcc being used for the "clang" builds because a wrapper script redefined the CC env variable. This now builds a single docker image (Debian) for all architectures that we test in CI on Linux.
real-or-random force-pushed on Mar 2, 2021ci: Print information about binaries using "file" b994a8be3cci: Run PRs on merge result instead of on the source branch
This is taken from Bitcoin Core's .cirrus.yml
ci: Split output of logs into multiple sections 28eccdf806real-or-random force-pushed on Mar 3, 2021in .cirrus.yml:19 in 9e5029e8dc outdated
15@@ -16,26 +16,38 @@ env: 16 CTIMETEST: yes 17 BENCH: yes 18 ITERS: 2 19- # We only need the top commit 20- CIRRUS_CLONE_DEPTH: 1 21+ MAKEFLAGS: -j2
jonasnick commented at 2:48 pm on March 3, 2021:What does this do?
real-or-random commented at 2:56 pm on March 3, 2021:MAKEFLAGS
is a way to set flags formake
globally.make
will pick this up. See in 5.7.3 ian the manual (https://www.gnu.org/software/make/manual/make.html#index-MAKEFLAGS) or in Appendix A:The MAKEFLAGS variable can also be useful if you want to have certain options, such as â-kâ (see Summary of Options), set each time you run make. You simply put a value for MAKEFLAGS in your environment.
real-or-random commented at 2:58 pm on March 3, 2021:ugh but now I need to remove-j2
from the invocations in the script… Sorry, I screwed up with too many fixups and squashes
jonasnick commented at 3:14 pm on March 3, 2021:Ah appendix A is helpful. The first sentence in the manual is “Flags […] are passed automatically to the sub-make through the variable MAKEFLAGS.” which would make this use of MAKEFLAGS rather esoteric.
real-or-random commented at 3:58 pm on March 3, 2021:Yeah, I admit the docs are somewhat obscure. (Maybe this was introduced as a way to pass stuff to sub-makes but it’s widely used to specify -j2 for example, and there’s nothing wrong with that.) See also POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html
I’ll update once again to remove the explicit flags from the command line.
real-or-random commented at 5:20 pm on March 3, 2021:fixedci: Select number of parallel make jobs depending on CI environment
This should improve compilation times on macOS. Things can certainly be improved further, e.g., by running the benchmarks in parallel.
real-or-random force-pushed on Mar 3, 2021jonasnick approvedjonasnick commented at 7:01 pm on March 3, 2021: contributorACK 9361f360bb04156c7a0fa8f2664680b74d463ed5in ci/cirrus.sh:27 in b994a8be3c outdated
20@@ -21,22 +21,27 @@ valgrind --version || true 21 --with-valgrind="$WITH_VALGRIND" \ 22 --host="$HOST" $EXTRAFLAGS 23 24+make -j2 25+ 26+# Print information about binaries so that we can see that the architecture is correct 27+file *tests || true
sipa commented at 7:03 pm on March 7, 2021:Why the “|| true” ?
real-or-random commented at 7:36 pm on March 7, 2021:Otherwise the statement will fail if the wildcard does not match anything, and the files may not exist in every job. This is a little bit more meaningful in the line below when checking for benchmarks (because we have jobs with benchmarks disabled). But I just thought it’s more consistent to apply the|| true
in every line. In the future we may have special jobs without tests enabled (cttimetests only etc.)sipa approvedsipa commented at 7:05 pm on March 7, 2021: contributorutACK 9361f360bb04156c7a0fa8f2664680b74d463ed5real-or-random merged this on Mar 7, 2021real-or-random closed this on Mar 7, 2021
sipa cross-referenced this on Mar 8, 2021 from issue Safegcd inverses, drop Jacobi symbols, remove libgmp by sipaFabcien referenced this in commit bbaaa10666 on Apr 13, 2021deadalnix referenced this in commit 73fca9b82e on Apr 14, 2021
github-metadata-mirror
This is a metadata mirror of the GitHub repository bitcoin-core/secp256k1. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-30 05:15 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me