WIP: add basic semaphore2 CI

jonasschnelli commented at 3:37 pm on June 18, 2019: contributor

Semaphore2 seems to be quite new though it has some nice features and it is running on fast hardware. It is also possible to ssh-connect to the build-instances for more detailed debugging. The cache tool is also super flexible.

The only missing thing is public view access to the build progress and log. :( One of the co-founders said that public access is planned for later this year.

Integration with GitHub is possible today and the build state (failed/succeeded) is visible for everyone.

This is WIP PR would add three environments (ARM/X86/W64 depends build with proper caching).

jonasschnelli added the label Tests on Jun 18, 2019

jonasschnelli commented at 3:39 pm on June 18, 2019: contributor

jonasschnelli commented at 3:45 pm on June 18, 2019: contributor

Until semaphore2 supports full public view access to the builds, mirroring the build log via a self-hosted solution (e.g. bitcoincibuilds.org/pr123456) would be a way how to provide more information about failed builds.

jonasschnelli force-pushed on Jun 18, 2019

Add basic semaphore2 CI setup 5598e9b056

jonasschnelli force-pushed on Jun 18, 2019

in .semaphore/semaphore.yml:71 in 5598e9b056

66+          value: "false"
67+        - name: BITCOIN_CONFIG
68+          value: "--enable-reduce-exports --disable-gui-tests --with-gui=qt5 --with-incompatible-bdb"
69+        commands:
70+          - g++ --version
71+          - sem-version cpp 6

MarcoFalke commented at 7:46 pm on June 18, 2019:

I’d prefer to keep the current approach of using docker images. That way it is easy to reproduce locally and we don’t have to rely on third-party images or ssh sessions for debugging.

instagibbs commented at 6:35 pm on June 21, 2019:

is there a suggested workflow for fetching docker images on travis? Last time I looked it wasn’t easy/possible/too confusing for me to figure out.

jonasschnelli commented at 6:41 pm on June 21, 2019:

I think using Docker within a CI-VM which is meant to give a blank build machine makes conceptually little sense. It’s probably a viable approach if the selection of CI-VM images is very limited.

MarcoFalke commented at 6:47 pm on June 21, 2019:

@instagibbs Look in our .travis folder for code samples ready to copy-paste @jonasschnelli At this time we should closely mirror the linux version in our gitian descriptors, which is bionic. At the same time we should test against older versions of linux (such as trusty). Generally images provided by ci services have a lot of crap installed that interferes with our configure scripts or test runner. If there is an issue, it is hard to reproduce locally. And finally the ci service might not even offer the distros we need.

fanquake added the label Needs Conceptual Review on Jun 19, 2019

hebasto commented at 10:21 am on June 19, 2019: member

https://github.com/marketplace/semaphore:

Is 4x parallel CI/CD jobs enough?

MarcoFalke commented at 1:01 pm on June 19, 2019: member

We need at least 1-1.5 times the number of jobs per build in parallelism to not get backlogged on busy weekdays.

jonasschnelli commented at 6:44 pm on June 21, 2019: contributor

As mentioned here… Semaphore has offered use 16 instances in parallel for free (one instance can do a full depends build in 22mins).

laanwj commented at 12:25 pm on June 27, 2019: member

The only missing thing is public view access to the build progress and log. :( One of the co-founders said that public access is planned for later this year.

To be fair, this part is not too great for Travis either, with the huge slow web page with the log. But no access at all makes it pretty pointless.

Until semaphore2 supports full public view access to the builds, mirroring the build log via a self-hosted solution (e.g. bitcoincibuilds.org/pr123456) would be a way how to provide more information about failed builds.

Yes, indeed. We can try.

fanquake commented at 8:02 am on August 14, 2019: member

@jonasschnelli I think you mentioned on IRC that Semaphore now has public build progress and log viewing, is that correct?

jonasschnelli commented at 11:53 am on August 14, 2019: contributor

Yes. I think Semaphore2 is now on a similar feature level than Travis https://twitter.com/semaphoreci/status/1158340743456600066 https://docs.semaphoreci.com/article/113-changelog

fanquake commented at 2:41 am on August 15, 2019: member

Ok. What are the next steps for this PR? Do you need to make more changes to the config file?

I think @MarcoFalke has mentioned he’d be ok with merging it here, without enabling it for the repo.

jonasschnelli commented at 7:26 am on August 15, 2019: contributor

I don’t think we should merge this now,… better just keep it as a backup plan and as a reminder that we have options.

in .semaphore/update_build_test.sh:10 in 5598e9b056

 5+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
 6+
 7+export LC_ALL=C.UTF-8
 8+
 9+sudo apt-get update
10+sudo apt-get install --no-install-recommends --no-upgrade -qq $PACKAGES

MarcoFalke commented at 12:27 pm on August 15, 2019:

I’d prefer if this used the ./ci/ system from #16582.

Otherwise we’d probably lose a lot of coverage, like the wine tests, when switching from travis to semaphore

MarcoFalke commented at 1:45 pm on August 19, 2019: member

What is the goal here? Be a drop in replacement for the current travis build matrix or extend the travis build matrix with configurations that are not yet tested?

fanquake commented at 6:26 am on February 3, 2020: member

Maybe we can revisit this in future. We’ve currently got Travis CI & Appveyor, that both seem to have stabilized (except for the s390x build, that occasionally runs out of disk space), as well as the config for Cirrus in tree, (we’ve just removed GitHub actions).

fanquake closed this on Feb 3, 2020

jonasschnelli commented at 8:11 am on February 3, 2020: contributor

Yeah. Agree there is no need for Semaphore2 right now,… though we can warm it up in cases there is a lack of stable CIs. Sempahore also offered us free services.

bitcoinbuilds.org is also running again (faster than ever).

DrahtBot locked this on Feb 15, 2022

WIP: add basic semaphore2 CI #16233