ci: Test with unreleased / experimental / nightly compilers #1271

real-or-random commented at 12:26 PM on April 11, 2023: contributor

If we do this, we can catch issues like the recent constant-time problem early enough. I already had some local notes on this. Here's a list of helpful resources for Debian:

fanquake commented at 9:09 AM on April 12, 2023: member

One thing to keep in mind here, is the increased likelyhood of spurious CI failures when running on experimental distros. Packages can be changing daily, and can be mis-packaged, or broken in various ways, causing build failures or other issues. The same apt install x y z invocation won't necessarily work one day to the next etc. We've seen this somewhat with debian:experimental.

real-or-random commented at 10:03 AM on April 12, 2023: contributor

Thanks, I'm fully aware that we may see random failures. My thinking is that it's still worth it in the end. It's probably a good idea to run these tests as nightly tests and not for PRs, so that random failures don't get in the way of normal dev work.

We've seen this somewhat with debian:experimental.

Has this been used in Core? I wasn't aware, and I'm happy to hear more about your experiences.

I could imagine that apt.llvm.org is the better choice for clang, so we'd need debian:experimental only for gcc.

fanquake commented at 10:07 AM on April 12, 2023: member

Has this been used in Core?

Not in the main repo. However other in repos / infra, it has been, i.e https://github.com/MarcoFalke/b-c-nightly. If you look through the commits in that repo, there are plenty of examples of random failures, i.e this merge: https://github.com/MarcoFalke/b-c-nightly/commit/458334fc1c141a6fc68f61f4bfe5a051c39d5699, all of the debian:experimental jobs failed because apt couldn't figure out how to install git:

The following packages have unmet dependencies:
 git : Depends: git-man (< 1:2.39.2-.) but 1:2.39.2+next.20230215-1 is to be installed
       Recommends: less but it is not going to be installed
       Recommends: ssh-client
E: Unable to correct problems, you have held broken packages.

real-or-random commented at 10:14 AM on April 12, 2023: contributor

Thanks, good to know. Our builds should be much simpler, but we could think about sharing some pieces for keeping Debian experimental work on Cirrus. I guess all we want is an experimental GCC in the end, but I can see that can also lead to failures.

jonasnick closed this on Jul 13, 2023