Clang sanitizer binaries segfault on CI

real-or-random commented at 12:54 pm on March 19, 2024: contributor

This is not related to a specific PR, the only reason why master is green is that it doesn’t have recent pushes.

The first bad build is https://github.com/bitcoin-core/secp256k1/actions/runs/8317860245?pr=1058, which includes a rebuild of the Docker image… The last good build of the Docker image is https://github.com/bitcoin-core/secp256k1/actions/runs/8208332997/job/22451543872?pr=1479. But I’m not sure which difference caused the issue. There was no change on our side, so something must have changed in Debian. But what? Here are some things to look into:

updating to clang-snapshot doesn’t help, see #1505
There was a recent clang packing fix related to the clang runtime, which looks suspicious: https://metadata.ftp-master.debian.org/changelogs//main/l/llvm-toolchain-14/llvm-toolchain-14_14.0.6-12_changelog, but the new version is apparently used already in the last good build
there’s this related bug report, but it’s two years old: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008309

We should probably diff the package lists (including package versions) from the CI outputs of the Docker image builds. This tells us what Debian packages got updated between the last good and the first bad build…

real-or-random added the label assurance on Mar 19, 2024

real-or-random added the label ci on Mar 19, 2024

jonasnick commented at 3:43 pm on March 19, 2024: contributor

Interestingly the MuSig2 PR fails with a segfault in the tests and not in ./configure like the signed-digit multicomb PR.

I tried reproducing this locally using our docker file but failed: the tests pass in my local docker image.

I started the container with

0docker build -t secp256k1-image -f ci/linux-debian.Dockerfile .
1docker run -it --rm secp256k1-image

and then executed

0git clone https://github.com/bitcoin-core/secp256k1.git
1cd secp256k1
2git fetch origin pull/1479/head:musig2-module
3git checkout musig2-module
4WERROR_CFLAGS="-Werror -pedantic-errors" MAKEFLAGS="-j4" BUILD="check" ECMULTWINDOW="auto" ECMULTGENPRECISION="auto" ASM="auto" WIDEMUL="auto" WITH_VALGRIND="yes" EXTRAFLAGS="" EXPERIMENTAL="no" ECDH="yes" RECOVERY="yes" SCHNORRSIG="yes" MUSIG="yes" ELLSWIFT="yes" SECP256K1_TEST_ITERS="32" BENCH="yes" SECP256K1_BENCH_ITERS="2" CTIMETESTS="no" EXAMPLES="yes" HOST="" WRAPPER_CMD="" CC="clang" CFLAGS="-fsanitize=undefined,address -g" ./ci/ci.sh

inside the container.

real-or-random commented at 3:48 pm on March 19, 2024: contributor

@jonasnick Can you try setting the additional sanitizer env variables that we set in the CI config? https://github.com/bitcoin-core/secp256k1/blob/427e86b9edcaa5cb19e71233ad93c2ec60dff10f/.github/workflows/ci.yml#L444-L446

Those should probably be added to our output for reproduction…

I’m just in the process of getting a diff of the Debian package versions.

real-or-random commented at 4:04 pm on March 19, 2024: contributor

I’m just in the process of getting a diff of the Debian package versions.

I double and triple checked that I used the right input, but believe it or not, the diff is empty…

For the record, this is the package list:

  0amd64 perl-modules-5.36 all 5.36.0-7+deb12u1
  1amd64 libgdbm6 amd64 1.23-3
  2amd64 libgdbm-compat4 amd64 1.23-3
  3amd64 libperl5.36 amd64 5.36.0-7+deb12u1
  4amd64 perl amd64 5.36.0-7+deb12u1
  5amd64 libssl3 amd64 3.0.11-1~deb12u2
  6amd64 libpython3.11-minimal amd64 3.11.2-6
  7amd64 libexpat1 amd64 2.5.0-1
  8amd64 python3.11-minimal amd64 3.11.2-6
  9amd64 python3-minimal amd64 3.11.2-1+b1
 10amd64 media-types all 10.0.0
 11amd64 libncursesw6 amd64 6.4-4
 12amd64 libkrb5support0 amd64 1.20.1-2+deb12u1
 13amd64 libk5crypto3 amd64 1.20.1-2+deb12u1
 14amd64 libkeyutils1 amd64 1.6.3-2
 15amd64 libkrb5-3 amd64 1.20.1-2+deb12u1
 16amd64 libgssapi-krb5-2 amd64 1.20.1-2+deb12u1
 17amd64 libtirpc-common all 1.3.3+ds-1
 18amd64 libtirpc3 amd64 1.3.3+ds-1
 19amd64 libnsl2 amd64 1.3.0-2
 20amd64 readline-common all 8.2-1.3
 21amd64 libreadline8 amd64 8.2-1.3
 22amd64 libsqlite3-0 amd64 3.40.1-2
 23amd64 libpython3.11-stdlib amd64 3.11.2-6
 24amd64 python3.11 amd64 3.11.2-6
 25amd64 libpython3-stdlib amd64 3.11.2-1+b1
 26amd64 python3 amd64 3.11.2-1+b1
 27amd64 bzip2 amd64 1.0.8-5+b1
 28amd64 openssl amd64 3.0.11-1~deb12u2
 29amd64 ca-certificates all 20230311
 30amd64 libmagic-mgc amd64 1:5.44-3
 31amd64 libmagic1 amd64 1:5.44-3
 32amd64 file amd64 1:5.44-3
 33amd64 xz-utils amd64 5.4.1-0.2
 34amd64 m4 amd64 1.4.19-3
 35amd64 autoconf all 2.71-3
 36amd64 autotools-dev all 20220109.1
 37amd64 automake all 1:1.16.5-1.3
 38amd64 binutils-common amd64 2.40-2
 39amd64 libbinutils amd64 2.40-2
 40amd64 libctf-nobfd0 amd64 2.40-2
 41amd64 libctf0 amd64 2.40-2
 42amd64 libgprofng0 amd64 2.40-2
 43amd64 libjansson4 amd64 2.14-2
 44amd64 binutils-x86-64-linux-gnu amd64 2.40-2
 45amd64 binutils amd64 2.40-2
 46amd64 binutils-arm-linux-gnueabihf amd64 2.40-2
 47amd64 binutils-mingw-w64-i686 amd64 2.40-2+10.4
 48amd64 binutils-mingw-w64-x86-64 amd64 2.40-2+10.4
 49amd64 binutils-powerpc64le-linux-gnu amd64 2.40-2
 50amd64 binutils-s390x-linux-gnu amd64 2.40-2
 51amd64 libbsd0 amd64 0.11.7-2
 52amd64 libedit2 amd64 3.1-20221030-2
 53amd64 libicu72 amd64 72.1-3
 54amd64 libxml2 amd64 2.9.14+dfsg-1.3~deb12u1
 55amd64 libz3-4 amd64 4.8.12-3.1
 56amd64 libllvm14 amd64 1:14.0.6-12
 57amd64 libclang-cpp14 amd64 1:14.0.6-12
 58amd64 libgomp1 amd64 12.2.0-14
 59amd64 libitm1 amd64 12.2.0-14
 60amd64 libatomic1 amd64 12.2.0-14
 61amd64 libasan8 amd64 12.2.0-14
 62amd64 liblsan0 amd64 12.2.0-14
 63amd64 libtsan2 amd64 12.2.0-14
 64amd64 libubsan1 amd64 12.2.0-14
 65amd64 libquadmath0 amd64 12.2.0-14
 66amd64 libgcc-12-dev amd64 12.2.0-14
 67amd64 libc-dev-bin amd64 2.36-9+deb12u4
 68amd64 linux-libc-dev amd64 6.1.76-1
 69amd64 libcrypt-dev amd64 1:4.4.33-2
 70amd64 libtirpc-dev amd64 1.3.3+ds-1
 71amd64 libnsl-dev amd64 1.3.0-2
 72amd64 rpcsvc-proto amd64 1.4.3-1
 73amd64 libc6-dev amd64 2.36-9+deb12u4
 74amd64 libstdc++-12-dev amd64 12.2.0-14
 75amd64 libgc1 amd64 1:8.2.2-3
 76amd64 libobjc4 amd64 12.2.0-14
 77amd64 libobjc-12-dev amd64 12.2.0-14
 78amd64 libclang-common-14-dev all 1:14.0.6-12
 79amd64 llvm-14-linker-tools amd64 1:14.0.6-12
 80amd64 libclang1-14 amd64 1:14.0.6-12
 81amd64 clang-14 amd64 1:14.0.6-12
 82amd64 clang amd64 1:14.0-55.7~deb12u1
 83amd64 libisl23 amd64 0.25-1.1
 84amd64 libmpfr6 amd64 4.2.0-1
 85amd64 libmpc3 amd64 1.3.1-1
 86amd64 cpp-12 amd64 12.2.0-14
 87amd64 cpp amd64 4:12.2.0-3
 88amd64 gcc-12-arm-linux-gnueabihf-base amd64 12.2.0-14cross1
 89amd64 cpp-12-arm-linux-gnueabihf amd64 12.2.0-14cross1
 90amd64 gcc-12-i686-linux-gnu-base amd64 12.2.0-14cross1
 91amd64 cpp-12-i686-linux-gnu amd64 12.2.0-14cross1
 92amd64 gcc-12-powerpc64le-linux-gnu-base amd64 12.2.0-14cross1
 93amd64 cpp-12-powerpc64le-linux-gnu amd64 12.2.0-14cross1
 94amd64 gcc-12-s390x-linux-gnu-base amd64 12.2.0-14cross1
 95amd64 cpp-12-s390x-linux-gnu amd64 12.2.0-14cross1
 96amd64 cpp-arm-linux-gnueabihf amd64 4:12.2.0-3
 97amd64 cpp-i686-linux-gnu amd64 4:12.2.0-3
 98amd64 cpp-powerpc64le-linux-gnu amd64 4:12.2.0-3
 99amd64 cpp-s390x-linux-gnu amd64 4:12.2.0-3
100amd64 libdpkg-perl all 1.21.22
101amd64 patch amd64 2.7.6-7
102amd64 make amd64 4.3-4.1
103amd64 dpkg-dev all 1.21.22
104amd64 fonts-dejavu-core all 2.37-6
105amd64 fontconfig-config amd64 2.14.1-4
106amd64 libcc1-0 amd64 12.2.0-14
107amd64 gcc-12 amd64 12.2.0-14
108amd64 gcc amd64 4:12.2.0-3
109amd64 g++-12 amd64 12.2.0-14
110amd64 g++ amd64 4:12.2.0-3
111amd64 gcc-12-cross-base all 12.2.0-14cross1
112amd64 libc6-armhf-cross all 2.36-8cross1
113amd64 libgcc-s1-armhf-cross all 12.2.0-14cross1
114amd64 libgomp1-armhf-cross all 12.2.0-14cross1
115amd64 libatomic1-armhf-cross all 12.2.0-14cross1
116amd64 libasan8-armhf-cross all 12.2.0-14cross1
117amd64 libstdc++6-armhf-cross all 12.2.0-14cross1
118amd64 libubsan1-armhf-cross all 12.2.0-14cross1
119amd64 libgcc-12-dev-armhf-cross all 12.2.0-14cross1
120amd64 gcc-12-arm-linux-gnueabihf amd64 12.2.0-14cross1
121armhf gcc-12-base armhf 12.2.0-14
122s390x gcc-12-base s390x 12.2.0-14
123i386 gcc-12-base i386 12.2.0-14
124ppc64el gcc-12-base ppc64el 12.2.0-14
125amd64 binutils-i686-linux-gnu amd64 2.40-2
126amd64 libc6-i386-cross all 2.36-8cross1
127amd64 libgcc-s1-i386-cross all 12.2.0-14cross1
128amd64 libgomp1-i386-cross all 12.2.0-14cross1
129amd64 libitm1-i386-cross all 12.2.0-14cross1
130amd64 libatomic1-i386-cross all 12.2.0-14cross1
131amd64 libasan8-i386-cross all 12.2.0-14cross1
132amd64 libstdc++6-i386-cross all 12.2.0-14cross1
133amd64 libubsan1-i386-cross all 12.2.0-14cross1
134amd64 libquadmath0-i386-cross all 12.2.0-14cross1
135amd64 libgcc-12-dev-i386-cross all 12.2.0-14cross1
136amd64 gcc-12-i686-linux-gnu amd64 12.2.0-14cross1
137amd64 libc6-ppc64el-cross all 2.36-8cross1
138amd64 libgcc-s1-ppc64el-cross all 12.2.0-14cross1
139amd64 libgomp1-ppc64el-cross all 12.2.0-14cross1
140amd64 libitm1-ppc64el-cross all 12.2.0-14cross1
141amd64 libatomic1-ppc64el-cross all 12.2.0-14cross1
142amd64 libasan8-ppc64el-cross all 12.2.0-14cross1
143amd64 liblsan0-ppc64el-cross all 12.2.0-14cross1
144amd64 libtsan2-ppc64el-cross all 12.2.0-14cross1
145amd64 libstdc++6-ppc64el-cross all 12.2.0-14cross1
146amd64 libubsan1-ppc64el-cross all 12.2.0-14cross1
147amd64 libquadmath0-ppc64el-cross all 12.2.0-14cross1
148amd64 libgcc-12-dev-ppc64el-cross all 12.2.0-14cross1
149amd64 gcc-12-powerpc64le-linux-gnu amd64 12.2.0-14cross1
150amd64 libc6-s390x-cross all 2.36-8cross1
151amd64 libgcc-s1-s390x-cross all 12.2.0-14cross1
152amd64 libgomp1-s390x-cross all 12.2.0-14cross1
153amd64 libitm1-s390x-cross all 12.2.0-14cross1
154amd64 libatomic1-s390x-cross all 12.2.0-14cross1
155amd64 libasan8-s390x-cross all 12.2.0-14cross1
156amd64 libstdc++6-s390x-cross all 12.2.0-14cross1
157amd64 libubsan1-s390x-cross all 12.2.0-14cross1
158amd64 libgcc-12-dev-s390x-cross all 12.2.0-14cross1
159amd64 gcc-12-s390x-linux-gnu amd64 12.2.0-14cross1
160amd64 gcc-arm-linux-gnueabihf amd64 4:12.2.0-3
161amd64 gcc-i686-linux-gnu amd64 4:12.2.0-3
162amd64 gcc-mingw-w64-base amd64 12.2.0-14+25.2
163amd64 mingw-w64-common all 10.0.0-3
164amd64 mingw-w64-i686-dev all 10.0.0-3
165amd64 gcc-mingw-w64-i686-win32-runtime amd64 12.2.0-14+25.2
166amd64 gcc-mingw-w64-i686-win32 amd64 12.2.0-14+25.2
167amd64 mingw-w64-x86-64-dev all 10.0.0-3
168amd64 gcc-mingw-w64-x86-64-win32-runtime amd64 12.2.0-14+25.2
169amd64 gcc-mingw-w64-x86-64-win32 amd64 12.2.0-14+25.2
170amd64 gcc-powerpc64le-linux-gnu amd64 4:12.2.0-3
171amd64 gcc-s390x-linux-gnu amd64 4:12.2.0-3
172amd64 libbrotli1 amd64 1.0.9-2+b6
173amd64 libsasl2-modules-db amd64 2.1.28+dfsg-10
174amd64 libsasl2-2 amd64 2.1.28+dfsg-10
175amd64 libldap-2.5-0 amd64 2.5.13+dfsg-5
176amd64 libnghttp2-14 amd64 1.52.0-1+deb12u1
177amd64 libpsl5 amd64 0.21.2-1
178amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2+b2
179amd64 libssh2-1 amd64 1.10.0-3+b1
180amd64 libcurl3-gnutls amd64 7.88.1-10+deb12u5
181amd64 liberror-perl all 0.17029-2
182amd64 git-man all 1:2.39.2-1.1
183amd64 git amd64 1:2.39.2-1.1
184amd64 iso-codes all 4.15.0-1
185amd64 libc6-i386 amd64 2.36-9+deb12u4
186amd64 lib32gcc-s1 amd64 12.2.0-14
187amd64 lib32stdc++6 amd64 12.2.0-14
188amd64 libabsl20220623 amd64 20220623.1-1
189i386 libgcc-s1 i386 12.2.0-14
190i386 libc6 i386 2.36-9+deb12u4
191i386 libstdc++6 i386 12.2.0-14
192i386 libabsl20220623 i386 20220623.1-1
193i386 libaom3 i386 3.6.0-1
194amd64 libaom3 amd64 3.6.0-1
195i386 libasan8 i386 12.2.0-14
196amd64 libasound2-data all 1.2.8-1
197i386 libasound2 i386 1.2.8-1+b1
198amd64 libasound2 amd64 1.2.8-1+b1
199amd64 libasyncns0 amd64 0.8-6+b3
200i386 libasyncns0 i386 0.8-6+b3
201amd64 libdav1d6 amd64 1.0.0-2
202amd64 libgav1-1 amd64 0.18.0-1+b1
203amd64 librav1e0 amd64 0.5.1-6
204amd64 libsvtav1enc1 amd64 1.4.1+dfsg-1
205amd64 libjpeg62-turbo amd64 1:2.1.5-2
206amd64 libyuv0 amd64 0.0~git20230123.b2528b0-1
207amd64 libavif15 amd64 0.11.1-1
208i386 libdav1d6 i386 1.0.0-2
209i386 libgav1-1 i386 0.18.0-1+b1
210i386 librav1e0 i386 0.5.1-6
211i386 libjpeg62-turbo i386 1:2.1.5-2
212i386 libyuv0 i386 0.0~git20230123.b2528b0-1
213i386 libavif15 i386 0.11.1-1
214i386 libblkid1 i386 2.38.1-5+b1
215i386 libbrotli1 i386 1.0.9-2+b6
216i386 libmd0 i386 1.0.4-2
217i386 libbsd0 i386 0.11.7-2
218i386 libbz2-1.0 i386 1.0.8-5+b1
219ppc64el libgcc-s1 ppc64el 12.2.0-14
220ppc64el libc6 ppc64el 2.36-9+deb12u4
221s390x libgcc-s1 s390x 12.2.0-14
222s390x libc6 s390x 2.36-9+deb12u4
223armhf libgcc-s1 armhf 12.2.0-14
224armhf libc6 armhf 2.36-9+deb12u4
225ppc64el libc6-dbg ppc64el 2.36-9+deb12u4
226amd64 libc6-dbg amd64 2.36-9+deb12u4
227armhf libc6-dbg armhf 2.36-9+deb12u4
228s390x libc6-dbg s390x 2.36-9+deb12u4
229i386 libc6-dbg i386 2.36-9+deb12u4
230amd64 linux-libc-dev-armhf-cross all 6.1.4-1cross1
231amd64 libc6-dev-armhf-cross all 2.36-8cross1
232amd64 linux-libc-dev-i386-cross all 6.1.4-1cross1
233amd64 libc6-dev-i386-cross all 2.36-8cross1
234amd64 linux-libc-dev-ppc64el-cross all 6.1.4-1cross1
235amd64 libc6-dev-ppc64el-cross all 2.36-8cross1
236amd64 linux-libc-dev-s390x-cross all 6.1.4-1cross1
237amd64 libc6-dev-s390x-cross all 2.36-8cross1
238i386 libcap2 i386 1:2.66-4
239amd64 libcap2-bin amd64 1:2.66-4
240amd64 libcapi20-3 amd64 1:3.27-3+b1
241i386 libcapi20-3 i386 1:3.27-3+b1
242amd64 libcapstone4 amd64 4.0.2-5
243amd64 libclang-rt-14-dev amd64 1:14.0.6-12
244amd64 libclang-rt-dev amd64 1:14.0-55.7~deb12u1
245i386 libcom-err2 i386 1.47.0-2
246amd64 libnspr4 amd64 2:4.35-1
247amd64 libnss3 amd64 2:3.87.1-1
248amd64 nss-plugin-pem amd64 1.0.8+1-1
249amd64 libcurl3-nss amd64 7.88.1-10+deb12u5
250amd64 libcurl4 amd64 7.88.1-10+deb12u5
251i386 libkrb5support0 i386 1.20.1-2+deb12u1
252i386 libk5crypto3 i386 1.20.1-2+deb12u1
253i386 libkeyutils1 i386 1.6.3-2
254i386 libssl3 i386 3.0.11-1~deb12u2
255i386 libkrb5-3 i386 1.20.1-2+deb12u1
256i386 libgssapi-krb5-2 i386 1.20.1-2+deb12u1
257i386 libunistring2 i386 1.0-2
258i386 libidn2-0 i386 2.3.3-1+b1
259i386 libgmp10 i386 2:6.2.1+dfsg1-1.1
260i386 libnettle8 i386 3.8.1-2
261i386 libhogweed6 i386 3.8.1-2
262i386 libffi8 i386 3.4.4-1
263i386 libp11-kit0 i386 0.24.1-2
264i386 libtasn1-6 i386 4.19.0-2
265i386 libgnutls30 i386 3.7.9-2+deb12u2
266i386 libdb5.3 i386 5.3.28+dfsg2-1
267i386 libsasl2-modules-db i386 2.1.28+dfsg-10
268i386 libsasl2-2 i386 2.1.28+dfsg-10
269i386 libldap-2.5-0 i386 2.5.13+dfsg-5
270i386 libnghttp2-14 i386 1.52.0-1+deb12u1
271i386 libpsl5 i386 0.21.2-1
272i386 zlib1g i386 1:1.2.13.dfsg-1
273i386 librtmp1 i386 2.4+20151223.gitfa8646d.1-2+b2
274i386 libssh2-1 i386 1.10.0-3+b1
275i386 libzstd1 i386 1.5.4+dfsg2-5
276i386 libcurl4 i386 7.88.1-10+deb12u5
277amd64 libdbus-1-3 amd64 1.14.10-1~deb12u1
278i386 libgpg-error0 i386 1.46-1
279i386 libgcrypt20 i386 1.10.1-3
280i386 liblz4-1 i386 1.9.4-1
281i386 liblzma5 i386 5.4.1-0.2
282i386 libsystemd0 i386 252.22-1~deb12u1
283i386 libdbus-1-3 i386 1.14.10-1~deb12u1
284i386 libde265-0 i386 1.0.11-1+deb12u2
285amd64 libde265-0 amd64 1.0.11-1+deb12u2
286i386 libdeflate0 i386 1.14-1
287amd64 libdeflate0 amd64 1.14-1
288amd64 libelf1 amd64 0.188-2.1
289amd64 libdw1 amd64 0.188-2.1
290i386 libelf1 i386 0.188-2.1
291i386 libdw1 i386 0.188-2.1
292i386 libexif12 i386 0.6.24-1+b1
293amd64 libexif12 amd64 0.6.24-1+b1
294i386 libexpat1 i386 2.5.0-1
295i386 libogg0 i386 1.3.5-3
296i386 libflac12 i386 1.4.2+ds-2
297amd64 libogg0 amd64 1.3.5-3
298amd64 libflac12 amd64 1.4.2+ds-2
299i386 libpng16-16 i386 1.6.39-2
300i386 libfreetype6 i386 2.12.1+dfsg-5
301i386 libfontconfig1 i386 2.14.1-4
302amd64 libpng16-16 amd64 1.6.39-2
303amd64 libfreetype6 amd64 2.12.1+dfsg-5
304amd64 libfontconfig1 amd64 2.14.1-4
305i386 libnuma1 i386 2.0.16-1
306i386 libx265-199 i386 3.5-2+b1
307i386 libheif1 i386 1.15.1-1
308i386 libjbig0 i386 2.1-6.1
309i386 liblerc4 i386 4.0.0+ds-2
310i386 libwebp7 i386 1.2.4-0.2+deb12u1
311i386 libtiff6 i386 4.5.0-6+deb12u1
312i386 libxau6 i386 1:1.0.9-1
313i386 libxdmcp6 i386 1:1.1.2-3
314i386 libxcb1 i386 1.15-1
315amd64 libx11-data all 2:1.8.4-2+deb12u2
316i386 libx11-6 i386 2:1.8.4-2+deb12u2
317i386 libxpm4 i386 1:3.5.12-1.1+deb12u1
318i386 libgd3 i386 2.3.3-9
319amd64 libnuma1 amd64 2.0.16-1
320amd64 libx265-199 amd64 3.5-2+b1
321amd64 libheif1 amd64 1.15.1-1
322amd64 libjbig0 amd64 2.1-6.1
323amd64 liblerc4 amd64 4.0.0+ds-2
324amd64 libwebp7 amd64 1.2.4-0.2+deb12u1
325amd64 libtiff6 amd64 4.5.0-6+deb12u1
326amd64 libxau6 amd64 1:1.0.9-1
327amd64 libxdmcp6 amd64 1:1.1.2-3
328amd64 libxcb1 amd64 1.15-1
329amd64 libx11-6 amd64 2:1.8.4-2+deb12u2
330amd64 libxpm4 amd64 1:3.5.12-1.1+deb12u1
331amd64 libgd3 amd64 2.3.3-9
332i386 libpcre2-8-0 i386 10.42-1
333i386 libselinux1 i386 3.4-1+b6
334i386 libmount1 i386 2.38.1-5+b1
335i386 libglib2.0-0 i386 2.74.6-2
336amd64 libglib2.0-0 amd64 2.74.6-2
337i386 libltdl7 i386 2.4.7-5
338i386 libudev1 i386 252.22-1~deb12u1
339i386 libusb-1.0-0 i386 2:1.0.26-1
340i386 libgphoto2-port12 i386 2.5.30-1
341i386 libicu72 i386 72.1-3
342i386 libxml2 i386 2.9.14+dfsg-1.3~deb12u1
343i386 libgphoto2-6 i386 2.5.30-1
344amd64 libltdl7 amd64 2.4.7-5
345amd64 libusb-1.0-0 amd64 2:1.0.26-1
346amd64 libgphoto2-port12 amd64 2.5.30-1
347amd64 libgphoto2-6 amd64 2.5.30-1
348i386 libunwind8 i386 1.6.2-3
349i386 libgstreamer1.0-0 i386 1.22.0-2
350i386 liborc-0.4-0 i386 1:0.4.33-2
351i386 libgstreamer-plugins-base1.0-0 i386 1.22.0-3+deb12u1
352amd64 libunwind8 amd64 1.6.2-3
353amd64 libgstreamer1.0-0 amd64 1.22.0-2
354amd64 liborc-0.4-0 amd64 1:0.4.33-2
355amd64 libgstreamer-plugins-base1.0-0 amd64 1.22.0-3+deb12u1
356amd64 libmp3lame0 amd64 3.100-6
357i386 libmp3lame0 i386 3.100-6
358i386 libmpg123-0 i386 1.31.2-1
359amd64 libmpg123-0 amd64 1.31.2-1
360amd64 libopus0 amd64 1.3.1-3
361i386 libopus0 i386 1.3.1-3
362amd64 libpcap0.8 amd64 1.10.3-1
363i386 libpcap0.8 i386 1.10.3-1
364amd64 libpfm4 amd64 4.13.0-1
365amd64 libpkgconf3 amd64 1.8.1-1
366amd64 libvorbis0a amd64 1.3.7-1
367amd64 libvorbisenc2 amd64 1.3.7-1
368amd64 libsndfile1 amd64 1.2.0-1
369amd64 libx11-xcb1 amd64 2:1.8.4-2+deb12u2
370amd64 libpulse0 amd64 16.1+dfsg1-2+b1
371i386 libvorbis0a i386 1.3.7-1
372i386 libvorbisenc2 i386 1.3.7-1
373i386 libsndfile1 i386 1.2.0-1
374i386 libx11-xcb1 i386 2:1.8.4-2+deb12u2
375i386 libpulse0 i386 16.1+dfsg1-2+b1
376amd64 libtool all 2.4.7-5
377i386 libubsan1 i386 12.2.0-14
378amd64 liburing2 amd64 2.3-3
379i386 libxext6 i386 2:1.3.4-1+b1
380i386 ocl-icd-libopencl1 i386 2.3.1-1
381amd64 libz-mingw-w64 all 1.2.13+dfsg-1
382i386 libwine i386 8.0~repack-4
383amd64 libxext6 amd64 2:1.3.4-1+b1
384amd64 ocl-icd-libopencl1 amd64 2.3.1-1
385amd64 libwine amd64 8.0~repack-4
386amd64 llvm-14-runtime amd64 1:14.0.6-12
387amd64 llvm-runtime amd64 1:14.0-55.7~deb12u1
388amd64 llvm-14 amd64 1:14.0.6-12
389amd64 llvm amd64 1:14.0-55.7~deb12u1
390amd64 pkgconf-bin amd64 1.8.1-1
391amd64 pkgconf amd64 1.8.1-1
392amd64 pkg-config amd64 1.8.1-1
393amd64 qemu-user amd64 1:7.2+dfsg-7+deb12u5
394amd64 valgrind amd64 1:3.19.0-1
395amd64 wine64 amd64 8.0~repack-4
396i386 wine32 i386 8.0~repack-4
397amd64 wine all 8.0~repack-4

there’s this related bug report, but it’s two years old: bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008309

I’m starting to believe that this is related. It mentions some sysctl settings, and searching for them gives me https://github.com/google/sanitizers/issues/1614 with a bunch of linked issues in the past days… We should probably try (sudo) sysctl vm.mmap_rnd_bits=28.

real-or-random referenced this in commit 595e28f28e on Mar 19, 2024

real-or-random referenced this in commit b1844d9983 on Mar 19, 2024

real-or-random referenced this in commit 0fccc33963 on Mar 19, 2024

jonasnick commented at 9:53 pm on March 19, 2024: contributor

Can you try setting the additional sanitizer env variables that we set in the CI config?

Good point. Still 0 crashes in 10 runs of

0UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1" ASAN_OPTIONS="strict_string_checks=1:detect_stack_use_after_return=1:detect_leaks=1" LSAN_OPTIONS="use_unaligned=1" WERROR_CFLAGS="-Werror -pedantic-errors" MAKEFLAGS="-j4" BUILD="check" ECMULTWINDOW="auto" ECMULTGENPRECISION="auto" ASM="auto" WIDEMUL="auto" WITH_VALGRIND="yes" EXTRAFLAGS="" EXPERIMENTAL="no" ECDH="yes" RECOVERY="yes" SCHNORRSIG="yes" MUSIG="yes" ELLSWIFT="yes" SECP256K1_TEST_ITERS="32" BENCH="yes" SECP256K1_BENCH_ITERS="2" CTIMETESTS="no" EXAMPLES="yes" HOST="" WRAPPER_CMD="" CC="clang" CFLAGS="-fsanitize=undefined,address -g" ./ci/ci.sh

real-or-random referenced this in commit 84a93de4d2 on Mar 19, 2024

jonasnick commented at 8:57 am on March 20, 2024: contributor

I can reproduce this now after looking into the bug reports. Need to set both

0vm.mmap_rnd_bits = 32
1vm.mmap_rnd_compat_bits = 16

on the host for the crashes to happen.

real-or-random closed this on Mar 20, 2024

real-or-random referenced this in commit 05bfab69ae on Mar 20, 2024

fanquake commented at 3:03 pm on March 21, 2024: member

Note that the upstream issue should now be fixed: https://github.com/actions/runner-images/issues/9491 (see also https://github.com/google/oss-fuzz/issues/11703).

real-or-random commented at 3:13 pm on March 21, 2024: contributor

Hm, okay, and this is their fix (it’s the same as our workaround): https://github.com/actions/runner-images/pull/9513/commits/7aba0ab8abda5860b693ed43d74241a652660356

I’d tend to keep the workaround in our code base. I think it’s good to have it for documentation purposes. We could maybe expand a bit on the comment, and explain that this will be resolved in future clang releases.

real-or-random commented at 9:03 am on March 22, 2024: contributor

We could maybe expand a bit on the comment, and explain that this will be resolved in future clang releases.

Let’s keep this open until https://github.com/llvm/llvm-project/commit/58f7251820b14c93168726a24816d8a094599be5 is in a LLVM release (check branch info displayed by GitHub or the table at https://github.com/google/sanitizers/issues/1614#issuecomment-2010316781). Then we can change the comment to say that the workaround can be removed once we move to that clang version.

real-or-random reopened this on Mar 22, 2024

fanquake commented at 9:37 am on March 25, 2024: member

Let’s keep this open until https://github.com/llvm/llvm-project/commit/58f7251820b14c93168726a24816d8a094599be5 is in a LLVM release

Should be backported soon: https://github.com/llvm/llvm-project/pull/86201.

thurstond commented at 11:32 pm on March 27, 2024: none

Let’s keep this open until llvm/llvm-project@58f7251 is in a LLVM release

Should be backported soon: llvm/llvm-project#86201.

Backport has landed in the 18.x branch (release pending): https://github.com/llvm/llvm-project/commit/c2a57034eff048cd36c563c8e0051db3a70991b3

real-or-random commented at 2:45 am on March 28, 2024: contributor

@thurstond Thanks for the explicit notification!

fanquake commented at 8:34 am on April 4, 2024: member

LLVM 18.1.3 is out, which contains the fix: https://github.com/llvm/llvm-project/releases/tag/llvmorg-18.1.3.

real-or-random referenced this in commit 070b2c98d5 on Apr 4, 2024

real-or-random referenced this in commit a40768a3f5 on Apr 4, 2024

real-or-random referenced this in commit a85e2233e7 on Apr 4, 2024

real-or-random closed this on Apr 4, 2024

real-or-random referenced this in commit d8311688bd on Apr 4, 2024

Clang sanitizer binaries segfault on CI #1506