[WIP] ci: test valgrind build from source #136

1. 
   fanquake commented at 4:37 pm on June 27, 2023: member
2. [WIP] ci: test valgrind build from source 7477440ce5
3. 
   fanquake commented at 9:25 am on June 28, 2023: member

   Looks like this is running faster. i.e, comparing a recent PR run:

    0Run base_encode_decode with args ['valgrind', '--quiet', '--error-exitcode=1', '/tmp/cirrus-build/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/fuzz/fuzz', '-runs=1', PosixPath('/tmp/cirrus-build/bitcoin-core/ci/scratch/qa-assets/fuzz_seed_corpus/base_encode_decode')]INFO: Running with entropic power schedule (0xFF, 100).
    1INFO: Seed: 3272178655
    2INFO: Loaded 1 modules   (248538 inline 8-bit counters): 248538 [0x27fd278, 0x2839d52), 
    3INFO: Loaded 1 PC tables (248538 PCs): 248538 [0x2839d58,0x2c04af8), 
    4INFO:     7233 files found in /tmp/cirrus-build/bitcoin-core/ci/scratch/qa-assets/fuzz_seed_corpus/base_encode_decode
    5INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
    6INFO: seed corpus: files: 7233 min: 1b max: 1048576b total: 83751368b rss: 180Mb
    7[#128](/bitcoin-core-qa-assets/128/)	pulse  cov: 815 ft: 1562 corp: 56/240b exec/s: 64 rss: 183Mb
    8[#256](/bitcoin-core-qa-assets/256/)	pulse  cov: 1016 ft: 1941 corp: 96/697b exec/s: 64 rss: 206Mb
    9[#512](/bitcoin-core-qa-assets/512/)	pulse  cov: 1411 ft: 2848 corp: 137/1524b exec/s: 73 rss: 218Mb
   10[#1024](/bitcoin-core-qa-assets/1024/)	pulse  cov: 2596 ft: 7733 corp: 274/10217b exec/s: 37 rss: 237Mb
   11[#2048](/bitcoin-core-qa-assets/2048/)	pulse  cov: 4184 ft: 13749 corp: 519/51Kb exec/s: 26 rss: 247Mb
   12[#4096](/bitcoin-core-qa-assets/4096/)	pulse  cov: 4641 ft: 19775 corp: 906/201Kb exec/s: 20 rss: 253Mb
   13Slowest unit: 13 s:
   14artifact_prefix='./'; Test unit written to ./slow-unit-4bbf474818229be7c3eaa56e66d8a671f61a8c4b
   15Slowest unit: 15 s:
   16artifact_prefix='./'; Test unit written to ./slow-unit-434da0af3356a442e2e6ea3bcef590755c6574e0
   17[#7234](/bitcoin-core-qa-assets/7234/)	INITED cov: 4730 ft: 25892 corp: 1581/12441Kb exec/s: 5 rss: 295Mb
   18[#7234](/bitcoin-core-qa-assets/7234/)	DONE   cov: 4730 ft: 25892 corp: 1581/12441Kb lim: 1048576 exec/s: 5 rss: 295Mb
   19Done 7234 runs in 1246 second(s)
   

   to this branch:

    0Run base_encode_decode with args ['valgrind', '--quiet', '--error-exitcode=1', '/tmp/cirrus-build-1174734651/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/fuzz/fuzz', '-runs=1', PosixPath('/tmp/cirrus-build-1174734651/bitcoin-core/ci/scratch/qa-assets/fuzz_seed_corpus/base_encode_decode')]INFO: Running with entropic power schedule (0xFF, 100).
    1INFO: Seed: 3639341585
    2INFO: Loaded 1 modules   (366489 inline 8-bit counters): 366489 [0x1c106d0, 0x1c69e69), 
    3INFO: Loaded 1 PC tables (366489 PCs): 366489 [0x1c69e70,0x2201800), 
    4INFO:     7233 files found in /tmp/cirrus-build-1174734651/bitcoin-core/ci/scratch/qa-assets/fuzz_seed_corpus/base_encode_decode
    5INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
    6INFO: seed corpus: files: 7233 min: 1b max: 1048576b total: 83751368b rss: 212Mb
    7[#256](/bitcoin-core-qa-assets/256/)	pulse  cov: 407 ft: 683 corp: 105/715b exec/s: 128 rss: 236Mb
    8[#512](/bitcoin-core-qa-assets/512/)	pulse  cov: 545 ft: 935 corp: 155/1712b exec/s: 128 rss: 245Mb
    9[#1024](/bitcoin-core-qa-assets/1024/)	pulse  cov: 1182 ft: 2772 corp: 321/12232b exec/s: 102 rss: 256Mb
   10[#2048](/bitcoin-core-qa-assets/2048/)	pulse  cov: 2032 ft: 5181 corp: 640/66Kb exec/s: 93 rss: 268Mb
   11[#4096](/bitcoin-core-qa-assets/4096/)	pulse  cov: 2481 ft: 8460 corp: 1223/292Kb exec/s: 81 rss: 278Mb
   12[#7234](/bitcoin-core-qa-assets/7234/)	INITED cov: 2588 ft: 11977 corp: 2183/10809Kb exec/s: 19 rss: 312Mb
   13[#7234](/bitcoin-core-qa-assets/7234/)	DONE   cov: 2588 ft: 11977 corp: 2183/10809Kb lim: 1042864 exec/s: 19 rss: 312Mb
   14Done 7234 runs in 373 second(s)
   

4. 
   maflcko commented at 9:28 am on June 28, 2023: contributor

   Again, are you sure you aren’t missing build options?
5. 
   fanquake commented at 9:35 am on June 28, 2023: member

   Again, are you sure you aren’t missing build options?

   The only option I can see passed by debian is --enable-tls, which is only used in valgrinds unit tests.

6. 
   maflcko commented at 9:41 am on June 28, 2023: contributor

   They may also pass some CFLAGS etc?
7. 
   maflcko commented at 9:42 am on June 28, 2023: contributor

   Also in the commit description you mention:

   0* No need to "bless" a specific compiler for use with Valgrind, (current
   1  discussion includes switch from Clang to GCC)
   

   Which seems unrelated, no? Or is compiling from source also going to fix that valgrind gcc false positive?

8. 
   fanquake commented at 9:44 am on June 28, 2023: member

   They may also pass some CFLAGS etc?

   They may do, and that could be the reason their binaries are so slow (i.e hardening flags). Unless you can point to something, I very much doubt that any additional cflags would be controlling valgrind features.

   Which seems unrelated, no?

   No. It’s related given we are/were considering dropping support for Clang and using GCC? Hence ending up with a “blessed” compiler.

9. 
   fanquake commented at 9:49 am on June 28, 2023: member

   Looking at https://buildd.debian.org/status/fetch.php?pkg=valgrind&arch=amd64&ver=1%3A3.19.0-1&stamp=1661686809&raw=0, which should be a fairly recent Debian valgrind build, the relevant ./configure / flags are:

   0dh_auto_configure -- --enable-tls CFLAGS="-g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 "-I/usr/include/x86_64-linux-gnu"   " LDFLAGS="-Wl,-z,relro"
   1	./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=\${prefix}/include --mandir=\${prefix}/share/man --infodir=\${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=\${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --enable-tls "CFLAGS=-g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include/x86_64-linux-gnu   " LDFLAGS=-Wl,-z,relro
   

   Other than --enable-tls, these are all standard autoconf flags, and I can’t see any special c/ldflags etc.

10. fanquake referenced this in commit 498ba4fc02 on Jun 28, 2023
11. fanquake referenced this in commit 29850324e9 on Jun 28, 2023
12. fanquake referenced this in commit 685a6846a1 on Jun 28, 2023
13. 
    fanquake commented at 2:36 pm on June 28, 2023: member

    Will close for now. This has completed in 5 and a half hours, compared to a recent 34 hour runtime.
14. fanquake closed this on Jun 28, 2023

    ---

15. fanquake referenced this in commit 83266cd4b5 on Jul 4, 2023

Contributors
fanquake maflcko