ci: USDT on mantic #28600

The asan CI uses Ubuntu 23.04, which doesn't have clang-17 and will also be EOL in January:

https://github.com/bitcoin/bitcoin/blob/52c6904c789103a316253df4cb20aa785b2afc42/ci/test/00_setup_env_native_asan.sh#L20

Thus, it would be good to bump to 23.10. However that fails, even outside the CI.

Steps to reproduce on a fresh install of Ubuntu 23.04/10:

export DEBIAN_FRONTEND=noninteractive && apt update && apt install curl wget htop git vim ccache -y && git clone https://github.com/bitcoin/bitcoin.git  --depth=1 ./bitcoin-core && cd bitcoin-core && apt install build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq     libevent-dev libboost-dev systemtap-sdt-dev clang-16 llvm-16 libclang-rt-16-dev  bpfcc-tools "linux-headers-$(uname --kernel-release)"  -y   &&  ./autogen.sh && ./configure   --enable-usdt CC=clang-16 CXX=clang++-16 && make -j $(nproc) && ./test/functional/interface_usdt_mempool.py

Output on 23.04: passes

On 23.10:

# ./test/functional/interface_usdt_mempool.py 
2023-10-05T15:22:55.501000Z TestFramework (INFO): PRNG seed is: 3896309821586557942
2023-10-05T15:22:55.502000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_930vj2wl
2023-10-05T15:22:55.852000Z TestFramework (INFO): Hooking into mempool:added tracepoint...
/virtual/main.c:97:24: error: call to undeclared function '_bpf_readarg_trace_removed_1'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  ({ u64 __addr = 0x0; _bpf_readarg_trace_removed_1(ctx, &__addr, sizeof(__addr));bpf_probe_read_user(&removed.hash, HASH_LENGTH, (void *)__addr);});
                       ^
/virtual/main.c:97:24: note: did you mean '_bpf_readarg_trace_added_1'?
/virtual/main.c:13:28: note: '_bpf_readarg_trace_added_1' declared here
static __always_inline int _bpf_readarg_trace_added_1(struct pt_regs *ctx, void *dest, size_t len) {
                           ^
/virtual/main.c:98:24: error: call to undeclared function '_bpf_readarg_trace_removed_2'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  ({ u64 __addr = 0x0; _bpf_readarg_trace_removed_2(ctx, &__addr, sizeof(__addr));bpf_probe_read_user(&removed.reason, MAX_REMOVAL_REASON_LENGTH, (void *)__addr);});
                       ^
/virtual/main.c:98:24: note: did you mean '_bpf_readarg_trace_added_2'?
/virtual/main.c:19:28: note: '_bpf_readarg_trace_added_2' declared here
static __always_inline int _bpf_readarg_trace_added_2(struct pt_regs *ctx, void *dest, size_t len) {
                           ^
/virtual/main.c:99:3: error: call to undeclared function '_bpf_readarg_trace_removed_3'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  _bpf_readarg_trace_removed_3(ctx, &removed.vsize, sizeof(*(&removed.vsize)));
  ^
/virtual/main.c:99:3: note: did you mean '_bpf_readarg_trace_added_3'?
/virtual/main.c:25:28: note: '_bpf_readarg_trace_added_3' declared here
static __always_inline int _bpf_readarg_trace_added_3(struct pt_regs *ctx, void *dest, size_t len) {
                           ^
/virtual/main.c:100:3: error: call to undeclared function '_bpf_readarg_trace_removed_4'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  _bpf_readarg_trace_removed_4(ctx, &removed.fee, sizeof(*(&removed.fee)));
  ^
/virtual/main.c:101:3: error: call to undeclared function '_bpf_readarg_trace_removed_5'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  _bpf_readarg_trace_removed_5(ctx, &removed.entry_time, sizeof(*(&removed.entry_time)));
  ^
/virtual/main.c:112:24: error: call to undeclared function '_bpf_readarg_trace_rejected_1'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  ({ u64 __addr = 0x0; _bpf_readarg_trace_rejected_1(ctx, &__addr, sizeof(__addr));bpf_probe_read_user(&rejected.hash, HASH_LENGTH, (void *)__addr);});
                       ^
/virtual/main.c:112:24: note: did you mean '_bpf_readarg_trace_added_1'?
/virtual/main.c:13:28: note: '_bpf_readarg_trace_added_1' declared here
static __always_inline int _bpf_readarg_trace_added_1(struct pt_regs *ctx, void *dest, size_t len) {
                           ^
/virtual/main.c:113:24: error: call to undeclared function '_bpf_readarg_trace_rejected_2'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  ({ u64 __addr = 0x0; _bpf_readarg_trace_rejected_2(ctx, &__addr, sizeof(__addr));bpf_probe_read_user(&rejected.reason, MAX_REJECT_REASON_LENGTH, (void *)__addr);});
                       ^
/virtual/main.c:113:24: note: did you mean '_bpf_readarg_trace_added_2'?
/virtual/main.c:19:28: note: '_bpf_readarg_trace_added_2' declared here
static __always_inline int _bpf_readarg_trace_added_2(struct pt_regs *ctx, void *dest, size_t len) {
                           ^
/virtual/main.c:124:24: error: call to undeclared function '_bpf_readarg_trace_replaced_1'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  ({ u64 __addr = 0x0; _bpf_readarg_trace_replaced_1(ctx, &__addr, sizeof(__addr));bpf_probe_read_user(&replaced.replaced_hash, HASH_LENGTH, (void *)__addr);});
                       ^
/virtual/main.c:124:24: note: did you mean '_bpf_readarg_trace_added_1'?
/virtual/main.c:13:28: note: '_bpf_readarg_trace_added_1' declared here
static __always_inline int _bpf_readarg_trace_added_1(struct pt_regs *ctx, void *dest, size_t len) {
                           ^
/virtual/main.c:125:3: error: call to undeclared function '_bpf_readarg_trace_replaced_2'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  _bpf_readarg_trace_replaced_2(ctx, &replaced.replaced_vsize, sizeof(*(&replaced.replaced_vsize)));
  ^
/virtual/main.c:125:3: note: did you mean '_bpf_readarg_trace_added_2'?
/virtual/main.c:19:28: note: '_bpf_readarg_trace_added_2' declared here
static __always_inline int _bpf_readarg_trace_added_2(struct pt_regs *ctx, void *dest, size_t len) {
                           ^
/virtual/main.c:126:3: error: call to undeclared function '_bpf_readarg_trace_replaced_3'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  _bpf_readarg_trace_replaced_3(ctx, &replaced.replaced_fee, sizeof(*(&replaced.replaced_fee)));
  ^
/virtual/main.c:127:3: error: call to undeclared function '_bpf_readarg_trace_replaced_4'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  _bpf_readarg_trace_replaced_4(ctx, &replaced.replaced_entry_time, sizeof(*(&replaced.replaced_entry_time)));
  ^
/virtual/main.c:128:24: error: call to undeclared function '_bpf_readarg_trace_replaced_5'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  ({ u64 __addr = 0x0; _bpf_readarg_trace_replaced_5(ctx, &__addr, sizeof(__addr));bpf_probe_read_user(&replaced.replacement_hash, HASH_LENGTH, (void *)__addr);});
                       ^
/virtual/main.c:129:3: error: call to undeclared function '_bpf_readarg_trace_replaced_6'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  _bpf_readarg_trace_replaced_6(ctx, &replaced.replacement_vsize, sizeof(*(&replaced.replacement_vsize)));
  ^
/virtual/main.c:130:3: error: call to undeclared function '_bpf_readarg_trace_replaced_7'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  _bpf_readarg_trace_replaced_7(ctx, &replaced.replacement_fee, sizeof(*(&replaced.replacement_fee)));
  ^
14 errors generated.
2023-10-05T15:22:57.476000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/root/bitcoin-core/test/functional/test_framework/test_framework.py", line 131, in main
    self.run_test()
  File "/root/bitcoin-core/./test/functional/interface_usdt_mempool.py", line 317, in run_test
    self.added_test()
  File "/root/bitcoin-core/./test/functional/interface_usdt_mempool.py", line 146, in added_test
    bpf = BPF(text=MEMPOOL_TRACEPOINTS_PROGRAM, usdt_contexts=[ctx], debug=0)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 479, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>
2023-10-05T15:22:57.528000Z TestFramework (INFO): Stopping nodes
2023-10-05T15:22:57.580000Z TestFramework (WARNING): Not cleaning up dir /tmp/bitcoin_func_test_930vj2wl
2023-10-05T15:22:57.580000Z TestFramework (ERROR): Test failed. Test logging available at /tmp/bitcoin_func_test_930vj2wl/test_framework.log
2023-10-05T15:22:57.580000Z TestFramework (ERROR): 
2023-10-05T15:22:57.580000Z TestFramework (ERROR): Hint: Call /root/bitcoin-core/test/functional/combine_logs.py '/tmp/bitcoin_func_test_930vj2wl' to consolidate all logs
2023-10-05T15:22:57.580000Z TestFramework (ERROR): 
2023-10-05T15:22:57.580000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2023-10-05T15:22:57.580000Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues
2023-10-05T15:22:57.580000Z TestFramework (ERROR): 

@0xB10C

As discussed offline, it looks like these errors are coming from the bpfcc-tools package version 0.26.0+ds-1ubuntu2 in Ubuntu 23.10 (mantic). For 23.04 (lunar) it's version 0.26.0+ds-1ubuntu1. As we need to upgrade to 23.10, maybe switching the USDT tests to a different job not using Ubuntu is a solution for now?

If no one fixes this issue within the next two months, I'd suggest to remove the bpfcc-tools package and not run the tests, temporarily, for now (TM).

As discussed offline, it looks like these errors are coming from the bpfcc-tools package version 0.26.0+ds-1ubuntu2

The changelog between these versions looks very small, and likely unrelated? It just adds riscv build:

diff -Nru bpfcc-0.26.0+ds/debian/changelog bpfcc-0.26.0+ds/debian/changelog
--- bpfcc-0.26.0+ds/debian/changelog	2023-02-14 13:52:10.000000000 +0000
+++ bpfcc-0.26.0+ds/debian/changelog	2023-07-31 11:51:23.000000000 +0000
@@ -1,3 +1,9 @@
+bpfcc (0.26.0+ds-1ubuntu2) mantic; urgency=medium
+
+  * Build for riscv64.
+
+ -- Matthias Klose <doko@ubuntu.com>  Mon, 31 Jul 2023 13:51:23 +0200
+
 bpfcc (0.26.0+ds-1ubuntu1) lunar; urgency=low
 
   * Merge from Debian unstable. Remaining changes:
diff -Nru bpfcc-0.26.0+ds/debian/control bpfcc-0.26.0+ds/debian/control
--- bpfcc-0.26.0+ds/debian/control	2023-02-13 09:16:10.000000000 +0000
+++ bpfcc-0.26.0+ds/debian/control	2023-07-31 10:43:48.000000000 +0000
@@ -38,7 +38,7 @@
 Rules-Requires-Root: no
 
 Package: libbpfcc
-Architecture: amd64 arm64 ppc64el ppc64 s390x armhf
+Architecture: amd64 arm64 ppc64el ppc64 s390x armhf riscv64
 Depends: ${misc:Depends}, ${shlibs:Depends}
 Multi-Arch: same
 Description: shared library for BPF Compiler Collection (BCC)
@@ -52,7 +52,7 @@
  to control BPF programs from userspace.
 
 Package: libbpfcc-dev
-Architecture: amd64 arm64 ppc64el ppc64 s390x armhf
+Architecture: amd64 arm64 ppc64el ppc64 s390x armhf riscv64
 Section: libdevel
 Depends: ${misc:Depends},
  libbpfcc (= ${binary:Version})
@@ -114,7 +114,7 @@
  At this time this package contains subset of tools from bpfcc-tools
 
 Package: bpfcc-introspection
-Architecture: amd64 arm64 ppc64el ppc64
+Architecture: amd64 arm64 ppc64el ppc64 riscv64
 Depends: ${misc:Depends},
  ${shlibs:Depends},
  libbpfcc (= ${binary:Version})
diff -Nru bpfcc-0.26.0+ds/debian/rules bpfcc-0.26.0+ds/debian/rules
--- bpfcc-0.26.0+ds/debian/rules	2023-02-13 09:16:10.000000000 +0000
+++ bpfcc-0.26.0+ds/debian/rules	2023-07-31 09:37:01.000000000 +0000
@@ -8,7 +8,7 @@
 
 # for building libbpf-tools on selective architecture
 include /usr/share/dpkg/architecture.mk
-libbpf_tool_skip_arch := armhf s390x ppc64
+libbpf_tool_skip_arch := armhf s390x ppc64 riscv64
 
 export BPFTOOL=$(shell echo /usr/lib/linux-tools-*/bpftool)
 # Disable tests as they require root to run

source: https://launchpadlibrarian.net/679695359/bpfcc_0.26.0+ds-1ubuntu1_0.26.0+ds-1ubuntu2.diff.gz

The changelog between these versions looks very small, and likely unrelated? It just adds riscv build:

Pretty sure the build system changes between the two Ubuntu versions, so the compiler used to compile is different.

If the compiler is also embedded or used by bpfcc, it will also affect the package (indirectly) without anything in the package changing.

So somehow pinning the gcc version to the previous one could be a workaround?

The changelog between these versions looks very small, and likely unrelated? It just adds riscv build:

Pretty sure the build system changes between the two Ubuntu versions, so the compiler used to compile is different.

If the compiler is also embedded or used by bpfcc, it will also affect the package (indirectly) without anything in the package changing.

Ah that makes sense. Will have a play around with it in a container and see if I can find a fix

I think this can be stop-gap-fixed by setting -Wno-error=implicit-function-declaration on the trace point program build, as done here: https://github.com/bitcoin/bitcoin/compare/master...willcl-ark:bitcoin:mantis-usdt-fix

As this will remain unsupported into the future, this doesn't seem to be a particularly resilient fix, IMO. However I am unsure if the "correct" way of fixing is to actually re-declare all the functions (which seems undesirable) or to wait for another upstream fix... There is a similar issue in the bcc repo: https://github.com/iovisor/bcc/issues/4740#issuecomment-1721830957 where the advice given was to re-declare all the functions.

What do you think @maflcko @0xB10C ?

edit: my commit message from the commit above also contains more background:

Recently usage of undeclared functions became an error, rather than a warning. https://reviews.llvm.org/D122983?id=420290

This change seems to have migrated into the build tools of Ubuntu 23.10, which now causes the USDT tests to fail to compile, see #28600

Fix this by setting -Wno-error=implicit-function-declaration for the traceing programs.

Happy to test your commit if you submit a PR