tracing: test `interface_usdt_net.py` fails due to garbage message type in `net:outbound_message` tracepoint #33246

The functional test interface_usdt_net.py currently fails on master (commit 73220fc0f958f9b65f66cf0cf042af220b312fc6) on my aarch64 machine, running Ubuntu 25.04:

$ cmake -B build -DWITH_USDT=ON
$ cmake --build build -j12
# ./build/test/functional/interface_usdt_net.py
2025-08-23T10:53:12.137229Z TestFramework (INFO): PRNG seed is: 2587673157417500984
2025-08-23T10:53:12.137976Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_iem787yt
2025-08-23T10:53:12.415409Z TestFramework (INFO): hook into the net:inbound_message and net:outbound_message tracepoints
2025-08-23T10:53:13.229315Z TestFramework (INFO): connect a P2P test node to our bitcoind node
2025-08-23T10:53:13.336504Z TestFramework (INFO): check receipt and content of in- and outbound version messages
2025-08-23T10:53:13.336668Z TestFramework (INFO): check_p2p_message(): inbound P2PMessage(peer=0, addr=127.0.0.1:54608, conn_type=inbound, msg_type=version, msg_size=111)
2025-08-23T10:53:13.341459Z TestFramework (ERROR): Unexpected exception
Traceback (most recent call last):
  File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 195, in main
    self.run_test()
    ~~~~~~~~~~~~~^^
  File "/home/thestack/bitcoin/./build/test/functional/interface_usdt_net.py", line 260, in run_test
    self.p2p_message_tracepoint_test()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/thestack/bitcoin/./build/test/functional/interface_usdt_net.py", line 334, in p2p_message_tracepoint_test
    check_p2p_message(event, is_inbound)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/thestack/bitcoin/./build/test/functional/interface_usdt_net.py", line 300, in check_p2p_message
    if event.msg_type.decode("utf-8") == "version":
       ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 0: invalid start byte
.....

The message type parameter in the net:outbound_message tracepoint seems to point to garbage at the time it is evaluated in the probe, as printing out the event.msg_type contents over a few test runs shows:

event.msg_type == b'\xa8\xc2\xffG\xff\xff'
event.msg_type == b'\xa8\xb2\xfbt\xff\xff'
event.msg_type == b'\xa8\xb2|i\xff\xff'
event.msg_type == b'\xa8\xc2\xffo\xff\xff'
event.msg_type == b'\xa8\xc2~S\xff\xff'
.....

It looks like a potential life-time issue, but it's unclear to me how this could be possible at all (see also e.g. #27380 (comment)), given that probes should be executed synchronously and the relevant msgtype object is clearly still alive after the tracepoint call (msg gets std::moved later in the function though): https://github.com/bitcoin/bitcoin/blob/73220fc0f958f9b65f66cf0cf042af220b312fc6/src/net.cpp#L3885

OS / compiler / packages:

$ head -1 /etc/issue
Ubuntu 25.04 \n \l
$ uname -s -r -v -m
Linux 6.16.0-21-qcom-x1e [#21](/bitcoin-bitcoin/21/)-Ubuntu SMP PREEMPT_DYNAMIC Sun Aug 10 00:54:48 UTC 2025 aarch64
$ gcc --version | head -1
gcc (Ubuntu 14.2.0-19ubuntu2) 14.2.0
$ apt info systemtap-sdt-dev | grep Version
Version: 5.1-4.1

I'm able to reproduce this with a release build, but not a debug build.

I can reproduce via -DAPPEND_CXXFLAGS='-O1 -g2', but it passes via -DAPPEND_CXXFLAGS='-O1 -fno-inline -g2', so I wonder if this is a compiler error, or if the error is something else.

Can you share the output of readelf -n ./build/bin/bitcoind | grep outbound_message -A 5 -B 3?

I wonder if this is related to https://github.com/bitcoin/bitcoin/issues/33227

Can you share the output of readelf -n ./build/bin/bitcoind | grep outbound_message -A 5 -B 3?

Sure:

$ readelf -n ./build/bin/bitcoind | grep outbound_message -A 5 -B 3
    Arguments: -8@x21 8@x22 8@x23 4@x20 8@x0
  stapsdt              0x00000057       NT_STAPSDT (SystemTap probe descriptors)
    Provider: net
    Name: outbound_message
    Location: 0x00000000000cf3ac, Base: 0x00000000009ca828, Semaphore: 0x0000000000b19428
    Arguments: -8@x21 8@x24 8@[sp, 136] 8@[x0] 8@x1 8@x2
  stapsdt              0x00000044       NT_STAPSDT (SystemTap probe descriptors)
    Provider: net
    Name: misbehaving_connection

I can reproduce via -DAPPEND_CXXFLAGS='-O1 -g2', but it passes via -DAPPEND_CXXFLAGS='-O1 -fno-inline -g2', so I wonder if this is a compiler error, or if the error is something else.

Can confirm that when I build with -DAPPEND_CXXFLAGS='-fno-inline', the problem doesn't occur anymore (didn't try any explicit optimization level / debug options yet).

Arguments: -8@x21 8@x24 8@[sp, 136] 8@[x0] 8@x1 8@x2

Are these the same if you build with -DAPPEND_CXXFLAGS='-fno-inline'?

Arguments: -8@x21 8@x24 8@[sp, 136] 8@[x0] 8@x1 8@x2

Are these the same if you build with -DAPPEND_CXXFLAGS='-fno-inline'?

They are quite different on a -fno-inline build:

    Name: outbound_message
    Location: 0x000000000012ed28, Base: 0x000000000111c6d8, Semaphore: 0x00000000016594d0
    Arguments: -8@x23 8@x24 8@x25 8@x26 8@x27 8@x0

These arguments translate to the net:outbound_message tracepoint arguments.

So I suspect there's a problem with either how:

• bcc on aarch64 reading the value from register x0 when it's written as 8@[x0]
• systemtap on aarch64 writes these ELF notes

As per https://github.com/iovisor/bcc/blob/791bf81ad9ec18ce2899f79509e4de96ddc21e9d/src/cc/usdt/usdt_args.cc#L228-L231, bcc on aarch64 claims to support parsing 8@[x0] via [-]<size>@[<reg>].

  // Support the following argument patterns:
  //   [-]<size>@<value>, [-]<size>@<reg>, [-]<size>@[<reg>], or
  //   [-]<size>@[<reg>,<offset>]
  //   [-]<size>@[<reg>,<index_reg>]