tracing: test interface_usdt_net.py fails due to garbage message type in net:outbound_message tracepoint #33246

issue theStack openend this issue on August 23, 2025
  1. theStack commented at 11:25 am on August 23, 2025: contributor

    The functional test interface_usdt_net.py currently fails on master (commit 73220fc0f958f9b65f66cf0cf042af220b312fc6) on my aarch64 machine, running Ubuntu 25.04:

     0$ cmake -B build -DWITH_USDT=ON
     1$ cmake --build build -j12
     2# ./build/test/functional/interface_usdt_net.py
     32025-08-23T10:53:12.137229Z TestFramework (INFO): PRNG seed is: 2587673157417500984
     42025-08-23T10:53:12.137976Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_iem787yt
     52025-08-23T10:53:12.415409Z TestFramework (INFO): hook into the net:inbound_message and net:outbound_message tracepoints
     62025-08-23T10:53:13.229315Z TestFramework (INFO): connect a P2P test node to our bitcoind node
     72025-08-23T10:53:13.336504Z TestFramework (INFO): check receipt and content of in- and outbound version messages
     82025-08-23T10:53:13.336668Z TestFramework (INFO): check_p2p_message(): inbound P2PMessage(peer=0, addr=127.0.0.1:54608, conn_type=inbound, msg_type=version, msg_size=111)
     92025-08-23T10:53:13.341459Z TestFramework (ERROR): Unexpected exception
    10Traceback (most recent call last):
    11  File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 195, in main
    12    self.run_test()
    13    ~~~~~~~~~~~~~^^
    14  File "/home/thestack/bitcoin/./build/test/functional/interface_usdt_net.py", line 260, in run_test
    15    self.p2p_message_tracepoint_test()
    16    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
    17  File "/home/thestack/bitcoin/./build/test/functional/interface_usdt_net.py", line 334, in p2p_message_tracepoint_test
    18    check_p2p_message(event, is_inbound)
    19    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
    20  File "/home/thestack/bitcoin/./build/test/functional/interface_usdt_net.py", line 300, in check_p2p_message
    21    if event.msg_type.decode("utf-8") == "version":
    22       ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
    23UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 0: invalid start byte
    24.....
    

    The message type parameter in the net:outbound_message tracepoint seems to point to garbage at the time it is evaluated in the probe, as printing out the event.msg_type contents over a few test runs shows:

    0event.msg_type == b'\xa8\xc2\xffG\xff\xff'
    1event.msg_type == b'\xa8\xb2\xfbt\xff\xff'
    2event.msg_type == b'\xa8\xb2|i\xff\xff'
    3event.msg_type == b'\xa8\xc2\xffo\xff\xff'
    4event.msg_type == b'\xa8\xc2~S\xff\xff'
    5.....
    

    It looks like a potential life-time issue, but it’s unclear to me how this could be possible at all (see also e.g. #27380 (comment)), given that probes should be executed synchronously and the relevant msgtype object is clearly still alive after the tracepoint call (msg gets std::moved later in the function though): https://github.com/bitcoin/bitcoin/blob/73220fc0f958f9b65f66cf0cf042af220b312fc6/src/net.cpp#L3885

    OS / compiler / packages:

    0$ head -1 /etc/issue
    1Ubuntu 25.04 \n \l
    2$ uname -s -r -v -m
    3Linux 6.16.0-21-qcom-x1e [#21](/bitcoin-bitcoin/21/)-Ubuntu SMP PREEMPT_DYNAMIC Sun Aug 10 00:54:48 UTC 2025 aarch64
    4$ gcc --version | head -1
    5gcc (Ubuntu 14.2.0-19ubuntu2) 14.2.0
    6$ apt info systemtap-sdt-dev | grep Version
    7Version: 5.1-4.1
    
  2. achow101 commented at 11:28 pm on August 26, 2025: member
    I’m able to reproduce this with a release build, but not a debug build.
  3. maflcko commented at 10:27 am on August 27, 2025: member
    I can reproduce via -DAPPEND_CXXFLAGS='-O1 -g2', but it passes via -DAPPEND_CXXFLAGS='-O1 -fno-inline -g2', so I wonder if this is a compiler error, or if the error is something else.
  4. 0xB10C commented at 8:44 pm on September 2, 2025: contributor

    Can you share the output of readelf -n ./build/bin/bitcoind | grep outbound_message -A 5 -B 3?

    I wonder if this is related to https://github.com/bitcoin/bitcoin/issues/33227

  5. theStack commented at 9:25 pm on September 2, 2025: contributor

    Can you share the output of readelf -n ./build/bin/bitcoind | grep outbound_message -A 5 -B 3?

    Sure:

    0$ readelf -n ./build/bin/bitcoind | grep outbound_message -A 5 -B 3
    1    Arguments: -8@x21 8@x22 8@x23 4@x20 8@x0
    2  stapsdt              0x00000057       NT_STAPSDT (SystemTap probe descriptors)
    3    Provider: net
    4    Name: outbound_message
    5    Location: 0x00000000000cf3ac, Base: 0x00000000009ca828, Semaphore: 0x0000000000b19428
    6    Arguments: -8@x21 8@x24 8@[sp, 136] 8@[x0] 8@x1 8@x2
    7  stapsdt              0x00000044       NT_STAPSDT (SystemTap probe descriptors)
    8    Provider: net
    9    Name: misbehaving_connection
    

    I can reproduce via -DAPPEND_CXXFLAGS='-O1 -g2', but it passes via -DAPPEND_CXXFLAGS='-O1 -fno-inline -g2', so I wonder if this is a compiler error, or if the error is something else.

    Can confirm that when I build with -DAPPEND_CXXFLAGS='-fno-inline', the problem doesn’t occur anymore (didn’t try any explicit optimization level / debug options yet).

  6. 0xB10C commented at 8:20 am on September 3, 2025: contributor
    Arguments: -8@x21 8@x24 8@[sp, 136] 8@[x0] 8@x1 8@x2
    

    Are these the same if you build with -DAPPEND_CXXFLAGS='-fno-inline'?

  7. theStack commented at 12:43 pm on September 3, 2025: contributor
    0Arguments: -8@x21 8@x24 8@[sp, 136] 8@[x0] 8@x1 8@x2
    

    Are these the same if you build with -DAPPEND_CXXFLAGS='-fno-inline'?

    They are quite different on a -fno-inline build:

    0    Name: outbound_message
    1    Location: 0x000000000012ed28, Base: 0x000000000111c6d8, Semaphore: 0x00000000016594d0
    2    Arguments: -8@x23 8@x24 8@x25 8@x26 8@x27 8@x0
    
  8. 0xB10C commented at 1:10 pm on September 3, 2025: contributor

    These arguments translate to the net:outbound_message tracepoint arguments.

    arg broken -fno-inline working
    pnode->GetId() -8@x21 -8@x23
    pnode->m_addr_name.c_str() 8@x24 8@x24
    pnode->ConnectionTypeAsString().c_str() 8@[sp, 136] 8@x25
    msg.m_type.c_str() 8@[x0] 8@x26
    msg.data.size() 8@x1 8@x27
    msg.data.data() 8@x2 8@x0

    So I suspect there’s a problem with either how:

    • bcc on aarch64 reading the value from register x0 when it’s written as 8@[x0]
    • systemtap on aarch64 writes these ELF notes
  9. 0xB10C commented at 1:15 pm on September 3, 2025: contributor

    As per https://github.com/iovisor/bcc/blob/791bf81ad9ec18ce2899f79509e4de96ddc21e9d/src/cc/usdt/usdt_args.cc#L228-L231, bcc on aarch64 claims to support parsing 8@[x0] via [-]<size>@[<reg>].

    0  // Support the following argument patterns:
    1  //   [-]<size>@<value>, [-]<size>@<reg>, [-]<size>@[<reg>], or
    2  //   [-]<size>@[<reg>,<offset>]
    3  //   [-]<size>@[<reg>,<index_reg>]
    
  10. willcl-ark added the label Tests on Jan 15, 2026
  11. willcl-ark added the label Scripts and tools on Jan 15, 2026
  12. willcl-ark added the label interfaces on Jan 15, 2026

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-01-21 00:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me