Tracepoint-like interface via libmultiprocess and IPC communication #185

1. 
   0xB10C commented at 3:39 pm on June 26, 2025: none

   Bitcoin Core’s current tracing and tracepoint implementation uses systemtap probes and eBPF for tracing (i.e. passing internal events to other processes via the linux kernel). While a powerful interface, it has its short comings:

   • the current tracepoint marco definitions only work on Linux, thus it can currently only be used on Linux. While e.g. macOS and FreeBSD is possible, there hasn’t been much developer interest in implementing them. And even if implemented, who is going to test and maintain them. The Linux scripts won’t work on other platforms.
   • loading eBPF scripts into the Linux kernel requires privileges, which makes the current tracing scripts harder to develop, use, and test in CI
   • the Python tooling we use to test these tracepoints, for example BCC, has been troublesome in our CI from time to time
   • the interfaces are not typed: we pass strings as C-style strings, and we can’t pass objects, lists, or more than 12 arguments to a tracepoint
   • …

   In a recent discussion I had, it came up if a tracepoint-like IPC interface could be an alternative. Superficially, this would have a few benefits:

   • Available on all platforms, and doesn’t need per-platform code
   • Connecting a tracing tool via IPC doesn’t require privileges, makes it easier to develop, use, and test
   • no extra tooling needed to test the ipc-tracepoints: interface can be tested alongside the other IPC interfaces
   • interface definitions will be clear, and data is typed. objects, lists, and strings are possible and a lot less error prone

   I have a few questions to figure out if this could be an alternative:

   1. I’m aware that there is a streaming-pattern for capnproto. Could something like the following work with libmultiprocess:
      • a client tells the node “I want to learn about incoming P2P message”
      • the node sends the client all incoming P2P messages as they arrive. Possibly implemented similarly to https://github.com/bitcoin/bitcoin/blob/67ea4b9994e668dcea5e5d0f62f886d92e3737dc/src/net_processing.cpp#L5042-L5051
      • the client disconnects, and the node stops sending incoming P2P messages
   2. One nice thing about the current tracepoint interface is that the overhead when not used is limited to a single instruction (check if someone is hooked into this tracepoint). And even when used, the overhead is minimal. Give then earlier example of passing all incoming P2P messages, is this doable over IPC without too much overhead? I guess the overhead is dependent on implementation (passing references or copies?), so this question is more about the gut feeling: yes this sounds doable / no, libmultiprocess/captnproto/IPC is not well suited to do this
2. 
   ryanofsky commented at 2:02 pm on June 27, 2025: collaborator

   so this question is more about the gut feeling: yes this sounds doable / no, libmultiprocess/captnproto/IPC is not well suited to do this

   I feel like it could work pretty well, but a naive implementation could have significantly more overhead than eBPF. I’m not sure how much of a concern overhead is if clients can choose which traces they are interested in, and there are not too many traces to send, but it’s worth thinking about.

   Basically the thing to be aware of is that I/O in Cap’n Proto is single threaded & asynchronous. So when other threads want to make or receive IPC calls, they can create and parse the messages (Cap’n Proto structs) themselves, but they have to pass the messages to and from the I/O thread in order to actually do the socket I/O. The I/O thread is typically idle, primarily handling wakeups from other threads, and making asynchronous I/O calls with a promise interface (very similar to javascript promises) to communicate with clients. But all of this may add overhead not present with eBPF.

   We could experiment and see if this is an issue. The Chain.handleNotifications method in https://github.com/bitcoin/bitcoin/pull/29409 demonstrates the streaming pattern you mentioned. The ChainNotifications interface argument to that method corresponds to the callback: Callback argument in the Cap’n Proto server -> client example, but instead of a single sendChunk method, it has multiple methods for different notifications that can be sent.

   It should be pretty easy to add a new Tracing interface similar to the Mining interface introduced in https://github.com/bitcoin/bitcoin/pull/30510 and if it would be helpful, I’d be happy to write a simple client & server implementation to show what this might look like.

   If overhead in a naive implementation were a problem, it’d be possible to make changes to reduce it. For example in the ChainNotifications interface example above, each time the node makes a transactionAddedToMempool or other method call to the client, the server blocks waiting for the client to process the call and respond. But since the transactionAddedToMempool method doesn’t return anything, the caller doesn’t actually need to block waiting for a response, and more optimally it might just send the call and continue without waiting. This would require implementing another form of flow control, however, otherwise a stuck client that wasn’t reading any data from the socket could cause unsent IPC calls to pile up in the server and make it run out of memory. This could be addressed by blocking or discarding new notifications when previous ones are backed up.

   Overall, I think an IPC tracing implementation could be practical and straightforward, and I’d be happy to provide a basic client/server example for testing. A simple implementation might have performance concerns which could be dealt with, though I wouldn’t want to overpromise, and my assumption would be that even an optimal IPC implementation would have more overhead than eBPF.

3. 
   0xB10C commented at 11:55 am on July 7, 2025: none

   Thanks for your detailed thoughts on this! Overall, this sounds promising and I’d be happy to experiment if IPC-overhead is a problem. I have a few ideas on how to test and compare this to the current eBPF alternative.

   It should be pretty easy to add a new Tracing interface similar to the Mining interface introduced in https://github.com/bitcoin/bitcoin/pull/30510 and if it would be helpful, I’d be happy to write a simple client & server implementation to show what this might look like.

   That would be great! I can pick it up from there then.

4. ryanofsky referenced this in commit 8e09fbe805 on Jul 7, 2025
5. 
   ryanofsky commented at 8:55 pm on July 7, 2025: collaborator

   That would be great! I can pick it up from there then.

   I opened https://github.com/bitcoin/bitcoin/pull/32898 with some code demonstrating using IPC for tracing that could help evaluate the approach. (I don’t think I’ll keep the PR open, but did want to post something more public that could get other feedback.)

6. ryanofsky closed this on Aug 23, 2025

   ---

7. 
   ryanofsky commented at 11:34 am on August 23, 2025: collaborator

   Will close, but feel free to reopen or post followup issues or questions

Contributors
0xB10C ryanofsky