Currently, if the tracepoints are compiled (e.g. in depends and release builds), we always prepare the tracepoint arguments regardless of the tracepoints being used or not. We made sure that the argument preparation is as cheap as possible, but we can almost completely eliminate any overhead for users not interested in the tracepoints (the vast majority), by gating the tracepoint argument preparation with an if(something is attached to this tracepoint). To achieve this, we use the optional semaphore feature provided by SystemTap.
The first commit simplifies and deduplicates our tracepoint macros from 13 TRACEx macros to a single TRACEPOINT macro. This makes them easier to use and also avoids more duplicate macro definitions in the second commit.
The Linux tracing tools I’m aware of (bcc, bpftrace, libbpf, and systemtap) all support the semaphore gating feature. Thus, all existing tracepoints and their argument preparation is gated in the second commit. For details, please refer to the commit messages and the updated documentation in doc/tracing.md.
Also adding unit tests that include all tracepoint macros to make sure there are no compiler problems with them (e.g. some varadiac extension not supported).
Reviewers might want to check:
- Do the tracepoints still work for you? Do the examples in
contrib/tracing/run on your system (as bpftrace frequently breaks on every new version, please test master too if it should’t work for you)? Do the CI interface tests still pass? - Is the new documentation clear?
- The
TRACEPOINT_SEMAPHORE(event, context)macros places global variables in our global namespace. Is this something we strictly want to avoid or maybe move to allTRACEPOINT_SEMAPHOREs to a separate .cpp file or even namespace? I like having theTRACEPOINT_SEMAPHORE()in same file as theTRACEPOINT(), but open for suggestion on alternative approaches. - Are newly added tracepoints in the unit tests visible when using
readelf -n /src/test/test_bitcoin? You can run the new unit tests with./src/test/test_bitcoin --run_test=util_trace_tests* --log_level=all.
fail_tests.bt:
0#!/usr/bin/env bpftrace
1
2usdt:./src/test/test_bitcoin:test:check_if_attached {
3 printf("the 'check_if_attached' test should have failed\n");
4}
5
6usdt:./src/test/test_bitcoin:test:expensive_section {
7 printf("the 'expensive_section' test should have failed\n");
8}
Run the unit tests with ./src/test/test_bitcoin and start bpftrace fail_tests.bt -p $(pidof test_bitcoin) in a separate terminal. The unit tests should fail with:
0Running 594 test cases...
1test/util_trace_tests.cpp(31): error: in "util_trace_tests/test_tracepoint_check_if_attached": check false has failed
2test/util_trace_tests.cpp(51): error: in "util_trace_tests/test_tracepoint_manual_tracepoint_active_check": check false has failed
3
4*** 2 failures are detected in the test module "Bitcoin Core Test Suite"
These links might provide more contextual information for reviewers:
- How SystemTap Userspace Probes Work by eklitzke (actually an example on Bitcoin Core; mentions that with semaphores “the overhead for an untraced process is effectively zero.”)
- libbpf comment on USDT semaphore handling (can recommend the whole comment for background on how the tracepoints and tracing tools work together)
- https://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation#Semaphore_Handling
