RFC: separate kernel logging infrastructure #34062

issue stickies-v openend this issue on December 12, 2025
  1. stickies-v commented at 3:43 pm on December 12, 2025: contributor

    tl;dr: kernel logging is cumbersome. I propose we layer logging so we can enable structured logging and a much simplified kernel logging interface without overcomplicating node logging.

    Motivation

    The bitcoinkernel library (#27587) exposes functionality to interface with the kernel logging. This includes registering callbacks for log statements, level/category filtering, string formatting options, and more.

    Kernel logging has a few problems:

    • P1: callbacks operate on formatted strings, so users need to parse the string to get the timestamp, category, level, … based on which options are set. This is cumbersome, brittle, and inefficient.
    • P2: the filtering interface is not really intuitive, requiring users to call combinations of btck_logging_set_level_category and btck_logging_enable_category when they want to produce debug or trace logs. The level/category system makes sense for node, because it directly controls what gets written to disk and stdout, and there are quite a lot more categories producing logs. Kernel doesn’t really need this - users control what happens to the logs, and can do any filtering/manipulation in the callback they provide.
    • P3: the node logging infrastructure has quite a bit more functionality than is necessary for a library, including ratelimiting, log formatting, outputting, buffering, … This introduces unnecessary code and interface complexity.
    • P4: it uses the global (node) logger, which goes against the otherwise context local (or stateless) design of the bitcoinkernel interface.

    Approach

    I propose we address problems P1, P2 and P3 by layering the logging infrastructure. Roughly, this would mean:

    1. Implement a minimal util::log::Logger class (see Appendix) that dispatches structured log entries to registered callbacks. This is a low-level logger that does not perform formatting, rate limiting, or category filtering - it simply dispatches to callbacks.
    2. Refactor BCLog::Logger into BCLog::Sink, which registers a callback on util::log::GetLogger() and handles all node-specific functionality (category filtering, rate limiting, file output, formatting, …).
    3. Update kernel code to include util/log.h instead of logging.h. The existing logging macros (LogInfo, LogDebug, etc.) continue to work unchanged.
    4. Simplify bitcoinkernel C API to reflect the simpler util::log::Logger interface (structured logging, level-based, …), we can now completely remove btck_LoggingOptions.
    5. Remove logging.h and logging.cpp from the kernel library completely.

    Note: I’m leaving problem P4 out of scope for this proposal. I think it already has sufficient merit with a global logger, and I want to keep scope limited. However, I do believe layered logging infrastructure will make implementing P4 easier.

    Balance

    The main benefits and downsides of this approach:

    Benefits:

    • B1: a kernel-native structured logging interface that minimizes complexity and makes for a much more ergonomic interface, enabling structured logging instead of string parsing.
    • B2: doesn’t complicate node logging purely for the purpose of accommodating diverging requirements for kernel. For example, I suspect projects like #30342 should be much easier with layered kernel logging.
    • B3: reduces friction if/when node migrates to use the public bitcoinkernel interface by having already completely decoupled logging.
    • … (what have I missed?)

    Downsides:

    • D1: levels-based filtering has some performance overhead by formatting all (kernel) debug log strings instead of just the subset of categories specified. Note that this only applies when debug is enabled, and that kernel has a lot less logging than node.
    • … (what have I missed?)

    Scope

    I have implemented this approach in https://github.com/bitcoin/bitcoin/compare/master...stickies-v:bitcoin:2025-12/kernel-logging-layering. Most of the changes are quite straightforward, including refactoring BCLog::Logger to BCLog::Sink, adding tests, and adding a util::log::Logger interface.

    Appendix

    Interfaces

    util::log::Logger

     0class Logger
     1{
     2public:
     3    using Callback = std::function<void(const Entry&)>;
     4    using CallbackHandle = std::list<Callback>::iterator;
     5
     6    //! Register a callback to receive log entries. Returns a handle for unregistration.
     7    [[nodiscard]] CallbackHandle RegisterCallback(Callback callback);
     8
     9    //! Unregister a previously registered callback.
    10    void UnregisterCallback(CallbackHandle handle);
    11
    12    //! Set the minimum log level. Messages below this level are discarded.
    13    void SetMinLevel(Level level);
    14
    15    //! Get the current minimum log level.
    16    Level GetMinLevel() const;
    17
    18    //! Returns true if a message at the given level would be logged.
    19    bool WillLog(Level level) const;
    20
    21    //! Returns true if any callbacks are registered.
    22    bool Enabled() const;
    23
    24    //! Format message and dispatch to all registered callbacks. No-op if minimum log level not met.
    25    template <typename... Args>
    26    void Log(Level level, uint64_t category, std::source_location loc, bool should_ratelimit,
    27             util::ConstevalFormatString<sizeof...(Args)> fmt, const Args&... args);
    28};
    

    bitcoinkernel C logging interface

     0typedef void (*btck_LogCallback)(void* user_data, const btck_LogEntry* entry);
     1
     2struct btck_LogEntry {
     3    const char* message;       //!< Log message
     4    size_t message_len;        //!< Log message length
     5    const char* file_name;     //!< Source file name
     6    size_t file_name_len;      //!< Source file name length
     7    const char* function_name; //!< Source function name
     8    size_t function_name_len;  //!< Source function name length
     9    const char* thread_name;   //!< Thread name
    10    size_t thread_name_len;    //!< Thread name length
    11    int64_t timestamp_ns;      //!< Timestamp in nanoseconds since epoch
    12    uint32_t line;             //!< Source line number
    13    btck_LogLevel level;       //!< Log level
    14    btck_LogCategory category; //!< Log category
    15};
    16
    17BITCOINKERNEL_API void btck_logging_set_min_level(btck_LogLevel level);
    18
    19BITCOINKERNEL_API btck_LoggingConnection* BITCOINKERNEL_WARN_UNUSED_RESULT btck_logging_connection_create(
    20    btck_LogCallback log_callback,
    21    void* user_data,
    22    btck_DestroyCallback user_data_destroy_callback) BITCOINKERNEL_ARG_NONNULL(1);
    23
    24BITCOINKERNEL_API void btck_logging_connection_destroy(btck_LoggingConnection* logging_connection);
    
  2. ryanofsky commented at 4:24 pm on December 12, 2025: contributor

    Nice work. The new interface proposed here and implemented in your branch seem much better than the current interface, and well thought out. I feel like we could pretty easily wire it into the existing Logger class without introducing a new class and new macros though, and without needing to change validation & util code.

    I wonder if there may be difficulties doing that I’m not aware of, or if it just seemed easier to start with a clean state in the current branch.

    I do think it would preferable to have one Logger class and one set of macros instead of two. I also think using kernel logging functions in util code (only src/util/fs_helpers.cpp right now but it seems like usage might expand) would probably not be desirable, since the dependency is supposed to to the other way with kernel depending on util, not util depending on kernel. Also, the wallet and GUI are supposed to use util but should not depend on the kernel. This could be another reason to want to try improving the Logger class instead of replacing it.

  3. stickies-v commented at 12:47 pm on December 15, 2025: contributor

    Thanks for looking into this, @ryanofsky! Upgrading vs separating is indeed the important decision to make here. I initially started wiring it into the existing Logger class, but changed my approach when I realized that when kernel becomes its own project, they will each need to have their own code anyway. Complicating node logging to accommodate kernel’s different requirements, to then take it away again, feels like unnecessary friction and code complexity (e.g. plumbing a level-based system on top of a cateogory-level-based system). On the other hand, prematurely optimizing for a future split that may never happen is a risk. (note: I don’t think we’re prematurely optimizing, because the kernel logger is so light-weight)

    A different approach that I’m happy to explore, and I think could make more sense than wiring low-level kernel into higher-level node logging, is to ~move the kernel::Logger to util::Logger, and have all logging (instead of just kernel) go through these structured callbacks. This way, the low-level logging interface is in util, and all the opinionated node stuff (i/o, ratelimiting, log formatting, …) would live in common. This would mean node logging fundamentally also becomes level-based (i.e. debug string formatting is all-or-nothing), but I think that’s a pretty small price to pay?

    and new macros though, and without needing to change validation

    In the mvp branch, I chose new KernelLog... macros just to keep things straightforward, and I’d be open to using macro/compile flag magic to minimize the diff there. However, I think the reality is that logging signatures in kernel code will have to change (for every single callsite) to make it context-aware, so I don’t think a new macro name is all that important?

    I also think using kernel logging functions in util code…

    I agree with most of your observations in this paragraph, and kernel macros should not be leaking into util, wallet, GUI. I think the proper solution here is to remove logging from util though. In my view, low-level (i.e. util) code should not be logging at all. I’ve implemented one way of doing that in https://github.com/bitcoin/bitcoin/compare/master...stickies-v:bitcoin:2025-12/util-no-logging. It seems the scope of that change is rather limited, but there are many approaches possible, so that would warrant its own discussion.

  4. sedited commented at 11:40 am on December 16, 2025: contributor

    Thanks for taking charge of this! The C interface you laid out looks really nice too and should be easier to wire against whatever is required.

    hey will each need to have their own code anyway

    One point that might be nice to mention here too is that we are also defining a bunch of categories that the kernel should not have to know about.

    Reading through your comment, I actually like what you are laying out as an alternative. Maybe a refinement of that idea could be to split the logger into a low-level and high-level implementation. Both can be implemented as global instances for now. The low-level class could take care of formatting, level detection, and providing a unified macro for all users. It can then feed into a higher level class through a callback. For a global logger, this would mean we’d retain a global low level logger, and introduce a global higher-level instance. We only initialize the global callback into the higher level logger when it is actually required by the respective binary, similar to what we do currently with the translations. For the kernel library itself, we can then surface this callback to the user through the API with the interface you are suggesting here.

    This would mean node logging fundamentally also becomes level-based (i.e. debug string formatting is all-or-nothing), but I think that’s a pretty small price to pay?

    I’m not really following why we can’t have a set of categories defined to filter against pre formatting. Are you suggesting this, because we’d ideally want to split off non-kernel categories?

  5. stickies-v commented at 12:25 pm on December 16, 2025: contributor

    The low-level class could take care of formatting, level detection, and providing a unified macro for all users. It can then feed into a higher level class through a callback.

    That sounds like the new design I’m currently working on. So far, it seems like it’ll be a much bigger code diff than the seperation branch (because a lot of logging.{h,cpp} code will need to move around, but there should be 1) less duplication and 2) more contained logging logic, so I think overall this might work out nicely.

    I’m not really following why we can’t have a set of categories defined to filter against pre formatting. Are you suggesting this, because we’d ideally want to split off non-kernel categories?

    Category filtering to me feels like a high-level (i/o) logging output concern rather than a low-level logging concern. We could make the low-level interface filter based on categories, but that I think would needlessly clutter the interface for users that don’t need it (incl kernel), so I’d much rather keep it purely levels-based. So for e.g. node, that would mean that if only a single debug category is enabled, all debug logs would be formatted*, but only the one enabled category printed. Unless measurements indicate it really is too much overhead, I think that’s not going to be a big deal when you’re in debug mode already?

    *note: there are 2 formatting steps: 1 the LogInfo(fmt, args...) formatting, and then 2) the formatting where we produce a single string containing timestamp, level, msg, … I’m generally talking about the first step. The second is of course only done when needed.

  6. ?
    added_to_project_v2 janb84
  7. ?
    project_v2_item_status_changed janb84
  8. stickies-v commented at 9:08 pm on December 17, 2025: contributor

    I have implemented a new approach (“Layering”) in https://github.com/bitcoin/bitcoin/compare/master...stickies-v:bitcoin:2025-12/kernel-logging-layering. The Layering approach turned out much less invasive than my initial attempts showed, and now has my preference.

    In a nutshell:

    • we distinguish between a (low-level) logger that produces log statements (util::log::Logger) and (higher-level) log sinks (BCLog::Sink, btck_LoggingConnection) that consume the logs by registering a callback on the logger.
    • the bitcoinkernel.h API is ~identical between both versions, as are the util::log::Logger and kernel::Logger interfaces
    • the total diff is quite small, ~500 lines diff excluding tests, including complete removal of logging.{h,cpp} from kernel

    Conceptually, it’s quite similar to the earlier Separation approach, but just organizes the code differently, reducing code churn and duplication. @ryanofsky I think this addresses your concerns about duplicated macros, about util code depending on kernel code, and about minimizing changes to validation and util code. It still introduces a new class, I don’t immediately see an elegant way around that that also allows kernel to not depend on node logging code. Does that seem like a reasonable enough trade-off to you to warrant opening a PR and discuss the details?

    I’ll open a PR if/when people are happy about the concept and initial approach, that’s probably a better place to discuss code in more depth.

  9. sedited commented at 9:54 pm on December 17, 2025: contributor

    I’ll open a PR if/when people are happy about the concept and initial approach, that’s probably a better place to discuss code in more depth.

    This newest approach looks good to me. I think some people might still have reservations about unconditionally logging all categories. I tend to agree with your opinion on this though. Once debug logging is activated, expectations on string serialization overhead should be the same, no matter the category selected.

  10. ajtowns commented at 4:20 am on December 20, 2025: contributor

    I have implemented this approach in master…stickies-v:bitcoin:2025-12/kernel-logging-separation

    Gross. Changing all the validation logs prefix the macro with Kernel and changing BCLog:: to kernel::Category is not an improvement.

    • D2: levels-based filtering has some performance overhead by formatting all (kernel) debug log strings instead of just the subset of categories specified.

    That pretty much destroys the use case of Trace logging which is intended for high-rate logging that’s probably not performant enough for regular usage. Which is fine today, because we only do trace logging in the wallet. But being able to do it for mempool analysis/debugging would also be useful.

    Likewise for:

    Category filtering to me feels like a high-level (i/o) logging output concern rather than a low-level logging concern.

    That’s not the case.

    As far as the problem statements go:

    • P1: callbacks operate on formatted strings,

    Outside of the kernel, the callbacks are only used in tests, so changing how the callbacks work seems trivial, not justification for a major rework.

    * **P2**: the [filtering interface](https://github.com/bitcoin/bitcoin/blob/b26762bdcb941096ccc6f53c3fb5a7786ad735e7/src/kernel/bitcoinkernel.h#L737) is not really intuitive, requiring users to call combinations of `btck_logging_set_level_category` and `btck_logging_enable_category` when they want to produce `debug` or `trace` logs.
    

    That interface isn’t a good design even for the node software. #34038 removes it from being exposed to users (via the -loglevel config option), so that a followup can simplify it further.

    At that point, having btck_logging_trace(VALIDATION) or btck_logging_setlevel(VALIDATION, TRACE) to enable trace logging for validation at a cost of slower validation, seems fine to me.

    I propose we address problems P1, P2 and P3 by completely separating the kernel logging infrastructure from the node logging infrastructure. Roughly, this would mean:

    1. Implement a minimal `kernel::Logger` interface
    

    This seems exactly backwards to me; I would instead have approached it as “let’s rip out the node specific stuff from logging into node/loghandler, and have it register itself in a similar way to how kernel users will register their log handling code”.

  11. ajtowns commented at 3:08 am on December 21, 2025: contributor

    Outside of the kernel, the callbacks are only used in tests, so changing how the callbacks work seems trivial, not justification for a major rework.

    For example, https://github.com/ajtowns/bitcoin/commit/57e6ba1739dff27b78819fbb9594ec88daff609f

  12. stickies-v commented at 11:07 am on January 5, 2026: contributor

    I have updated the Issue description to reflect the new layering approach that minimizes churn and code duplication, wiring all logging into a low-level util::log::Logger addressing review feedback (1, 2)


    @ajtowns thank you for the review, and apologies for the slow response - I was off for a little while. I wish I’d updated the Issue description earlier, because I think a lot of your concerns were already addressed in the new approach. Apologies for not doing that earlier.

    For example, https://github.com/ajtowns/bitcoin/commit/57e6ba1739dff27b78819fbb9594ec88daff609f

    Structured callbacks can be done with a much smaller diff indeed, but I think it doesn’t address two concerns:

    1. a lot of the node logging logic is irrelevant for kernel (e.g. non-kernel categories, ratelimiting, string formatting, …), so keeping node logging logic separate would be good
    2. it limits how much we can simplify the bitcoinkernel logging interface, forcing the user to do category configuration

    This seems exactly backwards to me; I would instead have approached it as “let’s rip out the node specific stuff from logging into node/loghandler, and have it register itself in a similar way to how kernel users will register their log handling code”.

    Agreed, see an earlier update (which is now the documented approach in this Issue).

    That pretty much destroys the use case of Trace logging which is intended for high-rate logging that’s probably not performant enough for regular usage. Which is fine today, because we only do trace logging in the wallet. But being able to do it for mempool analysis/debugging would also be useful.

    My understanding of your concern is: “with level-only filtering at the low (util::log::Logger) level, enabling Trace means paying formatting overhead for all trace logs regardless of category, making high-frequency trace logging impractical.” Is my understanding correct? I think we’ll have to add a LOT more Trace logging before that actually becomes a concern. I’m not sure we ever will, but if/when we ever need it, I think that can still be easily addressed later by adding a category filter predicate to util::log::Logger? Optimizing for that now seems premature to me? (Just in case there’s any confusion, my suggested approach still uses levels-based filtering to avoid unnecessary log string formatting.)


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-01-08 18:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me