multiprocess: Add libmultiprocess git subtree #31741

pull ryanofsky wants to merge 13 commits into bitcoin:master from ryanofsky:pr/subtree changing 79 files +6043 −64
  1. ryanofsky commented at 4:48 pm on January 27, 2025: contributor

    This adds the libmultiprocess library and code generator as a subtree in src/ipc/libmultiprocess and allows it to be built with the cmake -DENABLE_IPC option, which is disabled by default.

    This PR does not entirely remove the depends system libmultiprocess package because the package is useful when cross compiling. (A cross-compiling cmake build cannot easily build and run a native code generation tool.) However, it does update the depends package to build from the new git subtree, instead of being downloaded separately from github, so the same sources are used to build both the runtime library and the code generator.

    This PR includes the following manual changes (not created automatically with git subtree add) which just update the build system and documentation:


    Previous minisketch subtree PR #23114 may be useful for comparison

    Instructions for subtree verification can be found:

    TL&DR:

    0git remote add --fetch libmultiprocess https://github.com/chaincodelabs/libmultiprocess.git
    1test/lint/git-subtree-check.sh -r src/ipc/libmultiprocess
    

    This PR is part of the process separation project.

  2. DrahtBot commented at 4:48 pm on January 27, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31741.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept NACK luke-jr
    Concept ACK TheCharlatan, Sjors, BrandonOdiwuor, vasild, pablomartin4btc
    Approach ACK hebasto

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31982 (scripted-diff: rename libmultiprocess repository by fanquake)
    • #31161 (cmake: Set top-level target output locations by hebasto)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. ryanofsky marked this as a draft on Jan 27, 2025
  4. ryanofsky force-pushed on Jan 27, 2025
  5. DrahtBot commented at 5:01 pm on January 27, 2025: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/36237440607

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  6. DrahtBot added the label CI failed on Jan 27, 2025
  7. ryanofsky force-pushed on Jan 27, 2025
  8. ryanofsky commented at 7:01 pm on January 27, 2025: contributor
    Rebased eb70c90b06f068816ba09481f1cf37f49c4d95ee -> 9bb4bb9680b28b238ae5840f9de2963a81e01119 (pr/subtree.1 -> pr/subtree.2, compare) on updated base PR to fix CI errors Updated 9bb4bb9680b28b238ae5840f9de2963a81e01119 -> 93ba80b3d78fa1c4e7f614df6517b9cb492d05d7 (pr/subtree.2 -> pr/subtree.3, compare) to fix path typo in previous push
  9. ryanofsky force-pushed on Jan 27, 2025
  10. TheCharlatan commented at 7:33 pm on January 27, 2025: contributor
    Concept ACK
  11. DrahtBot removed the label CI failed on Jan 27, 2025
  12. theuni commented at 1:02 am on January 28, 2025: member

    @ryanofsky Please have a look here: https://github.com/theuni/bitcoin/commit/f2d74475433528c81c42af3232c257c443c1fc94

    It’s a quick hack to demonstrate the concept, please ignore the rough edges and consider the concept alone. I’ll clean it up if you’re interested.

    That approach would simplify things for (I suspect) 99% of builders, who never need to regenerate the files. Let me know what you think?

    Edit: Note that if you’re using depends, you’ll need to overwrite depends/$(host)/native/lib/cmake/Libmultiprocess/CopyOrGenerate.cmake with src/ipc/libmultiprocess/cmake/TargetCapnpSources.cmake. If we take this approach, that file would be updated upstream accordingly.

  13. in doc/multiprocess.md:32 in b8289e77aa outdated
    26@@ -25,9 +27,9 @@ build/src/bitcoin-node -regtest -printtoconsole -debug=ipc
    27 BITCOIND=$(pwd)/build/src/bitcoin-node build/test/functional/test_runner.py
    28 ```
    29 
    30-The `cmake` build will pick up settings and library locations from the depends directory, so there is no need to pass `-DWITH_MULTIPROCESS=ON` as a separate flag when using the depends system (it's controlled by the `MULTIPROCESS=1` option).
    31+The `cmake` build will pick up settings and library locations from the depends directory, so there is no need to pass `-DBUILD_MULTIPROCESS=ON` as a separate flag when using the depends system (it's controlled by the `MULTIPROCESS=1` option).
    32 
    33-Alternately, you can install [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) packages on your system, and just run `cmake -B build -DWITH_MULTIPROCESS=ON` without using the depends system. The `cmake` build will be able to locate the installed packages via [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/). See [Installation](https://github.com/chaincodelabs/libmultiprocess/blob/master/doc/install.md) section of the libmultiprocess readme for install steps. See [build-unix.md](build-unix.md) and [build-osx.md](build-osx.md) for information about installing dependencies in general.
    34+When not using depends, the [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) library can optionally be installed as an external dependency and enabled with the `-DWITH_LIBMULTIPROCESS=ON` option. If this is disabled, a local git submodule will be used instead. See [Installation](https://github.com/chaincodelabs/libmultiprocess/blob/master/doc/install.md) section of the libmultiprocess documentation for install steps. Installing libmultiprocess externally is only necessary when cross-compiling (to provide a code generation binary that can run on the native system). It can also be useful when developing changes meant to be submitted upstream.
    


    Sjors commented at 9:26 am on January 28, 2025:

    b8289e77aaa2478828caf7c91a59fc15c96fb187: “local git subtree”

    I would first explain the default behavior, and then point out that -DWITH_LIBMULTIPROCESS=ON can be used to use a locally installed library instead.


    ryanofsky commented at 1:54 pm on January 29, 2025:

    re: #31741 (review)

    Thanks I reworked this and adding headings to separate the cases. I think base case here is that capnproto is the only external dependency and you install it with package manager on your system. A second comon option may be to use depends, so that is described second. The third option of installing libmultiprocess externally is last because I was thinking that might be less commonly done, but I’m not sure.

  14. Sjors commented at 9:33 am on January 28, 2025: member

    Concept ACK

    It would be good to remind people in the PR description how one verifies a subtree: https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#subtrees (you should add ipc/multiprocess to the list there) and https://github.com/bitcoin/bitcoin/tree/master/test/lint#git-subtree-checksh

    TL&DR:

    0git remote add --fetch libmultiprocess https://github.com/chaincodelabs/libmultiprocess.git
    1test/lint/git-subtree-check.sh -r src/ipc/libmultiprocess
    

    It would be good to have an additional CI machine cover multiprocess, using the non-depends -DBUILD_MULTIPROCESS=1 and with BITCOIND=bitcoin-node. For easy rebasing in #30975, ci/test/00_setup_env_mac_native.sh would have my preference. @theuni I’m all for speeding up the build, though that should probably be a followup.

    On (M4) macOS 15.2 I get a flood of warnings, which means I can’t use -DWERROR=ON:

     0In file included from /opt/homebrew/include/capnp/common.h:31:
     1/opt/homebrew/include/kj/windows-sanity.h:39:6: warning: '_WIN32' is not defined, evaluates to 0 [-Wundef]
     2   39 | #if !_WIN32 && !__CYGWIN__
     3      |      ^
     4/opt/homebrew/include/kj/windows-sanity.h:39:17: warning: '__CYGWIN__' is not defined, evaluates to 0 [-Wundef]
     5   39 | #if !_WIN32 && !__CYGWIN__
     6      |     
     7
     8/opt/homebrew/include/capnp/common.h:33:5: warning: 'CAPNP_DEBUG_TYPES' is not defined, evaluates to 0 [-Wundef]
     9
    10/opt/homebrew/include/capnp/raw-schema.h:26:5: warning: '_MSC_VER' is not defined, evaluates to 0 [-Wundef]
    11
    12/opt/homebrew/include/kj/async-prelude.h:34:6: warning: 'KJ_NO_EXCEPTIONS' is not defined, evaluates to 0 [-Wundef]
    

    Building MULTIPROCESS=1 with depends now fails:

     0...
     1Postprocessing native_capnp...
     2Caching native_capnp...
     3Fetching from
     4curl: (3) URL rejected: No host part in the URL
     5Fetching from https://bitcoincore.org/depends-sources
     6  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
     7                                 Dload  Upload   Total   Spent    Left  Speed
     8  0  308k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
     9curl: (22) The requested URL returned error: 403
    10make: *** [/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 22
    

    Before commit 01452f8c0da58db549bc46d0cfa7de715344efc4 depends does build (as you would expect).

  15. vasild commented at 11:21 am on January 28, 2025: contributor
    Concept ACK
  16. ryanofsky commented at 4:19 pm on January 28, 2025: contributor

    re: #31741#pullrequestreview-2577588200

    Thanks @Sjors for the review. I’ll make changes you suggested and I opened two issues https://github.com/chaincodelabs/libmultiprocess/issues/138 and https://github.com/chaincodelabs/libmultiprocess/issues/139 to debug the problems you are seeing, and I will post next steps there. Thanks for reporting these.

  17. ryanofsky commented at 5:22 pm on January 28, 2025: contributor

    @ryanofsky Please have a look here: theuni@f2d7447

    It’s a quick hack to demonstrate the concept, please ignore the rough edges and consider the concept alone. I’ll clean it up if you’re interested.

    I feel like I’m missing a lot of context which led up to that implemention, and also I wonder if files are missing from that commit because just shows a new CopyOrGenerate.cmake script that does not actually seem to be called anywhere.

    From what I can understand though, this script seems to be acting like a wrapper around the mpgen executable that avoids calling mpgen if cached outputs are present. If they are present they will be copied from one location to another (unclear what source location is since it is not specified in the script).

    So I presume the idea behind this is that in cross-compiled builds, it will not be necessarily to have an mpgen utility built, because the mpgen outputs can be checked in to git, or downloaded from some external location, and they can be copied into the build instead of calling mpgen. And this will allow deleting the native_capnp.mk and libmultiprocess.mk packages, which I’m speculating is your ultimate goal? It is unclear what your goal is.

    I don’t have a problem with that approach, if that’s the approach you think is better. I would point out that it will have drawbacks. For one thing if the plan is to check these files into git then we will need some kind of CI rule to ensure that they are generated correctly and not just trust authors of PRs to generate them. The generated files will also be tied to a individual releases of capnproto so everybody will have to run the same capnproto version. Capnproto does not allow mixing headers generated by one version with the library of another version, and it has an explicit preprocessor check to prevent this. The generated files could also make PR review unwieldy, because a simple change to a capnp file could lead to much bigger changes in generated files (which contain binary data in C arrays and are not very friendly to diffing. Changes to generated files could be especially make PRs hard to review if they change .capnp files in multiple commits, and lead to headaches rebasing.

    If you want to go down this road it seems ok, but I think it would be very helpful if you could a little time to actually review this PR. I am happy with this PR currently and think the approach it takes is elegant. It leverages fantastic work @hebasto did in https://github.com/chaincodelabs/libmultiprocess/pull/96 and later PRs to simultaneously support cmake packaging while also simplifying the libmultiprocess build.

    There are two nontrivial commits in this PR. 01452f8c0da58db549bc46d0cfa7de715344efc4 adds an obvious feature to the depends system, that I’m surprised wasn’t in there from the beginning: the ability to build sources from a local directory instead of requiring them to be published to an http server. The feature is simple, ergonomic, clearly documented, and implemented in literally 12 lines of code. The second commit d70474da4d1f3bc623e3167dab63a7860f99875f is nontrivial but also pretty straightforward. It adds a add_subdirectory(libmultiprocess) call and updates a handful of other things in the cmake build to support that call.

    If I have one takeaway from implementing this PR, it is that find_package is great and add_subdirectory is terrible. add_subdirectoryis bad because it leaks all kinds of information you don’t want to leak from the parent directory to the child directory (include paths and other options) and vice versa (child targets leak into the parent build). find_package has standard scoping mechanisms and ways of passing information, while add_subdirectory is ad-hoc and requires PARENT_SCOPE assignments. So in terms of robustness and dealing with breakages in the future, I think keeping depends native_capnp.mk and libmultiprocess.mk packages, using find_package, and avoiding add_subdirectory when making releases and cross compiling will make debugging and support more straightforward. Then with find_package support working, additionally supporting add_subdirectory for local development is pretty easy and just requires a few extra lines of code (add_subdirectory call and some set PARENT_SCOPE lines).

  18. theuni commented at 7:04 pm on January 28, 2025: member

    @ryanofsky Please have a look here: theuni@f2d7447 It’s a quick hack to demonstrate the concept, please ignore the rough edges and consider the concept alone. I’ll clean it up if you’re interested.

    I feel like I’m missing a lot of context which led up to that implemention, and also I wonder if files are missing from that commit because just shows a new CopyOrGenerate.cmake script that does not actually seem to be called anywhere.

    Yes, sorry, I missed the chunk that actually turns it on. Pushed again, see https://github.com/theuni/bitcoin/commit/274cc5598db09af687e7383979b377460a59ef94.

    From what I can understand though, this script seems to be acting like a wrapper around the mpgen executable that avoids calling mpgen if cached outputs are present. If they are present they will be copied from one location to another (unclear what source location is since it is not specified in the script).

    Correct

    So I presume the idea behind this is that in cross-compiled builds, it will not be necessarily to have an mpgen utility built, because the mpgen outputs can be checked in to git, or downloaded from some external location, and they can be copied into the build instead of calling mpgen. And this will allow deleting the native_capnp.mk and libmultiprocess.mk packages, which I’m speculating is your ultimate goal? It is unclear what your goal is.

    Correct. Even if it’s not desirable to check these into master, it would be very useful to have them in release branches so that they end up in release tarballs. It’s very common for source packages to be pre-bootstrapped, and that’s exactly what this would do.

    Consider how much simpler this would be for builders/packagers (who don’t use depends) to build cross packages.

    I don’t have a problem with that approach, if that’s the approach you think is better. I would point out that it will have drawbacks. For one thing if the plan is to check these files into git then we will need some kind of CI rule to ensure that they are generated correctly and not just trust authors of PRs to generate them.

    Indeed.

    The generated files will also be tied to a individual releases of capnproto so everybody will have to run the same capnproto version. Capnproto does not allow mixing headers generated by one version with the library of another version, and it has an explicit preprocessor check to prevent this.

    I think we would want to enforce this anyway? One of the rules we decided on for CMake is “newer build-side tools should never lead to different/better code”. For example, A newer version of CMake should produce the same code as an older one. If newer versions of capnproto produce materially different code, I think we’d want force a pinning (for everyone) to whatever we’re using in guix, no?

    The generated files could also make PR review unwieldy, because a simple change to a capnp file could lead to much bigger changes in generated files (which contain binary data in C arrays and are not very friendly to diffing. Changes to generated files could be especially make PRs hard to review if they change .capnp files in multiple commits, and lead to headaches rebasing.

    Agreed, that does sound like it could be painful.

    If you want to go down this road it seems ok, but I think it would be very helpful if you could a little time to actually review this PR. I am happy with this PR currently and think the approach it takes is elegant. It leverages fantastic work @hebasto did in chaincodelabs/libmultiprocess#96 and later PRs to simultaneously support cmake packaging while also simplifying the libmultiprocess build.

    This is my process for review :) Please don’t think that just because I’m proposing a procedural change, that I’m somehow ignoring the work that has gone in. It also doesn’t mean that I plan on being pushy about this approach. This is a significant change for us and I want to make sure that I understand it and its drawbacks. Maybe the pre-bootstrapped-files idea is terrible, but I wanted to see what it would take to make it happen.

    To my knowledge, we’ve never (as part of our release builds) required any native tooling (beyond the toolchain) for pre-processing of our code. [edit: Er.. ok, that’s not true. QT requires pre-processing, but its native tools are widely available]. Adding to that, CMake doesn’t understand the concept of build/host split tools. So for something that is going to be a hard requirement for future builds, I’m doing my due diligence to understand how we can make this as painless for devs/packagers as possible.

    It’s easiest for me to review something like this by basically implementing a solution myself and seeing where we diverge. So far, I’m mostly arriving at the same conclusions as you. I wouldn’t have come up with the depends-source-tarball thing though. That one’s clever :)

    There are two nontrivial commits in this PR. 01452f8 adds an obvious feature to the depends system, that I’m surprised wasn’t in there from the beginning: the ability to build sources from a local directory instead of requiring them to be published to an http server. The feature is simple, ergonomic, clearly documented, and implemented in literally 12 lines of code. The second commit d70474d is nontrivial but also pretty straightforward. It adds a add_subdirectory(libmultiprocess) call and updates a handful of other things in the cmake build to support that call.

    I think the depends source-package thing is elegant and neat. But in the cross-compiled case, it effectively nullifies the effect of vendoring the code. Changes to the local libmultiprocess won’t be picked up without rebuilding depends. I think this could be improved by always using our internal target libmultiprocess, but selectively with an internel/external native one? In that case, I don’t think the non-native libmultiprocess in depends needs to exist? Maybe that’s nonsense… I’m still trying to understand how it all glues together, I’m nowhere near done yet. Again, hacking around is how I understand/review, please don’t take it as a criticism.

    If I have one takeaway from implementing this PR, it is that find_package is great and add_subdirectory is terrible. add_subdirectoryis bad because it leaks all kinds of information you don’t want to leak from the parent directory to the child directory (include paths and other options) and vice versa (child targets leak into the parent build). find_package has standard scoping mechanisms and ways of passing information, while add_subdirectory is ad-hoc and requires PARENT_SCOPE assignments. So in terms of robustness and dealing with breakages in the future, I think keeping depends native_capnp.mk and libmultiprocess.mk packages, using find_package, and avoiding add_subdirectory when making releases and cross compiling will make debugging and support more straightforward.

    This negates many of the benefits of vendoring, though. Namely hardening/optimizations. A libmultiprocess built in depends wouldn’t get the same build flags as it would if it were building from our project. So yes, while the leakiness of the subdirectory approach is annoying (see leveldb/crc32c for example, we completely ignore their CMake files), it also has its benefits.

  19. theuni commented at 7:13 pm on January 28, 2025: member

    I think the depends source-package thing is elegant and neat. But in the cross-compiled case, it effectively nullifies the effect of vendoring the code. Changes to the local libmultiprocess won’t be picked up without rebuilding depends. I think this could be improved by always using our internal target libmultiprocess, but selectively with an internel/external native one? In that case, I don’t think the non-native libmultiprocess in depends needs to exist?

    Basically (unless I’m missing something?) I think libmultiprocess needs an EXTERNAL_MPGEN option, same as capnproto has: https://github.com/capnproto/capnproto/blob/v2/c%2B%2B/CMakeLists.txt#L29. That would allow what I proposed above to work.

    I’m working on a POC for that now.

    Edit: wait, is mpgen not required to build libmultiprocess itself? See, this is a helpful exercise :)

  20. ryanofsky commented at 8:40 pm on January 28, 2025: contributor

    Thanks for the feedback and clearing things up. Will be interested to see the EXTERNAL_MPGEN option and maybe eliminate one of the depends packages.

    I think the depends source-package thing is elegant and neat. But in the cross-compiled case, it effectively nullifies the effect of vendoring the code. Changes to the local libmultiprocess won’t be picked up without rebuilding depends.

    I don’t think “effectively nullifies the effect of vendoring” is true at all, unless you have some different goals in mind that I’m not aware of. My main goal here is to make it trivial for developers to build multiprocess binaries by running “apt install libcapnp-dev capnprotoand-DBUILD_MULTIPROCESS=ON` and not having to do anything else. (At some point in a followup I would also like to change default to ON so even the latter is not necessary.) I also want the depends system to use the git subtree, so I am tweaking the depends system minimally to do that.

    The use-case you seem to be referencing is a more particular one where someone is using the depends system and also modifying the libmultiprocess library. The PR helps a lot with that use-case, and supports it by only requiring the developer to touch libmultiprocess directory and rerun their depends make command after changes to libmultiprocess code. This is much easier than what you have to do now, which is commit the changes to git, push them to github, update the version_hash and sha256_hash in the package file, and then run depends again.

    This build steps in this use-case are also basically the same steps I’ve been using to develop libmultiprocess up to this point for years now, where after I make a change, I run “make install” in the libmultiprocess build and then run “make” in the bitcoin build. I can tell you from experience it is not too onerous.

    I think this could be improved by always using our internal target libmultiprocess, but selectively with an internel/external native one?

    Yes we could support building with an internal libmultiprocess for the library an external libmultiprocess for the code generator. This is your EXTERNAL_MPGEN suggestion. It would just require defining a new option and adding a few if() statements to the build. It could help avoid the need to rerun depends in the use-case above if only the library not code generator is being modified. It would also allow bitcoin flags to automatically propagate to the libmultiprocess library.

    In that case, I don’t think the non-native libmultiprocess in depends needs to exist?

    That’s right. Overall EXTERNAL_MPGEN or an equivalent is a reasonable idea that may be worth doing as a followup. The implementation would be be a superset of this change, not replace or eliminate changes already here. One drawback of using this option and eliminating the cross-compiled libmultiprocess.mk package would be that the library and code generator would no longer be built together in lockstep. So you could make changes to the code generator and they might not be picked up while changes to the library are, which couldn’t happen with the current PR. I don’t think it is a big concern though.

    Maybe that’s nonsense… I’m still trying to understand how it all glues together, I’m nowhere near done yet. Again, hacking around is how I understand/review, please don’t take it as a criticism.

    No problem at all. An EXTERNAL_MPGEN is potentially useful idea I didn’t really think about. Just more of an add-on than a replacement for the changes here.

    This negates many of the benefits of vendoring, though. Namely hardening/optimizations. A libmultiprocess built in depends wouldn’t get the same build flags as it would if it were building from our project. So yes, while the leakiness of the subdirectory approach is annoying (see leveldb/crc32c for example, we completely ignore their CMake files), it also has its benefits.

    I can see this both ways. My inclination was that if you want flags to apply to a library you should just pass them to the library because, this broke the libmultiprocess build earlier and forced me to add a workaround. But I could imagine other cases where leaking flags are more helpful.

  21. ryanofsky commented at 9:27 pm on January 28, 2025: contributor

    Few other things I wanted to reply to.

    re: #31741 (comment)

    Correct. Even if it’s not desirable to check these into master, it would be very useful to have them in release branches so that they end up in release tarballs. It’s very common for source packages to be pre-bootstrapped, and that’s exactly what this would do.

    This is just a gut reaction, but as a long-time gentoo and nixos user, this sounds pretty bad to me. I’d want my package manager to build from human readable source code, not from pregenerated files that require a bespoke process to verify and won’t be verified in practice. Maybe human readable source isn’t enough either, and there are no good solutions, but if you are relying “pre-bootstrapped” files for convenience, it seems like you just asking for another xz backdoor to happen. (I don’t remember the details there but the wikipedia article says “A modified version of build-to-host.m4 was included in the release tar file uploaded on GitHub” which sounds exactly the the type of build cruft I would like my package manager to not trust and not use.)

    Consider how much simpler this would be for builders/packagers (who don’t use depends) to build cross packages.

    I don’t have much experience with cross compiling package managers other than working a little with nix, the depends system, and buildroot a long time ago, but I don’t think having pregenerated headers would simplify any of these systems.

    Generated code from protobufs, flex, bison, and many other tools are commonly used, it is common to not check generated sources into git repositories, which all of these systems support in some fashion. Even in our own depends system, running native packages is not anything special and at the time I started working with it there were actually 7 other native packages besides the two I added.

    I could be wrong on this point. Maybe there really are cross compiled packages that could be simplified by using pregenerated sources. But I’d want to see some actual example of that before accepting this was the case.

    Capnproto does not allow mixing headers generated by one version with the library of another version, and it has an explicit preprocessor check to prevent this.

    I think we would want to enforce this anyway?

    We might be saying different things here. I am saying if we check generated headers into the git development branch and they are used by default, then every revision in our git repository is hardcoded to a particular version of capnproto. You couldn’t for example check out code from 3 months ago while having capnproto 1.1 installed your system, you would have to downgrade to 1.02.

    One of the rules we decided on for CMake is “newer build-side tools should never lead to different/better code”. For example, A newer version of CMake should produce the same code as an older one. If newer versions of capnproto produce materially different code, I think we’d want force a pinning (for everyone) to whatever we’re using in guix, no?

    I can’t think of what benefits that may provide. Capnproto is very backwards compatible, it just doesn’t want you to use old generated headers and new library headers together or vice versa because the static code and generated code work together internally and there is not some public interface between them. If you just build from source, everything works perfectly well, but if you check pregenerated headers into git and try to use them you will have headaches. Maybe the headaches could be worth it. I don’t have the answer to that.

  22. in depends/funcs.mk:46 in 01452f8c0d outdated
    38@@ -39,8 +39,20 @@ define fetch_file
    39       $(call fetch_file_inner,$(1),$(FALLBACK_DOWNLOAD_PATH),$(3),$(4),$(5))))
    40 endef
    41 
    42+define int_get_local_dir
    43+$(eval $(1)_sha256_hash:=$(shell
    44+    if ! [ -f $($(1)_fetched) ] || [ $($(1)_fetched) -ot $($(1)_local_dir) ]; then \
    45+        mkdir -p $(dir $($(1)_fetched)); \
    46+        $(build_TAR) -c --sort=name --mtime=0 --owner=0 --group=0 --numeric-owner -f $($(1)_source) -C $($(1)_local_dir) . && \
    


    Sjors commented at 10:58 am on January 29, 2025:
    01452f8c0da58db549bc46d0cfa7de715344efc4: the Apple version of tar doesn’t have --sort, see https://github.com/chaincodelabs/libmultiprocess/issues/139#issuecomment-2621080394

    ryanofsky commented at 1:56 pm on January 29, 2025:

    re: #31741 (review)

    01452f8: the Apple version of tar doesn’t have --sort, see chaincodelabs/libmultiprocess#139 (comment)

    Thanks, this is dropped

  23. TheCharlatan commented at 12:07 pm on January 29, 2025: contributor
    As an alternative to reaching into depends or committing generated files, could it be feasible to set native compilers in the toolchain file instead for compiling the code generators?
  24. fanquake referenced this in commit ad2f9324c6 on Jan 29, 2025
  25. hebasto commented at 12:33 pm on January 29, 2025: member
    Concept ACK.
  26. ryanofsky force-pushed on Jan 29, 2025
  27. ryanofsky marked this as ready for review on Jan 29, 2025
  28. ryanofsky commented at 2:21 pm on January 29, 2025: contributor

    Rebased 93ba80b3d78fa1c4e7f614df6517b9cb492d05d7 -> ab041937289f4939758c1896f5cb57f556ae0615 (pr/subtree.3 -> pr/subtree.4, compare) after base pr was merged, also updated documentation as suggested, and fixed compatibility with gnu make 3.81 and bsd tar.


    re: #31741#pullrequestreview-2577588200

    It would be good to remind people in the PR description how one verifies a subtree

    Thanks! Added your notes about this.

    It would be good to have an additional CI machine cover multiprocess, using the non-depends -DBUILD_MULTIPROCESS=1 and with BITCOIND=bitcoin-node. For easy rebasing in #30975, ci/test/00_setup_env_mac_native.sh would have my preference.

    I feel like CI change should be a separate PR because there is already a lot to review and discuss here, but let me know if you think this would be important to include.

    The other followup change I hope we can make soon is turning the cmake option on by default

    0-option(BUILD_MULTIPROCESS "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    1+cmake_dependent_option(BUILD_MULTIPROCESS "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables." ON "NOT WIN32" OFF)
    

    re: #31741 (comment)

    As an alternative to reaching into depends or committing generated files, could it be feasible to set native compilers in the toolchain file instead for compiling the code generators?

    It might be technically possible. When I was looking to implement this I read a bunch of things about people using ExternalProject_Add to juggle multiple toolchains in cmake. But it looked very messy and seemed like cmake wasn’t designed to handle multiple toolchains. Luckily, we have system that is great at handling multiple toolchains called depends (and other package managers also support this). So with a simple, clean change to depends to support local directories, cross compiling the subtree is easy and works well.

    With some additional work implementing Cory’s EXTERNAL_MPGEN option, add_subdirectory could be used instead of depends to build the library (not the code generator). This way cmake could be used to control libmultiprocess compile flags even in depends builds, if we want that.

  29. theuni commented at 2:49 pm on January 29, 2025: member

    With some additional work implementing Cory’s EXTERNAL_MPGEN option, add_subdirectory could be used instead of depends to build the library (not the code generator). This way cmake could be used to control libmultiprocess compile flags even in depends builds, if we want that.

    I have this hacked up to work, though I’m still investigating some subdirectory leakage (ugh) that I don’t quite understand. I pretty strongly prefer this approach as it means everyone’s compiling in-tree. I’ll try to get a minimal POC pushed up today.

    As I mentioned above, that means we get all of our hardening/warning flags. My current hacked-up branch is full of warnings that would’ve gone unnoticed otherwise. They appear harmless, but still..

  30. Sjors commented at 3:04 pm on January 29, 2025: member

    I feel like CI change should be a separate PR because there is already a lot to review and discuss here, but let me know if you think this would be important to include.

    I’ll be testing this with #30975 so we’ll get some CI runs regardless.

  31. ryanofsky commented at 3:07 pm on January 29, 2025: contributor
    re: #31741 (comment) @theuni can the change your are working on be a followup? It doesn’t seem like this change will have any effect for developers unless they are using depends for development which I think few do normally. And even then the main effect is that the library is built with cmake flags instead of depends flags. The change would also increase complexity (would only increase not decrease changes in this PR) and would be easier to review and discuss separately.
  32. Sjors commented at 3:26 pm on January 29, 2025: member

    I’m now able to build depends with make MULTIPROCES=1 on macOS without using the gnu versions.

    However when using the result cmake complains about OpenSSL missing:

     0% cmake -B build --toolchain /Users/sjors/dev/bitcoin/depends/aarch64-apple-darwin24.3.0/toolchain.cmake 
     1-- The CXX compiler identification is AppleClang 16.0.0.16000026
     2-- Detecting CXX compiler ABI info
     3-- Detecting CXX compiler ABI info - done
     4-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ - skipped
     5-- Detecting CXX compile features
     6-- Detecting CXX compile features - done
     7CMake Error at /opt/homebrew/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
     8  Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
     9  system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
    10  OPENSSL_INCLUDE_DIR Crypto SSL)
    11Call Stack (most recent call first):
    12  /opt/homebrew/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
    13  /opt/homebrew/share/cmake/Modules/FindOpenSSL.cmake:691 (find_package_handle_standard_args)
    14  /opt/homebrew/share/cmake/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
    15  depends/aarch64-apple-darwin24.3.0/lib/cmake/CapnProto/CapnProtoConfig.cmake:85 (find_dependency)
    16  /opt/homebrew/share/cmake/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
    17  depends/aarch64-apple-darwin24.3.0/lib/cmake/Libmultiprocess/LibmultiprocessConfig.cmake:41 (find_dependency)
    18  CMakeLists.txt:147 (find_package)
    19
    20
    21-- Configuring incomplete, errors occurred!
    

    It does work when I build depends with GNU Make 4.4.1 (via homebrew), so there’s some other subtle detail left to chase…

    Verbose log with Apple GNU Make 3.81: https://gist.githubusercontent.com/Sjors/29733993ac2ef470611911d69f842095/raw/7a8d653562e34f90af0285ec858d028e1f479b1c/gistfile1.txt

    And cmake –trace: https://gist.github.com/Sjors/e9c7e7e45e0bb2a627c9c52fc4a9caf8

  33. BrandonOdiwuor commented at 4:07 pm on January 29, 2025: contributor
    Concept ACK
  34. ryanofsky commented at 4:23 pm on January 29, 2025: contributor

    However when using the result cmake complains about OpenSSL missing:

    Openssh issue you are seeing is puzzling because I don’t know what change here could cause it, and I wonder if you see the same error without this PR? When you are building with depends capnproto is built without ssl support should should not be looking for ssl at all.

    If you are not seeing this issue happen before this PR, but are seeing this issue happen after this PR, I think it would probably most likely be the changes in commit 36db25f0bcaab6af9358faaad9214d5b77a71cb2 and you could try to revert that commit. I don’t think any of the other commits could cause it because they don’t touch the cmake build.

  35. Sjors commented at 4:46 pm on January 29, 2025: member

    wonder if you see the same error without this PR?

    I tried with master @ ad2f9324c619ae608c434ee85ad9ae1b2812877d and indeed got the same error. So it seems unrelated.

  36. Sjors commented at 8:11 am on January 30, 2025: member

    I found several (build) issues while unleashing the CI on these changes: #30975 (comment)

    Let me know which ones you want to address here, and which can be dealt with later.

  37. Sjors commented at 11:50 am on January 30, 2025: member

    If I cherry-pick #31763 on top of this, I’m unable to make a Guix build with multiprocess, at least not with the hosts I tried (x86_64-linux-gnu, aarch64-linux-gnu and arm64-apple-darwin).

     0% DEP_OPTS="MULTIPROCESS=1" HOSTS="x86_64-linux-gnu" contrib/guix/guix-build
     1Checking that we can connect to the guix-daemon...
     2
     3Hint: If this hangs, you may want to try turning your guix-daemon off and on
     4      again.
     5
     6make: Entering directory '/home/sjors/bitcoin/depends'
     7make[1]: Entering directory '/home/sjors/bitcoin/depends'
     8Checksum missing or mismatched for native_libmultiprocess source. Forcing re-download.
     9Fetching from
    10curl: (3) URL rejected: No host part in the URL
    11Fetching from https://bitcoincore.org/depends-sources
    12  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
    13                                 Dload  Upload   Total   Spent    Left  Speed
    14  0  308k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    15curl: (22) The requested URL returned error: 403
    16make[1]: *** [funcs.mk:320: /home/sjors/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 22
    17make[1]: Leaving directory '/home/sjors/bitcoin/depends'
    18make: *** [Makefile:282: download-linux] Error 2
    19make: Leaving directory '/home/sjors/bitcoin/depends'
    

    (it does work on master)

  38. Sjors commented at 4:47 pm on January 31, 2025: member

    #30975 now passes, using a bunch of patches that will make their way here.

    One of those patches fixes the above guix build issue too (no, it doesn’t, will try again later).

  39. theuni commented at 5:50 pm on January 31, 2025: member

    @theuni can the change your are working on be a followup? It doesn’t seem like this change will have any effect for developers unless they are using depends for development which I think few do normally. And even then the main effect is that the library is built with cmake flags instead of depends flags. The change would also increase complexity (would only increase not decrease changes in this PR) and would be easier to review and discuss separately.

    Sorry, I just don’t think this makes sense. The PR here is to include a vendored multiprocess, but it includes a circuitous and unnecessary way of building it for cross/depends that is unnecessary after a few upstream changes. https://github.com/chaincodelabs/libmultiprocess/pull/142 and https://github.com/chaincodelabs/libmultiprocess/pull/140 are enough to enable a much simpler and unified approach for us.

    Devs use depends (I use nothing BUT depends), ci uses depends, and guix uses depends. So I don’t know why we’d work on an approach here that we plan to change significantly later, as opposed to just fixing it up and using the simpler approach to begin with.

  40. Sjors commented at 6:18 pm on January 31, 2025: member

    enable a much simpler and unified approach

    This sounds appealing…

  41. ryanofsky commented at 6:21 pm on January 31, 2025: contributor

    So I don’t know why we’d work on an approach here that we plan to change significantly later, as opposed to just fixing it up and using the simpler approach to begin with.

    I might be misunderstanding but from everything I’ve seen, the change you are implementing is a superset of this change. That is why I am suggesting it could be a followup. No part of this change would be reverted or simplified by the additional work you are doing so it is a natural follow-up.

    I don’t think devs are typically using depends when building locally, but if they are, this PR supports that perfectly well. The only significant differences with your additional change is that flags will be shared between the Bitcoin and libmultiprocess cmake builds, instead of being specified separately for each build. But separate flags are perfectly fine (obviously) because the flags have been separate since the libmultiprocess was originally added to depends in 2019. Again, as I understand it, your change is a superset of my change. It is builds on top of this change. It does not undo things being done in this change.

    Specifically, my changes (f23c3c47ee4db6fb0ee6a4b4378a2734fd95fb03, f23c3c47ee4db6fb0ee6a4b4378a2734fd95fb03, 19eef973cb4ae37d202c9f584b6ed913482aab4a, 36db25f0bcaab6af9358faaad9214d5b77a71cb2, ab041937289f4939758c1896f5cb57f556ae0615) are all pretty small. If your changes are also small, I am happy to add them here when they are ready. Otherwise, I think they should be a followup. Either way they definitely could be a followup.

  42. ryanofsky commented at 6:29 pm on January 31, 2025: contributor

    enable a much simpler and unified approach

    This sounds appealing…

    To be sure, it is good change, and should be adopted. But unless you are using depends locally it will not affect you at all, and even if you are using depends locally it’s just gives some minor quality-of-life improvements, such as you only have to run one make command instead of two if you modify the libmultiprocess subtree locally, and you only have to change flags one place instead of two places if you are changing flags and care whether the flags are apply to libmultiprocess runtime library.

    I don’t these things are worth holding up this PR significantly or making this PR significantly more complicated, but assuming the proposed changes work well and are reasonably small, we should totally add them here.

  43. theuni commented at 6:30 pm on January 31, 2025: member

    I think we do have some miscommunication going on, because yes, I don’t think we’re understanding each other.

    I’m suggesting that the WITH_LIBMULTIPROCESS option can go away entirely. That means that there are no longer 2 ways to build the multiprocess code. Meaning that everyone gets the same build, same flags, etc. No need to pass tsan for tsan builds, for example. It will just work.

    The only significant differences with your additional change is that flags will be shared between the Bitcoin and libmultiprocess cmake builds, instead of being specified separately for each build. But separate flags are perfectly fine (obviously) because the flags have been separate since the libmultiprocess was originally added to depends in 2019.

    This is the point of vendoring the code heh. I don’t want to review what tidy/ci thinks of this until I’m comfortable knowing that we’re all (including CI and guix) building the same way.

    Again, as I understand it, your change is a superset of my change. It is builds on top of this change. It does not undo things being done in this change.

    Again, I’m suggesting that we would get rid of one of the weird build mechanisms here.. never add it in the first place because it’s unnecessary. I don’t see how that could be described as a superset, so I think we must be talking past each other :(

  44. theuni commented at 6:41 pm on January 31, 2025: member

    To be sure, it is good change, and should be adopted. But unless you are using depends locally it will not affect you at all, and even if you are using depends locally it’s just gives some minor quality-of-life improvements, such as you only have to run one make command instead of two if you modify the libmultiprocess subtree locally, and you only have to change flags one place instead of two places if you are changing flags and care whether the flags are apply to libmultiprocess runtime library.

    Among other things, libmultiprocess as-built here is not hardened. When we (the build nerds) asked to have it vendored, we’re asking for it to get the same treatment as the rest of our code. This PR was opened to do the logistical code movement, but it doesn’t demonstrate the benefits of living within our source tree. And further, it gets built differently depending on whether you’re a dev doing local hacking, or ci, or doing a release.

    So I’m sorry, but I don’t want to spend time reviewing that approach. Especially when it’s so easily avoidable. I don’t consider this to be in a state to be merged until those benefits can be demonstrated. I’m really trying to help with that though! :)

    Edit: See https://github.com/chaincodelabs/libmultiprocess/pull/143 as an example of a benefit of building like the rest of our code. In that case it’s dumb and trivial, but we’re likely to find (or guard against) actual issues by building this way across the board.

  45. ryanofsky commented at 7:27 pm on January 31, 2025: contributor

    I’m suggesting that the WITH_LIBMULTIPROCESS option can go away entirely.

    If you look at the PR, you can see that WITH_LIBMULTIPROCESS option adds 5 lines of code and 2 of them are endif(). So I wouldn’t consider that a real coup. Also I don’t think WITH_LIBMULTIPROCESS should be dropped just because depends will not use it, because having it makes it easier to develop and submit changes to the upstream repository, and makes sense to support other packaging systems (see https://github.com/NixOS/nixpkgs/issues/359902) that are more flexible and less monolithic than the depends system.

    This is the point of vendoring the code heh.

    No offense (really), but this strikes me as a crazy thing to say. I didn’t open this PR and spend the last entire week fixing CI bugs because I was concerned setting compiler flags in the Bitcoin cmake build didn’t automatically set the same flags in the libmultiprocess cmake build. Libmultiprocess has been using cmake since 2019, bitcoin just started using cmake last year, so the approach you are favoring literally only became possible a few months ago. I understand sharing compiler flags is a very important thing to you, and I think it is a nice idea, and I am happy to review or accept additional changes in this PR to support it. But libmultiprocess has had separate compiler flags for as long as it existed and since it was added as a depends package in 2019.

    I can’t remember it ever happening to me, but probably, sometime over the last 5 years I did have to set some compiler flags in two builds instead of one. So it’s great I may no longer have to do this. But this is a small quality of life issue in a corner case that will never affect developers unless they are trying to build in an unusual way, and requires a fairly obvious extra step if they are. It is a strange thing to fixate on. But again, I think it’s great to share flags and would like to do that.

    The actual point of vendoring this code as I wrote earlier in #31741 (comment) is:

    My main goal here is to make it trivial for developers to build multiprocess binaries by running “apt install libcapnp-dev capnprotoand-DBUILD_MULTIPROCESS=ON` and not having to do anything else. (At some point in a followup I would also like to change default to ON so even the latter is not necessary.) I also want the depends system to use the git subtree, so I am tweaking the depends system minimally to do that.

    That is why this PR exists. That is why I’ve been working on it the past week.

    re: #31741 (comment)

    When we (the build nerds) asked to have it vendored, we’re asking for it to get the same treatment as the rest of our code.

    The phrasing here is ambiguous, but if you just wanted libmultiprocess to be compiled with the same flags, we could pass the same flags to depends, or specify the same flags in depends. Either of these approaches would be easier than adding a subtree.

    #31741 (comment)

    Again, I’m suggesting that we would get rid of one of the weird build mechanisms here.. never add it in the first place because it’s unnecessary.

    Which one? We are definitely talking past each each other. I think the problem might be (1) your changes might be incomplete, so you think there are parts of this PR that are unnecessary when they are not (2) you haven’t reviewed this PR so you think it is doing unnecessary things it is not doing or (3) which is very possible, that this PR is actually doing something unnecessary and your changes will simplify it.

    In any case, I am looking forward to see your changes, and appreciate all the feedback and improvements you’ve made so far, which have been very helpful.

  46. theuni commented at 8:48 pm on January 31, 2025: member

    Also I don’t think WITH_LIBMULTIPROCESS should be dropped just because depends will not use it, because having it makes it easier to develop and submit changes to the upstream repository, and makes sense to support other packaging systems (see NixOS/nixpkgs#359902) that are more flexible and less monolithic than the depends system.

    Perhaps this is the source of our breakdown. I would consider this an anti-feature. Or AT LEAST, as you’ve been saying, one that can be added on top of the changes here once it’s been discussed. We don’t provide this option for secp or leveldb or crc32 or univalue or minisketch.

    Anyway, I’ve pushed a demo of my changes here: https://github.com/theuni/bitcoin/commits/libmultiprocess-vendor/

    The internal libmultiprocess is now the only the option, and the non-native version has been removed from depends.

    It contains https://github.com/chaincodelabs/libmultiprocess/pull/141, https://github.com/chaincodelabs/libmultiprocess/pull/142, and https://github.com/chaincodelabs/libmultiprocess/pull/143. I understand that 141 probably has a better solution and 142 hasn’t been reviewed or acked yet.

  47. in depends/packages/native_libmultiprocess.mk:5 in 19eef973cb outdated
    0@@ -1,8 +1,5 @@
    1 package=native_libmultiprocess
    2-$(package)_version=07c917f7ca910d66abc6d3873162fc9061704074
    3-$(package)_download_path=https://github.com/chaincodelabs/libmultiprocess/archive
    4-$(package)_file_name=$($(package)_version).tar.gz
    5-$(package)_sha256_hash=ac9db311e3b22aac3c7b7b7b3f6b7fee5cf3043ebb3c3bf412049e8b17166de8
    


    theuni commented at 9:03 pm on January 31, 2025:
    This unfortunately leaves us open to a supply-chain attack as it depends on the contents of the builders’s environment. We’ll need to commit to the hash of a deterministic tarball.

    ryanofsky commented at 9:14 pm on January 31, 2025:
    I am not following. Can you describe the concern?

    ryanofsky commented at 9:43 pm on January 31, 2025:

    I am not following. Can you describe the concern?

    I would think if anything we are more vulnerable to supply-chain attack without this change, because, for example if a few bitcoin core developers conspired and made a version of libmultiprocess with a backdoor and pushed it to github, it wouldn’t be obvious to other developers that the hashes pointed to a backdoor version. It also wouldn’t be obvious to external observers diffing one version of bitcoin to the next to check for introduction of malicious code.

    By contrast, with this change depends system would no longer be downloading a second copy of sources already sit in the local git repository. With this change, in order to introduce a backdoor into libmultiprocess, that backdoor would have be visible in src/ipc/libmultiprocess, show up in normal diffs, and sitting in plain sight.


    theuni commented at 10:26 pm on January 31, 2025:

    Ok, you’re right. I should not have framed this as a potential attack.

    We maintain hard-coded hashes in depends for 2 reasons: To protect against real supply-chain attacks (Someone hijacks github and replaces tarballs with malicious ones) and for determinism (our sources can be described exactly).

    The output of a depends build is locally deterministic based on the build id, which is comprised of the environment, toolchain, and source hashes. After this change, as far as I can tell, that’s no longer true because it also depends on the contents of your hard drive, which may have been modified after checkout.

    In order to maintain the determinism guarantee, we’d need to pin the tarball to an agreed-up hash, same as our other sources.

    So if I’ve been hacking in my local libmultiprocess dir (as I have today), and I do a depends build, it should fail and yell at me that the sources don’t match the agreed-upon hash.


    ryanofsky commented at 6:28 pm on February 3, 2025:

    re: #31741 (review)

    for determinism (our sources can be described exactly).

    The sources are described by the git hash. The tarball could be generated deterministically too, but should be ok to just use default tar options. If during a deterministic build, you put a directory into a intermediate tarball and extract it and compile it, whether the intermediate tarball was deterministic should not effect whether build outputs are deterministic.

    So if I’ve been hacking in my local libmultiprocess dir (as I have today), and I do a depends build, it should fail and yell at me that the sources don’t match the agreed-upon hash.

    Current version now refreshes the tarball in this case. The previous caching behavior was documented and consistent with how URL sources were treated. If a URL contents change after a URL source downloaded, you don’t yelled at in that case either. But refreshing the tarball is more developer-friendly, so it’s good to implement that change.


    vasild commented at 8:16 am on February 5, 2025:

    The output of a depends build is locally deterministic based on the build id, which is comprised of the environment, toolchain, and source hashes. After this change, as far as I can tell, that’s no longer true because it also depends on the contents of your hard drive, which may have been modified after checkout.

    Even without this PR the output of depends build could change based on the contents of the hard drive, no? E.g. you can checkout github/bitcoin/bitcoin, then modify something in src/ and do the depends build.

    Isn’t the hash of an external tarball more or less equivalent in this aspect to the git commit hash? In both cases the sources can be modified after the tarball checksum is verified and extracted or after git clone/checkout.

    So if I’ve been hacking in my local libmultiprocess dir (as I have today), and I do a depends build, it should fail and yell at me that the sources don’t match the agreed-upon hash.

    Doesn’t that apply to any source as well, not only libmultiprocess dir? Should depends build yell if the tree has local mods?


    ryanofsky commented at 6:05 pm on February 7, 2025:

    re: #31741 (review)

    So if I’ve been hacking in my local libmultiprocess dir (as I have today), and I do a depends build, it should fail and yell at me that the sources don’t match the agreed-upon hash.

    Doesn’t that apply to any source as well, not only libmultiprocess dir? Should depends build yell if the tree has local mods?

    The PR has changed since this comment was written, but the idea is for the depends build to detect if any sources affecting the output of the depends build change (including sources of the depends build). So local modifications will cause things to be rebuilt, depending on what was modified. The earlier version of this PR when that comment was written didn’t try very hard to refresh the src/ipc/libmultiprocess tarball if changes were made to the local directory after the tarball was created, and the current version is better about that.

  48. ryanofsky commented at 9:13 pm on January 31, 2025: contributor

    I would consider this an anti-feature. Or AT LEAST, as you’ve been saying, one that can be added on top of the changes here once it’s been discussed.

    Well, if it is an anti-feature, it is a 5 line anti-feature and I am happy to drop it here. If I do this though I will probably make a separate PR to add it later, because I think it will be painful to develop and test upstream libmultiprocess changes without it.

    Anyway, I’ve pushed a demo of my changes here: https://github.com/theuni/bitcoin/commits/libmultiprocess-vendor/

    Thanks, will take a look and incorporate it here.

  49. theuni commented at 9:32 pm on January 31, 2025: member

    @ryanofsky Thanks.

    I think we might just have been disagreeing on the importance of a unified build for everyone for something that will presumably (in the future) become mission-critical. I can see how from your POV it might seems like I’m just being persnickety about flags, but to me it’s about making sure we’re being clear (internally and externally) about ownership/canon/etc… the flags were just illustrative (to me) of the lack of clarity there.

    Thanks for your patience with me on this :)

  50. ryanofsky commented at 9:53 pm on January 31, 2025: contributor

    disagreeing on the importance of a unified build for everyone for something that will presumably (in the future) become mission-critical.

    I really don’t think this is true. I think you have made number a lot of claims that sound great, don’t stand up to scrutiny. I probably have too. These issues can be hard to reason about, so it’s good that we are going through this exercise.

    But I don’t take security or hardening less seriously than anyone else, and I believe this approach has been 100% compatible with both.

    I was balking at the idea that the main reason for adding a subtree would to be add some hardening flags. There are much easier ways to add hardening flags, and much more important benefits to having a subtree.

  51. ryanofsky force-pushed on Feb 3, 2025
  52. ryanofsky commented at 6:48 pm on February 3, 2025: contributor
    Rebased ab041937289f4939758c1896f5cb57f556ae0615 -> 1c10bfc167b0f3c17780b2ae3747999ee56c4e1c (pr/subtree.4 -> pr/subtree.5, compare) incorporating changes from https://github.com/theuni/bitcoin/commits/libmultiprocess-vendor/ and making various build fixes to let multiprocess CI jobs succeed in PR #30975 when libmultiprocess is built as a cmake subdirectory instead of as a cmake module.
  53. DrahtBot added the label CI failed on Feb 3, 2025
  54. DrahtBot commented at 7:51 pm on February 3, 2025: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/36604138479

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  55. ryanofsky force-pushed on Feb 3, 2025
  56. ryanofsky commented at 9:18 pm on February 3, 2025: contributor
    Updated 1c10bfc167b0f3c17780b2ae3747999ee56c4e1c -> e4289ddc3cf361cfc2ec8c5c64b96e57950d17ab (pr/subtree.5 -> pr/subtree.6, compare) to fix ci link error https://cirrus-ci.com/task/6281383380779008 caused by applying core_interface debug flags to multiprocess target but not mptest target depending on it Rebased e4289ddc3cf361cfc2ec8c5c64b96e57950d17ab -> b4d397903d652bae1497140775591f755d857e65 (pr/subtree.6 -> pr/subtree.7, compare) to fix more errors from core_interface
  57. ryanofsky force-pushed on Feb 4, 2025
  58. DrahtBot removed the label CI failed on Feb 4, 2025
  59. Sjors commented at 10:05 am on February 4, 2025: member

    I’m unable to make a Guix build. I’m using a branch sjors/2025/02/ipc-guix which enables multiprocess in the simplest possible way, without modifying the depends system, by setting MULTIPROCESS=1 in the pre-download and depends make step.

    It downloads and builds capnp as expected, but in the build says:

     0make: Entering directory '/bitcoin/depends'
     1Checksum missing or mismatched for native_libmultiprocess source. Forcing re-download.
     2...
     3Extracting native_capnp..
     4...
     5Caching native_capnp...
     6Fetching from
     7/gnu/store/7k364lx15sq567ivd79pp9dmnallcxz1-bash-minimal-5.1.16/bin/sh: line 1: curl: command not found
     8Fetching from https://bitcoincore.org/depends-sources
     9/gnu/store/7k364lx15sq567ivd79pp9dmnallcxz1-bash-minimal-5.1.16/bin/sh: line 1: curl: command not found
    10make: *** [funcs.mk:324: /bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 127
    

    I can add more debugging if you can’t reproduce.

  60. ryanofsky commented at 1:11 pm on February 4, 2025: contributor

    re: #31741 (comment)

    I’m unable to make a Guix build.

    Thanks, created https://github.com/chaincodelabs/libmultiprocess/issues/150 to track this

  61. luke-jr commented at 3:57 am on February 5, 2025: member
    Concept NACK: there’s no rationale for why this can’t always be external.
  62. ryanofsky commented at 4:50 am on February 5, 2025: contributor

    Concept NACK: there’s no rationale for why this can’t always be external.

    FWIW, I agree with you and want this to always be external. I personally find it more convenient to work on the library in its standalone git repository, and think cmake’s find_package mechanism is a cleaner and easier to maintain form of integration than the add_subdirectory trick used in this PR. So this PR intentionally does not remove any ability to use the library externally, and it makes git subtree a pure mirror of the external repository with no internal changes.

    The point of this PR, at least for me, is not to change anything about releases or packaging. It is purely to (1) make IPC features easier for developers to test, so they don’t have to install a local dependency, and to (2) get more eyes on the code, so when changes are merged in they look like normal diffs and not like checksum updates. I think both of these things are real pure benefits and should not have any downsides. Practically speaking this PR just makes it so if developer wants to enable IPC features all they have to do is toggle the ENABLE_IPC cmake option without downloading or installing anything separately.

    The only possible downside of this PR is the last commit which does change the depends release build to integrate libmultiprocess differently than before. I feel like it makes the build more fragile and could be a maintenance burden, but theuni thought it was important to make sure libmultiprocess code was automatically compiled with same flags and options as Bitcoin core code. So last commit also seems reasonable to me, even if it wasn’t my initial preference.

  63. ryanofsky force-pushed on Feb 5, 2025
  64. ryanofsky commented at 5:20 am on February 5, 2025: contributor
    Rebased b4d397903d652bae1497140775591f755d857e65 -> 99b7b857d3b8b9ce9bd7501e2480583369740c55 (pr/subtree.7 -> pr/subtree.8, compare) with some more fixes for downstream PR #30975 needed after switching depends build from find_package to add_subdirectory. Also includes a fix for guix multiprocess build issue https://github.com/chaincodelabs/libmultiprocess/issues/150
  65. vasild commented at 7:29 am on February 5, 2025: contributor
    @luke-jr, some pros of this PR in @ryanofsky’s comment above. What do you think are the cons?
  66. in CMakeLists.txt:151 in 99b7b857d3 outdated
    140@@ -141,8 +141,9 @@ endif()
    141 
    142 cmake_dependent_option(WITH_DBUS "Enable DBus support." ON "CMAKE_SYSTEM_NAME STREQUAL \"Linux\" AND BUILD_GUI" OFF)
    143 
    


    vasild commented at 8:53 am on February 5, 2025:

    pico nit: in the commit message of b75e13e6ecce1b2b406ef83099200df42acf37e0 scripted-diff: s/WITH_MULTIPROCESS/ENABLE_IPC/ in cmake:

    0- a find_package is call is made
    1+ a find_package call is made
    

    ryanofsky commented at 5:19 pm on February 7, 2025:

    re: #31741 (review)

    Thanks, fixed

  67. in CMakeLists.txt:145 in 99b7b857d3 outdated
    140@@ -141,8 +141,9 @@ endif()
    141 
    142 cmake_dependent_option(WITH_DBUS "Enable DBus support." ON "CMAKE_SYSTEM_NAME STREQUAL \"Linux\" AND BUILD_GUI" OFF)
    143 
    144-option(WITH_MULTIPROCESS "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    145-if(WITH_MULTIPROCESS)
    146+option(ENABLE_IPC "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    147+option(WITH_LIBMULTIPROCESS "Build with external libmultiprocess library instead of with local git submodule when ENABLE_IPC is enabled. This is not normally recommended, but can be useful when cross-compiling or making changes to the upstream project." OFF)
    


    vasild commented at 8:56 am on February 5, 2025:
    s/git submodule/git subtree/

    ryanofsky commented at 5:19 pm on February 7, 2025:

    re: #31741 (review)

    Thanks, fixed

  68. in CMakeLists.txt:146 in 99b7b857d3 outdated
    140@@ -141,8 +141,9 @@ endif()
    141 
    142 cmake_dependent_option(WITH_DBUS "Enable DBus support." ON "CMAKE_SYSTEM_NAME STREQUAL \"Linux\" AND BUILD_GUI" OFF)
    143 
    144-option(WITH_MULTIPROCESS "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    145-if(WITH_MULTIPROCESS)
    146+option(ENABLE_IPC "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    147+option(WITH_LIBMULTIPROCESS "Build with external libmultiprocess library instead of with local git submodule when ENABLE_IPC is enabled. This is not normally recommended, but can be useful when cross-compiling or making changes to the upstream project." OFF)
    148+if(WITH_LIBMULTIPROCESS)
    


    vasild commented at 8:56 am on February 5, 2025:
    I find it could be confusing to have the same option WITH_LIBMULTIPROCESS before and after this PR with different semantics. Maybe WITH_EXTERNAL_LIBMULTIPROCESS is a better name for the new one?

    vasild commented at 8:58 am on February 5, 2025:
    The comment says “when ENABLE_IPC is enabled” but the code is if(WITH_LIBMULTIPROCESS). Shouldn’t that be if(ENABLE_IPC AND WITH_LIBMULTIPROCESS)?

    ryanofsky commented at 5:19 pm on February 7, 2025:

    re: #31741 (review)

    Agree, changed to suggested name


    ryanofsky commented at 5:27 pm on February 7, 2025:

    re: #31741 (review)

    Agree that’s a little clearer and more consistent with the documentation, so updated.

  69. in test/lint/lint-locale-dependence.py:54 in 99b7b857d3 outdated
    50@@ -51,6 +51,7 @@
    51 
    52 REGEXP_EXTERNAL_DEPENDENCIES_EXCLUSIONS = [
    53     "src/crypto/ctaes/",
    54+    "src/ipc/libmultiprocess/",
    


    vasild commented at 10:24 am on February 5, 2025:

    Commit 17329d17ef2f640c5f03e36351f1e19f43acf108 lint: Add exclusions for libmultiprocess subtree looks suboptimal because it adds exceptions for the entire src/ipc/libmultiprocess/ subtree for a whole class of issues. Because there are a few specific issues currently. Even if they are ok, it means new issues will sneak in src/ipc/libmultiprocess/ unnoticed.

    It would be better to either resolve the issues and not exclude src/ipc/libmultiprocess/ from linters or add exceptions for the particular issues mentioned in the commit message, so that newly added issues in the future will be detected.

    Here is a patch to resolve the locale-dependent issues:

      0diff --git i/src/ipc/libmultiprocess/example/calculator.cpp w/src/ipc/libmultiprocess/example/calculator.cpp
      1index 4290d68733..6926fb8b57 100644
      2--- i/src/ipc/libmultiprocess/example/calculator.cpp
      3+++ w/src/ipc/libmultiprocess/example/calculator.cpp
      4@@ -1,11 +1,12 @@
      5 // Copyright (c) 2021 The Bitcoin Core developers
      6 // Distributed under the MIT software license, see the accompanying
      7 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
      8 
      9 #include <calculator.h>
     10+#include <charconv>
     11 #include <fstream>
     12 #include <init.capnp.h>
     13 #include <init.capnp.proxy.h> // NOLINT(misc-include-cleaner)
     14 #include <init.h>
     15 #include <iostream>
     16 #include <memory>
     17@@ -41,13 +42,17 @@ static void LogPrint(bool raise, const std::string& message)
     18 int main(int argc, char** argv)
     19 {
     20     if (argc != 2) {
     21         std::cout << "Usage: mpcalculator <fd>\n";
     22         return 1;
     23     }
     24+    int fd;
     25+    if (std::from_chars(argv[1], argv[1] + strlen(argv[1]), fd).ec != std::errc{}) {
     26+        std::cout << argv[1] << " is not a number or is larger than an int\n";
     27+        return 1;
     28+    }
     29     mp::EventLoop loop("mpcalculator", LogPrint);
     30-    const int fd = std::stoi(argv[1]);
     31     std::unique_ptr<Init> init = std::make_unique<InitImpl>();
     32     mp::ServeStream<InitInterface>(loop, fd, *init);
     33     loop.loop();
     34     return 0;
     35 }
     36diff --git i/src/ipc/libmultiprocess/example/example.cpp w/src/ipc/libmultiprocess/example/example.cpp
     37index a4f84c55a7..d0e9458f5c 100644
     38--- i/src/ipc/libmultiprocess/example/example.cpp
     39+++ w/src/ipc/libmultiprocess/example/example.cpp
     40@@ -22,13 +22,13 @@ static auto Spawn(mp::EventLoop& loop, const std::string& process_argv0, const s
     41 {
     42     int pid;
     43     const int fd = mp::SpawnProcess(pid, [&](int fd) -> std::vector<std::string> {
     44         fs::path path = process_argv0;
     45         path.remove_filename();
     46         path.append(new_exe_name);
     47-        return {path.string(), std::to_string(fd)};
     48+        return {path.string(), std::format("{:d}", fd)};
     49     });
     50     return std::make_tuple(mp::ConnectStream<InitInterface>(loop, fd), pid);
     51 }
     52 
     53 static void LogPrint(bool raise, const std::string& message)
     54 {
     55diff --git i/src/ipc/libmultiprocess/example/printer.cpp w/src/ipc/libmultiprocess/example/printer.cpp
     56index ccaed6890c..924c651784 100644
     57--- i/src/ipc/libmultiprocess/example/printer.cpp
     58+++ w/src/ipc/libmultiprocess/example/printer.cpp
     59@@ -1,11 +1,12 @@
     60 // Copyright (c) 2021 The Bitcoin Core developers
     61 // Distributed under the MIT software license, see the accompanying
     62 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
     63 
     64 #include <fstream>
     65+#include <charconv>
     66 #include <init.capnp.h>
     67 #include <init.capnp.proxy.h> // NOLINT(misc-include-cleaner)
     68 #include <init.h>
     69 #include <iostream>
     70 #include <memory>
     71 #include <mp/proxy-io.h>
     72@@ -34,13 +35,17 @@ static void LogPrint(bool raise, const std::string& message)
     73 int main(int argc, char** argv)
     74 {
     75     if (argc != 2) {
     76         std::cout << "Usage: mpprinter <fd>\n";
     77         return 1;
     78     }
     79+    int fd;
     80+    if (std::from_chars(argv[1], argv[1] + strlen(argv[1]), fd).ec != std::errc{}) {
     81+        std::cout << argv[1] << " is not a number or is larger than an int\n";
     82+        return 1;
     83+    }
     84     mp::EventLoop loop("mpprinter", LogPrint);
     85-    const int fd = std::stoi(argv[1]);
     86     std::unique_ptr<Init> init = std::make_unique<InitImpl>();
     87     mp::ServeStream<InitInterface>(loop, fd, *init);
     88     loop.loop();
     89     return 0;
     90 }
     91diff --git i/src/ipc/libmultiprocess/src/mp/gen.cpp w/src/ipc/libmultiprocess/src/mp/gen.cpp
     92index c4d9a51675..267c2837a6 100644
     93--- i/src/ipc/libmultiprocess/src/mp/gen.cpp
     94+++ w/src/ipc/libmultiprocess/src/mp/gen.cpp
     95@@ -10,12 +10,13 @@
     96 #include <cstdint>
     97 #include <cstdio>
     98 #include <cstdlib>
     99 #include <errno.h>
    100 #include <fstream>
    101 #include <functional>
    102+#include <iostream>
    103 #include <kj/array.h>
    104 #include <kj/common.h>
    105 #include <kj/filesystem.h>
    106 #include <kj/memory.h>
    107 #include <kj/string.h>
    108 #include <map>
    109@@ -634,13 +635,13 @@ static void Generate(kj::StringPtr src_prefix,
    110     h << "#endif\n";
    111 }
    112 
    113 int main(int argc, char** argv)
    114 {
    115     if (argc < 3) {
    116-        fprintf(stderr, "Usage: " PROXY_BIN " SRC_PREFIX INCLUDE_PREFIX SRC_FILE [IMPORT_PATH...]\n");
    117+        std::cerr << "Usage: " << PROXY_BIN << " SRC_PREFIX INCLUDE_PREFIX SRC_FILE [IMPORT_PATH...]\n";
    118         exit(1);
    119     }
    120     std::vector<kj::StringPtr> import_paths;
    121     std::vector<kj::Own<const kj::ReadableDirectory>> import_dirs;
    122     auto fs = kj::newDiskFilesystem();
    123     auto cwd = fs->getCurrentPath();
    124diff --git i/src/ipc/libmultiprocess/src/mp/util.cpp w/src/ipc/libmultiprocess/src/mp/util.cpp
    125index 691ae0b34b..15c40b3373 100644
    126--- i/src/ipc/libmultiprocess/src/mp/util.cpp
    127+++ w/src/ipc/libmultiprocess/src/mp/util.cpp
    128@@ -3,12 +3,13 @@
    129 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
    130 
    131 #include <mp/config.h>
    132 #include <mp/util.h>
    133 
    134 #include <errno.h>
    135+#include <format>
    136 #include <kj/common.h>
    137 #include <kj/string-tree.h>
    138 #include <pthread.h>
    139 #include <sstream>
    140 #include <stdio.h>
    141 #include <string>
    142@@ -82,15 +83,13 @@ std::string LogEscape(const kj::StringTree& string)
    143     string.visit([&](const kj::ArrayPtr<const char>& piece) {
    144         if (result.size() > MAX_SIZE) return;
    145         for (const char c : piece) {
    146             if (c == '\\') {
    147                 result.append("\\\\");
    148             } else if (c < 0x20 || c > 0x7e) {
    149-                char escape[4];
    150-                snprintf(escape, 4, "\\%02x", c);
    151-                result.append(escape);
    152+                result.append(std::format("\\{:02x}", c));
    153             } else {
    154                 result.push_back(c);
    155             }
    156             if (result.size() > MAX_SIZE) {
    157                 result += "...";
    158                 break;
    159diff --git i/src/ipc/libmultiprocess/test/mp/test/test.cpp w/src/ipc/libmultiprocess/test/mp/test/test.cpp
    160index 7721d41ff6..7fc64f6741 100644
    161--- i/src/ipc/libmultiprocess/test/mp/test/test.cpp
    162+++ w/src/ipc/libmultiprocess/test/mp/test/test.cpp
    163@@ -8,12 +8,13 @@
    164 #include <mp/test/foo.h>
    165 
    166 #include <capnp/capability.h>
    167 #include <cstdio>
    168 #include <future>
    169 #include <functional>
    170+#include <iostream>
    171 #include <memory>
    172 #include <kj/common.h>
    173 #include <kj/memory.h>
    174 #include <kj/test.h>
    175 #include <string>
    176 #include <thread>
    177@@ -24,13 +25,15 @@ namespace test {
    178 
    179 KJ_TEST("Call FooInterface methods")
    180 {
    181     std::promise<std::unique_ptr<ProxyClient<messages::FooInterface>>> foo_promise;
    182     std::function<void()> disconnect_client;
    183     std::thread thread([&]() {
    184-        EventLoop loop("mptest", [](bool raise, const std::string& log) { printf("LOG%i: %s\n", raise, log.c_str()); });
    185+        EventLoop loop("mptest", [](bool raise, const std::string& log) {
    186+            std::cout << "LOG" << raise << ": " << log << "\n";
    187+        });
    188         auto pipe = loop.m_io_context.provider->newTwoWayPipe();
    189 
    190         auto connection_client = std::make_unique<Connection>(loop, kj::mv(pipe.ends[0]));
    191         auto foo_client = std::make_unique<ProxyClient<messages::FooInterface>>(
    192             connection_client->m_rpc_system->bootstrap(ServerVatId().vat_id).castAs<messages::FooInterface>(),
    193             connection_client.get(), /* destroy_connection= */ false);
    194diff --git i/test/lint/lint-locale-dependence.py w/test/lint/lint-locale-dependence.py
    195index 1b1aeffdcb..d84e458bb1 100755
    196--- i/test/lint/lint-locale-dependence.py
    197+++ w/test/lint/lint-locale-dependence.py
    198@@ -48,13 +48,12 @@ KNOWN_VIOLATIONS = [
    199     "src/wallet/bdb.cpp:.*DbEnv::strerror",  # False positive
    200     "src/util/syserror.cpp:.*strerror",      # Outside this function use `SysErrorString`
    201 ]
    202 
    203 REGEXP_EXTERNAL_DEPENDENCIES_EXCLUSIONS = [
    204     "src/crypto/ctaes/",
    205-    "src/ipc/libmultiprocess/",
    206     "src/leveldb/",
    207     "src/secp256k1/",
    208     "src/minisketch/",
    209     "src/tinyformat.h",
    210 ]
    211 
    

    And also another one to still run test_runner on src/ipc/libmultiprocess/ but only skip std::filesystem check for example.cpp and the include guard test for the entire src/ipc/libmultiprocess/:

     0diff --git i/test/lint/lint-include-guards.py w/test/lint/lint-include-guards.py
     1index 77af05c1c2..ed10deaa90 100755
     2--- i/test/lint/lint-include-guards.py
     3+++ w/test/lint/lint-include-guards.py
     4@@ -17,12 +17,13 @@ from lint_ignore_dirs import SHARED_EXCLUDED_SUBTREES
     5 
     6 HEADER_ID_PREFIX = 'BITCOIN_'
     7 HEADER_ID_SUFFIX = '_H'
     8 
     9 EXCLUDE_FILES_WITH_PREFIX = ['contrib/devtools/bitcoin-tidy',
    10                              'src/crypto/ctaes',
    11+                             'src/ipc/libmultiprocess',
    12                              'src/tinyformat.h',
    13                              'src/bench/nanobench.h',
    14                              'src/test/fuzz/FuzzedDataProvider.h'] + SHARED_EXCLUDED_SUBTREES
    15 
    16 
    17 def _get_header_file_lst() -> list[str]:
    18diff --git i/test/lint/lint_ignore_dirs.py w/test/lint/lint_ignore_dirs.py
    19index 7525eac341..af9ee7ef6b 100644
    20--- i/test/lint/lint_ignore_dirs.py
    21+++ w/test/lint/lint_ignore_dirs.py
    22@@ -1,6 +1,5 @@
    23 SHARED_EXCLUDED_SUBTREES = ["src/leveldb/",
    24                  "src/crc32c/",
    25                  "src/secp256k1/",
    26                  "src/minisketch/",
    27-                 "src/ipc/libmultiprocess/",
    28                 ]
    29diff --git i/test/lint/test_runner/src/main.rs w/test/lint/test_runner/src/main.rs
    30index fe77a98c29..a633f5d43d 100644
    31--- i/test/lint/test_runner/src/main.rs
    32+++ w/test/lint/test_runner/src/main.rs
    33@@ -284,13 +284,13 @@ fn lint_std_filesystem() -> LintResult {
    34         .args([
    35             "grep",
    36             "--line-number",
    37             "std::filesystem",
    38             "--",
    39             "./src/",
    40-            ":(exclude)src/ipc/libmultiprocess/",
    41+            ":(exclude)src/ipc/libmultiprocess/example/example.cpp",
    42             ":(exclude)src/util/fs.h",
    43         ])
    44         .status()
    45         .expect("command error")
    46         .success();
    47     if found {
    

    ryanofsky commented at 5:27 pm on February 7, 2025:

    re: #31741 (review)

    Thanks, applied both patches.

  70. in depends/packages.md:65 in 99b7b857d3 outdated
    60+build id, so if the directory contents change, the package and everything
    61+depending on it will be rebuilt. For efficiency, the tarball is cached once it
    62+has been created, but if the local directory is touched, it will be rebuilt.
    63+
    64+Local packages can be useful for using git submodules or subtrees to manage
    65+package sources, or for  testing local changes that are not available to
    


    vasild commented at 11:04 am on February 5, 2025:
    0package sources, or for testing local changes that are not available to
    

    ryanofsky commented at 5:28 pm on February 7, 2025:

    re: #31741 (review)

    Thanks, updated

  71. vasild commented at 11:11 am on February 5, 2025: contributor

    Approach ACK 99b7b857d3b8b9ce9bd7501e2480583369740c55

    I think it would be best if all sources are build with the same flags and the same build mechanism is used by devs, CI and depends. I also do not see a harm of an extra option which is off by default, to use an external libmultiprocess, if that makes libmultiprocess’ devs life easier.

    PS I find 97058b53e8 depends: Switch libmultiprocess packages to use local git subtree hard to review.

  72. luke-jr commented at 2:21 pm on February 5, 2025: member
    I think all those benefits could be achieved by adding the glue yet having the developer who wants to test checkout the upstream repo into a subdirectory manually to use it (cmake can complain clearly if missing)
  73. ryanofsky commented at 2:44 pm on February 5, 2025: contributor

    I think all those benefits could be achieved by adding the glue yet having the developer who wants to test checkout the upstream repo into a subdirectory manually to use it (cmake can complain clearly if missing) @luke-jr, Yes, we could provide a glue script to download the git repository instead of mirroring it in a git subtree, pinning it to a hash to avoid bringing in unreviewed changes. This is essentially what depends system does now. Do you think a glue script would be better than a subtree though? That just seems more awkward all around.

  74. luke-jr commented at 3:01 pm on February 5, 2025: member

    Yes, I think it would be better.

    I would just have the git clone command documented though, no need to automate it?

  75. ryanofsky commented at 3:13 pm on February 5, 2025: contributor

    @luke-jr Can you say why you think manually cloning the repository might be better? I can’t see any upsides to that, and the problem with just cloning it is it would put everybody at risk because arbitrary changes could be pushed to the external repository and we don’t want anybody to run arbitrary code, only code that we have reviewed and pinned at a specific git revision.

    Right now we are pinning this revision with the depends system, and this PR is just moving the pinning from the depends system to the git subtree mechanism instead. Alternately we could pin the revision using a git submodule, a script, installation instructions, package managers, or any other number of mechanisms but the external revision does need to be pinned somehow to provide review and a basic level of security.

  76. Sjors commented at 3:28 pm on February 5, 2025: member

    Unless I’m missing something, I don’t think @luke-jr’s critique is specific to libmultiprocess, but rather his consistent position on including libraries. See e.g. #6637 (comment) and #28981#pullrequestreview-1763876066.

    I think we should subtree libmultiprocess the same way we do with the other subtrees (to the extend possible). We can revisit that entire process later imo.

  77. ryanofsky commented at 4:02 pm on February 5, 2025: contributor

    Unless I’m missing something, I don’t think @luke-jr’s critique is specific to libmultiprocess, but rather his consistent position on including libraries.

    I see this but I think I actually agree with (or at least sympathize with) Luke’s position on libraries, and am trying to ensure more clean integration with this library than other libraries that we depend on like leveldb, univalue, libevent. This PR is not forking the ilbrary or make making local changes to it. It is not replacing its build system. It is not dropping support for building with an external version. The point of the subtree is just to pin a reviewed version of the sources, and make them available so they don’t have to be downloaded separately. If external packagers want use the subtree, they can use it. If they want to ignore it and use a separate package they can do that too. There are benefits to monolithic builds favored by Cory and benefits to more modular builds favored by Luke. This PR is not picking a side because in this case the modular build is simple and has been supported for years and the monolithic build is the new thing that requires (slightly) more complicated code to support.

  78. luke-jr commented at 4:04 pm on February 5, 2025: member

    Downloading something separately is not a real issue.

    But once it’s a subtree, you can’t avoid downloading it, and it might even be messy to remove it (to ensure it isn’t used).

  79. ryanofsky commented at 4:24 pm on February 5, 2025: contributor

    But once it’s a subtree, you can’t avoid downloading it, and it might even be messy to remove it (to ensure it isn’t used).

    By design, it is not messy to remove. The subtree is referenced in exactly one add_subdirectory call, and nowhere else in the cmake build. If the -DWITH_LIBMULTIPROCESS=ON is set the add_subdirectory call is not made and a find_package call is made instead to locate a separately installed version of the library.

    Adding the subtree does increase the size of source slightly from 12.74mb before this change to 12.80mb after (with git archive | gzip -c).

  80. Sjors commented at 1:12 pm on February 6, 2025: member
    @ryanofsky if you need to retouch can you also rebase? That makes it easier for my followup PRs since 00_setup_env_i686_multiprocess.sh changed in master.
  81. ryanofsky referenced this in commit 38459f174f on Feb 7, 2025
  82. ryanofsky referenced this in commit 6fb68977ca on Feb 7, 2025
  83. ryanofsky referenced this in commit c8351c5f0b on Feb 7, 2025
  84. ryanofsky referenced this in commit 9ba88dc2c7 on Feb 7, 2025
  85. ryanofsky force-pushed on Feb 7, 2025
  86. DrahtBot added the label interfaces on Feb 7, 2025
  87. ryanofsky commented at 5:51 pm on February 7, 2025: contributor

    Rebased 99b7b857d3b8b9ce9bd7501e2480583369740c55 -> 4ca2462bf516846c5552378345e243bc819553a3 (pr/subtree.8 -> pr/subtree.9, compare) to fix conflict with #31800. Also applied review suggestions from vasild, renamed Libmultiprocess_EXTERNAL_MPGEN cmake option to MPGEN_EXECUTABLE as suggested https://github.com/chaincodelabs/libmultiprocess/pull/147#discussion_r1939821452


    re: #31741#pullrequestreview-2595012158

    Thanks for the review!

    PS I find 97058b5 depends: Switch libmultiprocess packages to use local git subtree hard to review.

    Thanks if you have more information on what aspects of that were hard to review, I could try to make improvements. This commit only adds a few lines of code, but maybe surrounding code or commit message could be to be improved to make context clearer. In meantime I added a few comments about the new code. Also I have to say I can’t recommend chatgpt highly enough for understanding and explaining makefile code. This change would have taken me 10 times as long to implement without it because makefile syntax and idioms are so unfamiliar.

  88. Sjors commented at 5:58 pm on February 7, 2025: member

    Getting a bunch of CI failings in #30975:

    0 error: no member named 'format' in namespace 'std'
    1                result.append(std::format("\\{:02x}", c));
    

    e.g. macOS native: https://github.com/bitcoin/bitcoin/actions/runs/13205261715/job/36866826238?pr=30975

    ARM, libbitcoinkernel, previous releases jobs:

    017:54:12.396] [ 14%] Building CXX object CMakeFiles/mputil.dir/src/mp/util.cpp.o
    1[17:54:12.396] /ci_container_base/depends/work/build/arm-linux-gnueabihf/native_libmultiprocess/-ef29ff76f60/src/mp/util.cpp:9:10: fatal error: format: No such file or directory
    2[17:54:12.396]     9 | #include <format>
    

    https://cirrus-ci.com/task/5553844643430400

    The latter also happens when I make a guix build with #31802 (rebased, but not pushed yet).

  89. ryanofsky referenced this in commit eca8fd3eee on Feb 7, 2025
  90. ryanofsky referenced this in commit 408990787f on Feb 7, 2025
  91. ryanofsky force-pushed on Feb 7, 2025
  92. ryanofsky force-pushed on Feb 7, 2025
  93. DrahtBot added the label CI failed on Feb 7, 2025
  94. DrahtBot commented at 7:19 pm on February 7, 2025: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/36870768941

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  95. ryanofsky commented at 7:23 pm on February 7, 2025: contributor
    Rebased 4ca2462bf516846c5552378345e243bc819553a3 -> 658176160026a3e44c5eaa941880aed34f5926d4 (pr/subtree.9 -> pr/subtree.10, compare) dropping std::format calls to address #31741 (comment) Updated 658176160026a3e44c5eaa941880aed34f5926d4 -> 5e39b44d83d2d23b6fbd9ff6fb9267817f90f8be (pr/subtree.10 -> pr/subtree.11, compare) adding back linter exclusion to fix https://cirrus-ci.com/task/5654367178588160 Updated 5e39b44d83d2d23b6fbd9ff6fb9267817f90f8be -> b375aeaeefb70023c3c2cd74e3841e382473b292 (pr/subtree.11 -> pr/subtree.12, compare) with more documentation updates
  96. ryanofsky force-pushed on Feb 7, 2025
  97. DrahtBot removed the label CI failed on Feb 7, 2025
  98. hebasto commented at 7:21 am on February 8, 2025: member
    Could the CAPNP_EXECUTABLE, CAPNPC_CXX_EXECUTABLE and all CapnProto_* CMake variables be marked as advanced to avoid cluttering the CMake cache view when inspected by the user?
  99. ryanofsky force-pushed on Feb 10, 2025
  100. ryanofsky commented at 4:55 am on February 10, 2025: contributor

    Rebased b375aeaeefb70023c3c2cd74e3841e382473b292 -> e7743aa5df6387964c23f0629debf8c3cbb8ed4c (pr/subtree.12 -> pr/subtree.13, compare) adding https://github.com/chaincodelabs/libmultiprocess/pull/159 and marking capnp cmake variables as advanced as requested.

    Could the CAPNP_EXECUTABLE, CAPNPC_CXX_EXECUTABLE and all CapnProto_* CMake variables be marked as advanced to avoid cluttering the CMake cache view when inspected by the user?

    Sure, done in latest push.

  101. in depends/funcs.mk:47 in e7743aa5df outdated
    38@@ -39,8 +39,32 @@ define fetch_file
    39       $(call fetch_file_inner,$(1),$(FALLBACK_DOWNLOAD_PATH),$(3),$(4),$(5))))
    40 endef
    41 
    42+# Shell script to create a source tarball in $(1)_source from local directory
    43+# $(1)_local_dir instead of downloading remote sources. Tarball is recreated if
    44+# any paths in the local directory have a newer mtime, and checksum of the
    45+# tarball is saved to $(1)_fetched and returned as output.
    46+define fetch_local_dir_sha256
    47+    if ! [ -f $($(1)_source) ] || [ -n "$$(find $($(1)_local_dir) -newer $($(1)_source) | head -n1)" ]; then \
    


    vasild commented at 8:45 am on February 10, 2025:
    nit: | head -n1 is unnecessary

    ryanofsky commented at 3:17 pm on February 10, 2025:

    re: #31741 (review)

    nit: | head -n1 is unnecessary

    I think it’s natural and important to prevent bad worst-case behavior. Having head -n1 lets find command exit early and not continue looking for newer files after it has already found some, because its stdout will be closed. Also (probably more importantly) it prevents the shell from having to build a up a huge string with a list of new paths.

  102. in depends/funcs.mk:97 in e7743aa5df outdated
     98+$(1)_download_file?=$($(1)_file_name)
     99+$(1)_source_dir:=$(SOURCES_PATH)
    100+# If $(1)_file_name) is empty and $(1)_local_dir is nonempty, set file name to a
    101+# .tar file named after the directory path (with dots and slashes collapsed and
    102+# replaced by dashes) that will be created from the directory.
    103+$(if $($(1)_file_name),,$(if $($(1)_local_dir),$(eval $(1)_file_name:=$(subst $(null) $(null),-,$(strip $(subst ., ,$(subst /, ,$($(1)_local_dir))))).tar)))
    


    vasild commented at 8:46 am on February 10, 2025:

    nit: $(1)_file_name) -> $(1)_file_name

    Thanks for the comment. That line is still !@*&(!@*&~&_ in my eyes. I wonder if it can be split into shorter and more easily understandable define foo functions…


    ryanofsky commented at 3:17 pm on February 10, 2025:

    re: #31741 (review)

    nit: $(1)_file_name) -> $(1)_file_name

    Thanks for the comment. That line is still !@*&(!@*&~&_ in my eyes. I wonder if it can be split into shorter and more easily understandable define foo functions…

    Thanks for the close reading. Fixed comment and split this up to use a helper function. I think it should be more readable now.

  103. vasild approved
  104. vasild commented at 8:54 am on February 10, 2025: contributor
    ACK e7743aa5df6387964c23f0629debf8c3cbb8ed4c
  105. DrahtBot requested review from hebasto on Feb 10, 2025
  106. DrahtBot requested review from TheCharlatan on Feb 10, 2025
  107. DrahtBot requested review from Sjors on Feb 10, 2025
  108. DrahtBot requested review from BrandonOdiwuor on Feb 10, 2025
  109. in src/test/CMakeLists.txt:167 in e7743aa5df outdated
    160@@ -161,9 +161,10 @@ if(ENABLE_WALLET)
    161   add_subdirectory(${PROJECT_SOURCE_DIR}/src/wallet/test wallet)
    162 endif()
    163 
    164-if(WITH_MULTIPROCESS)
    165+if(ENABLE_IPC)
    166   target_link_libraries(bitcoin_ipc_test
    167     PRIVATE
    168+      Boost::headers
    


    hebasto commented at 9:17 am on February 10, 2025:

    1b3613505deb4baeacc3c4c2fc91ccca1a8ecec5

    Usually, external dependencies follow the internal ones.


    ryanofsky commented at 3:18 pm on February 10, 2025:

    re: #31741 (review)

    1b36135

    Usually, external dependencies follow the internal ones.

    Makes sense, fixed this.

  110. DrahtBot requested review from hebasto on Feb 10, 2025
  111. hebasto commented at 9:31 am on February 10, 2025: member

    Could the CAPNP_EXECUTABLE, CAPNPC_CXX_EXECUTABLE and all CapnProto_* CMake variables be marked as advanced to avoid cluttering the CMake cache view when inspected by the user?

    Sure, done in latest push.

    The CapnProto_* CMake variables are still lingering:

     0$ cmake -B build -DENABLE_IPC=ON -L 2>&1 | grep CapnProto_
     1CapnProto_capnp-json_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libcapnp-json-1.0.1.so
     2CapnProto_capnp-rpc_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libcapnp-rpc-1.0.1.so
     3CapnProto_capnp-websocket_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libcapnp-websocket-1.0.1.so
     4CapnProto_capnp_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libcapnp-1.0.1.so
     5CapnProto_capnpc_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libcapnpc-1.0.1.so
     6CapnProto_kj-async_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libkj-async-1.0.1.so
     7CapnProto_kj-gzip_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libkj-gzip-1.0.1.so
     8CapnProto_kj-http_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libkj-http-1.0.1.so
     9CapnProto_kj-test_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libkj-test-1.0.1.so
    10CapnProto_kj-tls_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libkj-tls-1.0.1.so
    11CapnProto_kj_IMPORTED_LOCATION:FILEPATH=/usr/lib/x86_64-linux-gnu/libkj-1.0.1.so
    
  112. ryanofsky force-pushed on Feb 10, 2025
  113. ryanofsky force-pushed on Feb 10, 2025
  114. ryanofsky commented at 3:35 pm on February 10, 2025: contributor

    Rebased e7743aa5df6387964c23f0629debf8c3cbb8ed4c -> e484d0bd11408489d83aaecf49a238f98a117696 (pr/subtree.13 -> pr/subtree.14, compare) adding https://github.com/chaincodelabs/libmultiprocess/pull/161 and trying another suggested readability improvement to depends code, hiding more capnp variables, and rebasing for #30975 (comment) Updated e484d0bd11408489d83aaecf49a238f98a117696 -> c6658d675d0bb945f53dc62f7e9b95c7bba973bb (pr/subtree.14 -> pr/subtree.15, compare) tweaking target_link_libraries order as suggested

    re: #31741 (comment)

    The CapnProto_* CMake variables are still lingering:

    Oddly I do not see any of those on my system (I am using nixpkgs capnproto 1.0.2 and those variables do not appear in the cache at all) but I added all of them, thanks.

  115. hebasto commented at 3:43 pm on February 10, 2025: member
    Approach ACK c6658d675d0bb945f53dc62f7e9b95c7bba973bb.
  116. theuni commented at 10:20 pm on February 10, 2025: member

    Could the CAPNP_EXECUTABLE, CAPNPC_CXX_EXECUTABLE and all CapnProto_* CMake variables be marked as advanced to avoid cluttering the CMake cache view when inspected by the user?

    Agreed on the rest, but CAPNP_EXECUTABLE and CAPNPC_CXX_EXECUTABLE are relevant to cross-compilers and shouldn’t be hidden. In fact, to be thorough, multiprocess.md should mention them as well.

  117. Sjors referenced this in commit 334dcb7e6a on Feb 11, 2025
  118. Sjors referenced this in commit 7a686bf97f on Feb 11, 2025
  119. DrahtBot added the label Needs rebase on Feb 12, 2025
  120. vasild approved
  121. vasild commented at 9:07 am on February 12, 2025: contributor

    ACK c6658d675d0bb945f53dc62f7e9b95c7bba973bb

    CAPNP_EXECUTABLE and CAPNPC_CXX_EXECUTABLEmultiprocess.md should mention them as well

    :+1: I guess in the “Alternately …” paragraph.

  122. DrahtBot requested review from hebasto on Feb 12, 2025
  123. ryanofsky force-pushed on Feb 13, 2025
  124. DrahtBot removed the label Needs rebase on Feb 13, 2025
  125. ryanofsky commented at 5:27 pm on February 13, 2025: contributor

    Rebased c6658d675d0bb945f53dc62f7e9b95c7bba973bb -> b2108240dec5d858d17af798fdc79037f894786d (pr/subtree.15 -> pr/subtree.16, compare) to fix conflict with #31834. Also added another fix for #30975 and documented & unhid CAPNP_EXECUTABLE variable as suggested

    re: #31741 (comment)

    Agreed on the rest, but CAPNP_EXECUTABLE and CAPNPC_CXX_EXECUTABLE are relevant to cross-compilers and shouldn’t be hidden. In fact, to be thorough, multiprocess.md should mention them as well.

    Thanks, added these to documentation and unhid them in cmake since more likely to be set manually than other variables that are hidden, though to be fair these are advanced, esoteric variables that you would only set if you were cross compiling bitcoin without a package manger.

  126. hebasto commented at 5:42 pm on February 16, 2025: member

    96d3b2a0bb0c267569162b055ca306f709ec0b4e “cmake: Fix fuzzer “multiple definition of `main’” errors”

    cc @dergoegge

  127. maflcko commented at 7:04 am on February 17, 2025: member

    96d3b2a “cmake: Fix fuzzer “multiple definition of `main’” errors”

    It may be good to explain this commit, or add steps to reproduce. When using libFuzzer one would only want to build for fuzzing, so not setting the corresponding build flag to disable all other executables seems odd. See also:

    0ci/test/00_setup_env_mac_native_fuzz.sh:export BITCOIN_CONFIG="-DBUILD_FOR_FUZZING=ON"
    1ci/test/00_setup_env_native_fuzz.sh: -DBUILD_FOR_FUZZING=ON \
    2ci/test/00_setup_env_native_fuzz_with_msan.sh: -DBUILD_FOR_FUZZING=ON \
    3ci/test/00_setup_env_native_fuzz_with_valgrind.sh: -DBUILD_FOR_FUZZING=ON \
    
  128. ryanofsky commented at 1:59 pm on February 17, 2025: contributor

    96d3b2a “cmake: Fix fuzzer “multiple definition of `main’” errors”

    It may be good to explain this commit, or add steps to reproduce. @maflcko guessing you maybe did not notice the comments or commit message? All the points you raised should be covered there, but let me know if anything should be updated or clarified.

    Steps to reproduce are to check out the branch, revert the commit, and try to build with fuzzing and IPC at the same time. A simple command line is: cmake -B build -DBUILD_FOR_FUZZING=ON -DSANITIZERS=fuzzer -DENABLE_IPC=ON. And CI failures were:

    https://cirrus-ci.com/task/5366255504326656?logs=ci#L2817 https://cirrus-ci.com/task/5233064575500288?logs=ci#L2384

    When using libFuzzer one would only want to build for fuzzing, so not setting the corresponding build flag to disable all other executables seems odd.

    The build flag that disables building other executables is set, but the mpgen executable is needed to generate code that is linked into the fuzz executable.

    In general, there is no need to disable building all other executables when instrumenting libraries for fuzzing. The requirement to do for bitcoin libraries is specific limitation of our build system and a consequence of some internal design decisions including the decision not to have an normal assert macro that is completely disabled in debug builds.

  129. maflcko commented at 2:20 pm on February 17, 2025: member
    Thanks for explaining. I must have skipped over the last paragraph in the commit message, which mentions the mpgen executable. However, given the lack of any fuzz target that could possibly call into mp code, I wonder if it may be easier to just disable the MP configure option when compiling for fuzzing. Otherwise, it seems like changing the build code to support compiling code that will never be executed anyway.
  130. ryanofsky force-pushed on Feb 17, 2025
  131. ryanofsky commented at 3:43 pm on February 17, 2025: contributor

    I wonder if it may be easier to just disable the MP configure option when compiling for fuzzing

    Not sure I understand the suggestion. Is it to not set ENABLE_IPC in the CI configuration? Show an error in the build when ipc and fuzzing are enabled together? Ignore the ipc option when fuzzing is enabled?

    The bugfix here is simple, even though the bug is complicated to explain. It builds libraries with -fsanitize=fuzzer-no-link which is how libraries should be built, and builds the fuzz executable with -fsanitize=fuzzer which is how it should be built. Unlike other workarounds, it doesn’t make assumptions about what code will be fuzzed or how code may be generated.

    However, given the lack of any fuzz target that could possibly call into mp code […] Otherwise, it seems like changing the build code to support compiling code that will never be executed anyway.

    Agree you would probably not fuzz test multiprocess code, but IPC code should be a good candidate for fuzz testing. With #29409, the FuzzedWallet could be passed a capnproto Chain interface instead of the default one, and IPC code could be used without even writing new tests.

    Your feedback did made me realize that the bugfix here actually works too well, because it fixes the “multiple definition of main” link error when -DSANITIZERS=fuzzer is specified without -DBUILD_FOR_FUZZING=ON, and we actually do want an error in that case. So latest push adds a cmake error for that, providing much faster feedback and much less confusing message than the current link error. I also shortened the commit message and tried to make it clearer.

    Updated b2108240dec5d858d17af798fdc79037f894786d -> 66c318abe1b7df3e2ff1035376bc5f4b2059626a (pr/subtree.16 -> pr/subtree.17, compare) with updates to the “multiple definition of main” commit

  132. maflcko commented at 4:01 pm on February 17, 2025: member

    Agree you would probably not fuzz test multiprocess code, but IPC code should be a good candidate for fuzz testing. With #29409, the FuzzedWallet could be passed a capnproto Chain interface instead of the default one, and IPC code could be used without even writing new tests.

    Sure, but the ipc code is mostly just serialization/wrapper code, so if there is full coverage via non-fuzz tests, running objects in the fuzz tests through it doesn’t increase coverage. For example, if the serialize code is broken for CScript and crashes (https://github.com/chaincodelabs/libmultiprocess/issues/122#issuecomment-2512566578), it will likely crash for any CScript already in the existing tests. In fact, if devs see a fuzzing failure due to this, they may be confused into thinking that the fuzz engine found a particular byte sequence that failed, which is not the case. But no strong opinion.

  133. ryanofsky commented at 4:31 pm on February 17, 2025: contributor

    It’s definitely not true that if serialization code works in one case it works in all cases. Bitcoin serialization code does all kinds of weird things including crashing when it is in a state it doesn’t feel like serializing. And IPC code in general seems like an natural place for fuzz testing. For example, writing a fuzz test to write bytes to a socket serving the Echo interface could expose possible crash bugs in capnproto code.

    If there are problems you see with the bugfix here, they would be helpful to know about. But to me the change seems like an obvious improvement. (1) It fixes the CI bug (2) It documents sanitize_interface and removes an attribute(weak) hack. (3) It doesn’t compile libraries with a flag intended to bring in a main() function. (4) It provides immediate and clear feedback if -DSANITIZERS=fuzzer is used with without -DBUILD_FOR_FUZZING=ON which is is a perfectly logical thing to try and easy thing to forget about, instead of providing an opaque error message and failing at the very end of a slow build. (5) It lets us write useful fuzz tests that cover IPC code.

  134. in CMakeLists.txt:374 in 66c318abe1 outdated
    369+  string(REGEX REPLACE "(^|,)fuzzer($|,)" "\\1fuzzer-no-link\\2" SANITIZE_FLAG "${SANITIZERS}")
    370+  set(FUZZ_FLAG "")
    371+  if (NOT "${SANITIZE_FLAG}" STREQUAL "${SANITIZERS}")
    372+      set(FUZZ_FLAG "-fsanitize=fuzzer")
    373+      if (NOT BUILD_FOR_FUZZING)
    374+          message(FATAL_ERROR "Error: Enabling -DSANITIZERS=fuzzer without -DBUILD_FOR_FUZZING=ON is not supported.")
    


    vasild commented at 10:39 am on February 19, 2025:

    style: Our CMake files use 2-space for indentation and no space after if like this: if(FOO. When we were switching to CMake I assumed this is ok because it was all consistent. Now I realize that people will keep adding changes that use 4-space indentation and a space after if because the rest of the code base uses that.

    Maybe just ignore this and accept that our CMake files will end up with an inconsistent mixture of different styles?


    ryanofsky commented at 3:20 pm on February 19, 2025:

    re: #31741 (review)

    Good catch, fixed indentation and spacing. I don’t think it should be too much of a burden to enforce a consistent style for cmake code given how little cmake code there is relatively. There also seems to be a cmake-lint tool that could maybe help.


    vasild commented at 9:30 am on February 20, 2025:
    Wow! I didn’t know cmake-lint existed and didn’t even imagine it could exist ;) It would be perfect to enforce this in CI.
  135. vasild approved
  136. vasild commented at 10:40 am on February 19, 2025: contributor
    ACK 66c318abe1b7df3e2ff1035376bc5f4b2059626a
  137. DrahtBot requested review from hebasto on Feb 19, 2025
  138. ryanofsky force-pushed on Feb 19, 2025
  139. ryanofsky commented at 3:26 pm on February 19, 2025: contributor
    Updated 66c318abe1b7df3e2ff1035376bc5f4b2059626a -> 828c440b0dea29ab41ffc32d88969176d1286655 (pr/subtree.17 -> pr/subtree.18, compare) fixing cmake whitespace.
  140. DrahtBot added the label CI failed on Feb 19, 2025
  141. DrahtBot removed the label CI failed on Feb 19, 2025
  142. vasild approved
  143. vasild commented at 9:22 am on February 20, 2025: contributor

    ACK 828c440b0dea29ab41ffc32d88969176d1286655

    Strong Concept ACK on #31741 (review) (enforce CMake style, unrelated to this PR).

  144. DrahtBot added the label Needs rebase on Feb 20, 2025
  145. Squashed 'src/ipc/libmultiprocess/' content from commit 1954f7f65661
    git-subtree-dir: src/ipc/libmultiprocess
    git-subtree-split: 1954f7f65661d49e700c344eae0fc8092decf975
    db6b95d53a
  146. Merge commit 'db6b95d53a69a009330dd69e3cc25b40bf121d7f' as 'src/ipc/libmultiprocess' 76b5b4310e
  147. scripted-diff: s/WITH_MULTIPROCESS/ENABLE_IPC/ in cmake
    Rename WITH_MULTIPROCESS to ENABLE_IPC, because ENABLE_IPC is a more accurate
    name for the feature. It controls whether the src/ipc/ directory is built and
    whether IPC features like -ipcbind, -ipcconnect, and -ipcfd are available. It
    does NOT currently enable multiprocess features which are implemented in #10102
    building on top of the IPC features. It will also no longer (as of the next
    commit), control whether a find_package call is made so the "WITH_" prefix is
    also inappropriate.
    
    -BEGIN VERIFY SCRIPT-
    git grep -l WITH_MULTIPROCESS | xargs sed -i s/WITH_MULTIPROCESS/ENABLE_IPC/g
    -END VERIFY SCRIPT-
    d61d7e8ddf
  148. cmake: Support building with libmultiprocess subtree
    When ENABLE_IPC option is on, build with libmultiprocess subtree and
    `add_subdirectory(src/ipc/libmultiprocess)` instead of external package
    and `find_package(Libmultiprocess)` by default.
    
    Behavior can be toggled with `WITH_LIBMULTIPROCESS` option. Using a subtree
    should be more convenient for most bitcoin developers, but using an external
    package is more convenient for developing in the libmultiprocess repository.
    
    The `WITH_LIBMULTIPROCESS` option is also used to avoid needing to changing the
    depends build here. But in later commits, the depends build is switched to use
    the add_subdirectory build as well.
    
    Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
    24f53bef69
  149. cmake: Fix ctest mptest "Unable to find executable" errors
    This change is technically not needed to add libmultiprocess as a subtree, but
    it avoids a CI failure in followup PR #30975 which enables multiprocess build
    option in more CI jobs. In that PR, several jobs fail due to the mptest
    executable not being built by default, as reported
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2623801480
    95bbdd3523
  150. cmake: Fix warnings from boost headers
    This change is technically not needed to add libmultiprocess as a subtree, but
    it avoids a CI failure in followup PR #30975 which enables multiprocess build
    option in more CI jobs. In that PR, the "macOS 14 native no depends job" fails
    due to warnings in boost headers treated as errors, reported in
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2623801480 and
    https://github.com/chaincodelabs/libmultiprocess/issues/138
    a36912427a
  151. cmake: Fix clang-tidy "no input files" errors
    This change is technically not needed to add libmultiprocess as a subtree, but
    it avoids a CI failure in followup PR #30975 which enables multiprocess build
    option in more CI jobs. In that PR, clang-tidy job fails due to missing
    generated example files as reported
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2627116832
    
    Different fixes were suggested
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2627152623 and
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2627403382
    
    Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
    00b1406430
  152. cmake: Fix fuzzer "multiple definition of `main'" errors
    This change is technically not needed to add libmultiprocess as a subtree, but
    it avoids a CI failure in followup PR #30975 which enables multiprocess build
    option in more CI jobs. Specifically this makes it possible to use the
    BUILD_FOR_FUZZING and ENABLE_IPC options together easily.
    
    The change just updates the cmake build so when -DSANITIZERS=fuzzer is
    specified, the fuzz test binary is built with -fsanitize=fuzzer (so it can use
    libFuzzer's main function), and libraries are built with
    -fsanitize=fuzzer-no-link (so they can be linked into other executables with
    their own main functions).
    
    Previously when -DSANITIZERS=fuzzer was specified, -fsanitize=fuzzer was
    applied to ALL libraries and executables.  This was inappropriate because it
    made it impossible to build any executables other than the fuzz test executable
    without triggering link errors:
    
    - "multiple definition of `main'"
    - "undefined reference to `LLVMFuzzerTestOneInput'"
    
    if they depended on any libraries instrumented for fuzzing.
    
    This was especially a problem when the ENABLE_IPC option was set because the
    mpgen code generator could not be built, so nothing else that depended on
    generated sources including the fuzz test binary could be built either.
    6f1875c8ae
  153. doc: Update documentation to explain libmultiprocess subtree
    Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
    9394be1ad4
  154. lint: Add exclusions for libmultiprocess subtree
    Without this change linter produces errors about:
    
    - Use of std::filesystem the libmultiprocess example program.
    - Use of locale-dependent functions in example program, in the build time code
      generator, and in the runtime library for debug logging.
    - Include guards not beginning with BITCOIN_
    7b58c3f47b
  155. depends, moveonly: split up int_get_build_id function
    Move parts of the int_get_build_id into a new int_get_build_properties
    function. There is no change in behavior. This just organizes assignments
    better so some build properties can be used to help compute build ids in the
    next commit.
    e9259f537c
  156. depends: Switch libmultiprocess packages to use local git subtree
    With newly introduced libmultiprocess subtree, there's no need for depends
    system to download and track changes to the upstream repository.
    
    Note that adding the libmultiprocess subtree does not allow dropping
    libmultiprocess packages from the depends build, because libmultiprocess
    includes a code generation tool called mpgen, and in cross-compiled builds,
    bitcoin core's cmake build system doesn't have access to a native toolchain and
    can't build mpgen itself, so the depends system (or the native environment if
    not using depends) needs to supply it.
    61e4a1ee9d
  157. depends: remove non-native libmultiprocess build
    Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
    c2c5a0f492
  158. ryanofsky force-pushed on Feb 24, 2025
  159. ryanofsky commented at 1:10 pm on February 24, 2025: contributor
    Rebased 828c440b0dea29ab41ffc32d88969176d1286655 -> da25311e3a88941bf99a732be1aa3568a65f2273 (pr/subtree.18 -> pr/subtree.19, compare) due to conflict with #31662
  160. DrahtBot removed the label Needs rebase on Feb 24, 2025
  161. maflcko commented at 2:05 pm on February 24, 2025: member
    The first commit is split up in #31945 and I tagged it for 29.0 as a bugfix. Thus, reviewers of this pull may want to review #31945 first.
  162. ryanofsky force-pushed on Feb 25, 2025
  163. ryanofsky force-pushed on Feb 25, 2025
  164. ryanofsky commented at 8:04 pm on February 25, 2025: contributor
    Rebased da25311e3a88941bf99a732be1aa3568a65f2273 -> c2c5a0f492ae2ce54afdd031e8a2f2689a8c4942 (pr/subtree.19 -> pr/subtree.20, compare) with no changes after #31945 merge
  165. pablomartin4btc referenced this in commit bd2453d134 on Feb 26, 2025
  166. pablomartin4btc referenced this in commit 75d5d235a6 on Feb 26, 2025
  167. pablomartin4btc commented at 10:04 pm on February 26, 2025: member
    Concept ACK
  168. fanquake referenced this in commit 3c1f72a367 on Feb 27, 2025
  169. ngn999 referenced this in commit fd12bfb763 on Feb 27, 2025
  170. fanquake commented at 9:52 pm on February 28, 2025: member
    Before we do this, can we also move the libmultiprocess repository into the bitcoin-core org.
  171. ryanofsky referenced this in commit 34aeb70748 on Mar 4, 2025
  172. DrahtBot added the label Needs rebase on Mar 5, 2025
  173. DrahtBot commented at 2:55 pm on March 5, 2025: contributor

    🐙 This pull request conflicts with the target branch and needs rebase.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-03-09 21:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me