multiprocess: Add libmultiprocess git subtree #31741

pull ryanofsky wants to merge 13 commits into bitcoin:master from ryanofsky:pr/subtree changing 78 files +5963 −59
  1. ryanofsky commented at 4:48 pm on January 27, 2025: contributor

    This adds the libmultiprocess library as a subtree in src/ipc/libmultiprocess and allows it to be built with the cmake -DBUILD_MULTIPROCESS option, which is disabled by default.

    A cmake -DWITH_LIBMULTIPROCESS option for building with external libmultiprocess packages is also kept, because this is needed to support cross compilation. (A cross-compiling cmake build cannot easily build and run a native code generation tool.)

    This PR includes the following manual changes (not created automatically with git subtree add) which make updates to the build system and documentation:


    Previous minisketch subtree PR #23114 may be useful for comparison

    Instructions for subtree verification can be found:

    TL&DR:

    0git remote add --fetch libmultiprocess https://github.com/chaincodelabs/libmultiprocess.git
    1test/lint/git-subtree-check.sh -r src/ipc/libmultiprocess
    

    This PR is part of the process separation project.

  2. DrahtBot commented at 4:48 pm on January 27, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31741.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Concept NACK luke-jr
    Concept ACK TheCharlatan, Sjors, hebasto, BrandonOdiwuor
    Approach ACK vasild

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #31765 (cmake: Install man pages for configured targets only by hebasto)
    • #31161 (cmake: Set top-level target output locations by hebasto)
    • #30975 (ci: build multiprocess on most jobs by Sjors)
    • #28710 (Remove the legacy wallet and BDB dependency by achow101)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. ryanofsky marked this as a draft on Jan 27, 2025
  4. ryanofsky force-pushed on Jan 27, 2025
  5. DrahtBot commented at 5:01 pm on January 27, 2025: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/36237440607

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  6. DrahtBot added the label CI failed on Jan 27, 2025
  7. ryanofsky force-pushed on Jan 27, 2025
  8. ryanofsky commented at 7:01 pm on January 27, 2025: contributor
    Rebased eb70c90b06f068816ba09481f1cf37f49c4d95ee -> 9bb4bb9680b28b238ae5840f9de2963a81e01119 (pr/subtree.1 -> pr/subtree.2, compare) on updated base PR to fix CI errors Updated 9bb4bb9680b28b238ae5840f9de2963a81e01119 -> 93ba80b3d78fa1c4e7f614df6517b9cb492d05d7 (pr/subtree.2 -> pr/subtree.3, compare) to fix path typo in previous push
  9. ryanofsky force-pushed on Jan 27, 2025
  10. TheCharlatan commented at 7:33 pm on January 27, 2025: contributor
    Concept ACK
  11. DrahtBot removed the label CI failed on Jan 27, 2025
  12. theuni commented at 1:02 am on January 28, 2025: member

    @ryanofsky Please have a look here: https://github.com/theuni/bitcoin/commit/f2d74475433528c81c42af3232c257c443c1fc94

    It’s a quick hack to demonstrate the concept, please ignore the rough edges and consider the concept alone. I’ll clean it up if you’re interested.

    That approach would simplify things for (I suspect) 99% of builders, who never need to regenerate the files. Let me know what you think?

    Edit: Note that if you’re using depends, you’ll need to overwrite depends/$(host)/native/lib/cmake/Libmultiprocess/CopyOrGenerate.cmake with src/ipc/libmultiprocess/cmake/TargetCapnpSources.cmake. If we take this approach, that file would be updated upstream accordingly.

  13. in doc/multiprocess.md:32 in b8289e77aa outdated
    26@@ -25,9 +27,9 @@ build/src/bitcoin-node -regtest -printtoconsole -debug=ipc
    27 BITCOIND=$(pwd)/build/src/bitcoin-node build/test/functional/test_runner.py
    28 ```
    29 
    30-The `cmake` build will pick up settings and library locations from the depends directory, so there is no need to pass `-DWITH_MULTIPROCESS=ON` as a separate flag when using the depends system (it's controlled by the `MULTIPROCESS=1` option).
    31+The `cmake` build will pick up settings and library locations from the depends directory, so there is no need to pass `-DBUILD_MULTIPROCESS=ON` as a separate flag when using the depends system (it's controlled by the `MULTIPROCESS=1` option).
    32 
    33-Alternately, you can install [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) packages on your system, and just run `cmake -B build -DWITH_MULTIPROCESS=ON` without using the depends system. The `cmake` build will be able to locate the installed packages via [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/). See [Installation](https://github.com/chaincodelabs/libmultiprocess/blob/master/doc/install.md) section of the libmultiprocess readme for install steps. See [build-unix.md](build-unix.md) and [build-osx.md](build-osx.md) for information about installing dependencies in general.
    34+When not using depends, the [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) library can optionally be installed as an external dependency and enabled with the `-DWITH_LIBMULTIPROCESS=ON` option. If this is disabled, a local git submodule will be used instead. See [Installation](https://github.com/chaincodelabs/libmultiprocess/blob/master/doc/install.md) section of the libmultiprocess documentation for install steps. Installing libmultiprocess externally is only necessary when cross-compiling (to provide a code generation binary that can run on the native system). It can also be useful when developing changes meant to be submitted upstream.
    


    Sjors commented at 9:26 am on January 28, 2025:

    b8289e77aaa2478828caf7c91a59fc15c96fb187: “local git subtree”

    I would first explain the default behavior, and then point out that -DWITH_LIBMULTIPROCESS=ON can be used to use a locally installed library instead.


    ryanofsky commented at 1:54 pm on January 29, 2025:

    re: #31741 (review)

    Thanks I reworked this and adding headings to separate the cases. I think base case here is that capnproto is the only external dependency and you install it with package manager on your system. A second comon option may be to use depends, so that is described second. The third option of installing libmultiprocess externally is last because I was thinking that might be less commonly done, but I’m not sure.

  14. Sjors commented at 9:33 am on January 28, 2025: member

    Concept ACK

    It would be good to remind people in the PR description how one verifies a subtree: https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#subtrees (you should add ipc/multiprocess to the list there) and https://github.com/bitcoin/bitcoin/tree/master/test/lint#git-subtree-checksh

    TL&DR:

    0git remote add --fetch libmultiprocess https://github.com/chaincodelabs/libmultiprocess.git
    1test/lint/git-subtree-check.sh -r src/ipc/libmultiprocess
    

    It would be good to have an additional CI machine cover multiprocess, using the non-depends -DBUILD_MULTIPROCESS=1 and with BITCOIND=bitcoin-node. For easy rebasing in #30975, ci/test/00_setup_env_mac_native.sh would have my preference. @theuni I’m all for speeding up the build, though that should probably be a followup.

    On (M4) macOS 15.2 I get a flood of warnings, which means I can’t use -DWERROR=ON:

     0In file included from /opt/homebrew/include/capnp/common.h:31:
     1/opt/homebrew/include/kj/windows-sanity.h:39:6: warning: '_WIN32' is not defined, evaluates to 0 [-Wundef]
     2   39 | #if !_WIN32 && !__CYGWIN__
     3      |      ^
     4/opt/homebrew/include/kj/windows-sanity.h:39:17: warning: '__CYGWIN__' is not defined, evaluates to 0 [-Wundef]
     5   39 | #if !_WIN32 && !__CYGWIN__
     6      |     
     7
     8/opt/homebrew/include/capnp/common.h:33:5: warning: 'CAPNP_DEBUG_TYPES' is not defined, evaluates to 0 [-Wundef]
     9
    10/opt/homebrew/include/capnp/raw-schema.h:26:5: warning: '_MSC_VER' is not defined, evaluates to 0 [-Wundef]
    11
    12/opt/homebrew/include/kj/async-prelude.h:34:6: warning: 'KJ_NO_EXCEPTIONS' is not defined, evaluates to 0 [-Wundef]
    

    Building MULTIPROCESS=1 with depends now fails:

     0...
     1Postprocessing native_capnp...
     2Caching native_capnp...
     3Fetching from
     4curl: (3) URL rejected: No host part in the URL
     5Fetching from https://bitcoincore.org/depends-sources
     6  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
     7                                 Dload  Upload   Total   Spent    Left  Speed
     8  0  308k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
     9curl: (22) The requested URL returned error: 403
    10make: *** [/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 22
    

    Before commit 01452f8c0da58db549bc46d0cfa7de715344efc4 depends does build (as you would expect).

  15. vasild commented at 11:21 am on January 28, 2025: contributor
    Concept ACK
  16. ryanofsky commented at 4:19 pm on January 28, 2025: contributor

    re: #31741#pullrequestreview-2577588200

    Thanks @Sjors for the review. I’ll make changes you suggested and I opened two issues https://github.com/chaincodelabs/libmultiprocess/issues/138 and https://github.com/chaincodelabs/libmultiprocess/issues/139 to debug the problems you are seeing, and I will post next steps there. Thanks for reporting these.

  17. ryanofsky commented at 5:22 pm on January 28, 2025: contributor

    @ryanofsky Please have a look here: theuni@f2d7447

    It’s a quick hack to demonstrate the concept, please ignore the rough edges and consider the concept alone. I’ll clean it up if you’re interested.

    I feel like I’m missing a lot of context which led up to that implemention, and also I wonder if files are missing from that commit because just shows a new CopyOrGenerate.cmake script that does not actually seem to be called anywhere.

    From what I can understand though, this script seems to be acting like a wrapper around the mpgen executable that avoids calling mpgen if cached outputs are present. If they are present they will be copied from one location to another (unclear what source location is since it is not specified in the script).

    So I presume the idea behind this is that in cross-compiled builds, it will not be necessarily to have an mpgen utility built, because the mpgen outputs can be checked in to git, or downloaded from some external location, and they can be copied into the build instead of calling mpgen. And this will allow deleting the native_capnp.mk and libmultiprocess.mk packages, which I’m speculating is your ultimate goal? It is unclear what your goal is.

    I don’t have a problem with that approach, if that’s the approach you think is better. I would point out that it will have drawbacks. For one thing if the plan is to check these files into git then we will need some kind of CI rule to ensure that they are generated correctly and not just trust authors of PRs to generate them. The generated files will also be tied to a individual releases of capnproto so everybody will have to run the same capnproto version. Capnproto does not allow mixing headers generated by one version with the library of another version, and it has an explicit preprocessor check to prevent this. The generated files could also make PR review unwieldy, because a simple change to a capnp file could lead to much bigger changes in generated files (which contain binary data in C arrays and are not very friendly to diffing. Changes to generated files could be especially make PRs hard to review if they change .capnp files in multiple commits, and lead to headaches rebasing.

    If you want to go down this road it seems ok, but I think it would be very helpful if you could a little time to actually review this PR. I am happy with this PR currently and think the approach it takes is elegant. It leverages fantastic work @hebasto did in https://github.com/chaincodelabs/libmultiprocess/pull/96 and later PRs to simultaneously support cmake packaging while also simplifying the libmultiprocess build.

    There are two nontrivial commits in this PR. 01452f8c0da58db549bc46d0cfa7de715344efc4 adds an obvious feature to the depends system, that I’m surprised wasn’t in there from the beginning: the ability to build sources from a local directory instead of requiring them to be published to an http server. The feature is simple, ergonomic, clearly documented, and implemented in literally 12 lines of code. The second commit d70474da4d1f3bc623e3167dab63a7860f99875f is nontrivial but also pretty straightforward. It adds a add_subdirectory(libmultiprocess) call and updates a handful of other things in the cmake build to support that call.

    If I have one takeaway from implementing this PR, it is that find_package is great and add_subdirectory is terrible. add_subdirectoryis bad because it leaks all kinds of information you don’t want to leak from the parent directory to the child directory (include paths and other options) and vice versa (child targets leak into the parent build). find_package has standard scoping mechanisms and ways of passing information, while add_subdirectory is ad-hoc and requires PARENT_SCOPE assignments. So in terms of robustness and dealing with breakages in the future, I think keeping depends native_capnp.mk and libmultiprocess.mk packages, using find_package, and avoiding add_subdirectory when making releases and cross compiling will make debugging and support more straightforward. Then with find_package support working, additionally supporting add_subdirectory for local development is pretty easy and just requires a few extra lines of code (add_subdirectory call and some set PARENT_SCOPE lines).

  18. theuni commented at 7:04 pm on January 28, 2025: member

    @ryanofsky Please have a look here: theuni@f2d7447 It’s a quick hack to demonstrate the concept, please ignore the rough edges and consider the concept alone. I’ll clean it up if you’re interested.

    I feel like I’m missing a lot of context which led up to that implemention, and also I wonder if files are missing from that commit because just shows a new CopyOrGenerate.cmake script that does not actually seem to be called anywhere.

    Yes, sorry, I missed the chunk that actually turns it on. Pushed again, see https://github.com/theuni/bitcoin/commit/274cc5598db09af687e7383979b377460a59ef94.

    From what I can understand though, this script seems to be acting like a wrapper around the mpgen executable that avoids calling mpgen if cached outputs are present. If they are present they will be copied from one location to another (unclear what source location is since it is not specified in the script).

    Correct

    So I presume the idea behind this is that in cross-compiled builds, it will not be necessarily to have an mpgen utility built, because the mpgen outputs can be checked in to git, or downloaded from some external location, and they can be copied into the build instead of calling mpgen. And this will allow deleting the native_capnp.mk and libmultiprocess.mk packages, which I’m speculating is your ultimate goal? It is unclear what your goal is.

    Correct. Even if it’s not desirable to check these into master, it would be very useful to have them in release branches so that they end up in release tarballs. It’s very common for source packages to be pre-bootstrapped, and that’s exactly what this would do.

    Consider how much simpler this would be for builders/packagers (who don’t use depends) to build cross packages.

    I don’t have a problem with that approach, if that’s the approach you think is better. I would point out that it will have drawbacks. For one thing if the plan is to check these files into git then we will need some kind of CI rule to ensure that they are generated correctly and not just trust authors of PRs to generate them.

    Indeed.

    The generated files will also be tied to a individual releases of capnproto so everybody will have to run the same capnproto version. Capnproto does not allow mixing headers generated by one version with the library of another version, and it has an explicit preprocessor check to prevent this.

    I think we would want to enforce this anyway? One of the rules we decided on for CMake is “newer build-side tools should never lead to different/better code”. For example, A newer version of CMake should produce the same code as an older one. If newer versions of capnproto produce materially different code, I think we’d want force a pinning (for everyone) to whatever we’re using in guix, no?

    The generated files could also make PR review unwieldy, because a simple change to a capnp file could lead to much bigger changes in generated files (which contain binary data in C arrays and are not very friendly to diffing. Changes to generated files could be especially make PRs hard to review if they change .capnp files in multiple commits, and lead to headaches rebasing.

    Agreed, that does sound like it could be painful.

    If you want to go down this road it seems ok, but I think it would be very helpful if you could a little time to actually review this PR. I am happy with this PR currently and think the approach it takes is elegant. It leverages fantastic work @hebasto did in chaincodelabs/libmultiprocess#96 and later PRs to simultaneously support cmake packaging while also simplifying the libmultiprocess build.

    This is my process for review :) Please don’t think that just because I’m proposing a procedural change, that I’m somehow ignoring the work that has gone in. It also doesn’t mean that I plan on being pushy about this approach. This is a significant change for us and I want to make sure that I understand it and its drawbacks. Maybe the pre-bootstrapped-files idea is terrible, but I wanted to see what it would take to make it happen.

    To my knowledge, we’ve never (as part of our release builds) required any native tooling (beyond the toolchain) for pre-processing of our code. [edit: Er.. ok, that’s not true. QT requires pre-processing, but its native tools are widely available]. Adding to that, CMake doesn’t understand the concept of build/host split tools. So for something that is going to be a hard requirement for future builds, I’m doing my due diligence to understand how we can make this as painless for devs/packagers as possible.

    It’s easiest for me to review something like this by basically implementing a solution myself and seeing where we diverge. So far, I’m mostly arriving at the same conclusions as you. I wouldn’t have come up with the depends-source-tarball thing though. That one’s clever :)

    There are two nontrivial commits in this PR. 01452f8 adds an obvious feature to the depends system, that I’m surprised wasn’t in there from the beginning: the ability to build sources from a local directory instead of requiring them to be published to an http server. The feature is simple, ergonomic, clearly documented, and implemented in literally 12 lines of code. The second commit d70474d is nontrivial but also pretty straightforward. It adds a add_subdirectory(libmultiprocess) call and updates a handful of other things in the cmake build to support that call.

    I think the depends source-package thing is elegant and neat. But in the cross-compiled case, it effectively nullifies the effect of vendoring the code. Changes to the local libmultiprocess won’t be picked up without rebuilding depends. I think this could be improved by always using our internal target libmultiprocess, but selectively with an internel/external native one? In that case, I don’t think the non-native libmultiprocess in depends needs to exist? Maybe that’s nonsense… I’m still trying to understand how it all glues together, I’m nowhere near done yet. Again, hacking around is how I understand/review, please don’t take it as a criticism.

    If I have one takeaway from implementing this PR, it is that find_package is great and add_subdirectory is terrible. add_subdirectoryis bad because it leaks all kinds of information you don’t want to leak from the parent directory to the child directory (include paths and other options) and vice versa (child targets leak into the parent build). find_package has standard scoping mechanisms and ways of passing information, while add_subdirectory is ad-hoc and requires PARENT_SCOPE assignments. So in terms of robustness and dealing with breakages in the future, I think keeping depends native_capnp.mk and libmultiprocess.mk packages, using find_package, and avoiding add_subdirectory when making releases and cross compiling will make debugging and support more straightforward.

    This negates many of the benefits of vendoring, though. Namely hardening/optimizations. A libmultiprocess built in depends wouldn’t get the same build flags as it would if it were building from our project. So yes, while the leakiness of the subdirectory approach is annoying (see leveldb/crc32c for example, we completely ignore their CMake files), it also has its benefits.

  19. theuni commented at 7:13 pm on January 28, 2025: member

    I think the depends source-package thing is elegant and neat. But in the cross-compiled case, it effectively nullifies the effect of vendoring the code. Changes to the local libmultiprocess won’t be picked up without rebuilding depends. I think this could be improved by always using our internal target libmultiprocess, but selectively with an internel/external native one? In that case, I don’t think the non-native libmultiprocess in depends needs to exist?

    Basically (unless I’m missing something?) I think libmultiprocess needs an EXTERNAL_MPGEN option, same as capnproto has: https://github.com/capnproto/capnproto/blob/v2/c%2B%2B/CMakeLists.txt#L29. That would allow what I proposed above to work.

    I’m working on a POC for that now.

    Edit: wait, is mpgen not required to build libmultiprocess itself? See, this is a helpful exercise :)

  20. ryanofsky commented at 8:40 pm on January 28, 2025: contributor

    Thanks for the feedback and clearing things up. Will be interested to see the EXTERNAL_MPGEN option and maybe eliminate one of the depends packages.

    I think the depends source-package thing is elegant and neat. But in the cross-compiled case, it effectively nullifies the effect of vendoring the code. Changes to the local libmultiprocess won’t be picked up without rebuilding depends.

    I don’t think “effectively nullifies the effect of vendoring” is true at all, unless you have some different goals in mind that I’m not aware of. My main goal here is to make it trivial for developers to build multiprocess binaries by running “apt install libcapnp-dev capnprotoand-DBUILD_MULTIPROCESS=ON` and not having to do anything else. (At some point in a followup I would also like to change default to ON so even the latter is not necessary.) I also want the depends system to use the git subtree, so I am tweaking the depends system minimally to do that.

    The use-case you seem to be referencing is a more particular one where someone is using the depends system and also modifying the libmultiprocess library. The PR helps a lot with that use-case, and supports it by only requiring the developer to touch libmultiprocess directory and rerun their depends make command after changes to libmultiprocess code. This is much easier than what you have to do now, which is commit the changes to git, push them to github, update the version_hash and sha256_hash in the package file, and then run depends again.

    This build steps in this use-case are also basically the same steps I’ve been using to develop libmultiprocess up to this point for years now, where after I make a change, I run “make install” in the libmultiprocess build and then run “make” in the bitcoin build. I can tell you from experience it is not too onerous.

    I think this could be improved by always using our internal target libmultiprocess, but selectively with an internel/external native one?

    Yes we could support building with an internal libmultiprocess for the library an external libmultiprocess for the code generator. This is your EXTERNAL_MPGEN suggestion. It would just require defining a new option and adding a few if() statements to the build. It could help avoid the need to rerun depends in the use-case above if only the library not code generator is being modified. It would also allow bitcoin flags to automatically propagate to the libmultiprocess library.

    In that case, I don’t think the non-native libmultiprocess in depends needs to exist?

    That’s right. Overall EXTERNAL_MPGEN or an equivalent is a reasonable idea that may be worth doing as a followup. The implementation would be be a superset of this change, not replace or eliminate changes already here. One drawback of using this option and eliminating the cross-compiled libmultiprocess.mk package would be that the library and code generator would no longer be built together in lockstep. So you could make changes to the code generator and they might not be picked up while changes to the library are, which couldn’t happen with the current PR. I don’t think it is a big concern though.

    Maybe that’s nonsense… I’m still trying to understand how it all glues together, I’m nowhere near done yet. Again, hacking around is how I understand/review, please don’t take it as a criticism.

    No problem at all. An EXTERNAL_MPGEN is potentially useful idea I didn’t really think about. Just more of an add-on than a replacement for the changes here.

    This negates many of the benefits of vendoring, though. Namely hardening/optimizations. A libmultiprocess built in depends wouldn’t get the same build flags as it would if it were building from our project. So yes, while the leakiness of the subdirectory approach is annoying (see leveldb/crc32c for example, we completely ignore their CMake files), it also has its benefits.

    I can see this both ways. My inclination was that if you want flags to apply to a library you should just pass them to the library because, this broke the libmultiprocess build earlier and forced me to add a workaround. But I could imagine other cases where leaking flags are more helpful.

  21. ryanofsky commented at 9:27 pm on January 28, 2025: contributor

    Few other things I wanted to reply to.

    re: #31741 (comment)

    Correct. Even if it’s not desirable to check these into master, it would be very useful to have them in release branches so that they end up in release tarballs. It’s very common for source packages to be pre-bootstrapped, and that’s exactly what this would do.

    This is just a gut reaction, but as a long-time gentoo and nixos user, this sounds pretty bad to me. I’d want my package manager to build from human readable source code, not from pregenerated files that require a bespoke process to verify and won’t be verified in practice. Maybe human readable source isn’t enough either, and there are no good solutions, but if you are relying “pre-bootstrapped” files for convenience, it seems like you just asking for another xz backdoor to happen. (I don’t remember the details there but the wikipedia article says “A modified version of build-to-host.m4 was included in the release tar file uploaded on GitHub” which sounds exactly the the type of build cruft I would like my package manager to not trust and not use.)

    Consider how much simpler this would be for builders/packagers (who don’t use depends) to build cross packages.

    I don’t have much experience with cross compiling package managers other than working a little with nix, the depends system, and buildroot a long time ago, but I don’t think having pregenerated headers would simplify any of these systems.

    Generated code from protobufs, flex, bison, and many other tools are commonly used, it is common to not check generated sources into git repositories, which all of these systems support in some fashion. Even in our own depends system, running native packages is not anything special and at the time I started working with it there were actually 7 other native packages besides the two I added.

    I could be wrong on this point. Maybe there really are cross compiled packages that could be simplified by using pregenerated sources. But I’d want to see some actual example of that before accepting this was the case.

    Capnproto does not allow mixing headers generated by one version with the library of another version, and it has an explicit preprocessor check to prevent this.

    I think we would want to enforce this anyway?

    We might be saying different things here. I am saying if we check generated headers into the git development branch and they are used by default, then every revision in our git repository is hardcoded to a particular version of capnproto. You couldn’t for example check out code from 3 months ago while having capnproto 1.1 installed your system, you would have to downgrade to 1.02.

    One of the rules we decided on for CMake is “newer build-side tools should never lead to different/better code”. For example, A newer version of CMake should produce the same code as an older one. If newer versions of capnproto produce materially different code, I think we’d want force a pinning (for everyone) to whatever we’re using in guix, no?

    I can’t think of what benefits that may provide. Capnproto is very backwards compatible, it just doesn’t want you to use old generated headers and new library headers together or vice versa because the static code and generated code work together internally and there is not some public interface between them. If you just build from source, everything works perfectly well, but if you check pregenerated headers into git and try to use them you will have headaches. Maybe the headaches could be worth it. I don’t have the answer to that.

  22. in depends/funcs.mk:46 in 01452f8c0d outdated
    38@@ -39,8 +39,20 @@ define fetch_file
    39       $(call fetch_file_inner,$(1),$(FALLBACK_DOWNLOAD_PATH),$(3),$(4),$(5))))
    40 endef
    41 
    42+define int_get_local_dir
    43+$(eval $(1)_sha256_hash:=$(shell
    44+    if ! [ -f $($(1)_fetched) ] || [ $($(1)_fetched) -ot $($(1)_local_dir) ]; then \
    45+        mkdir -p $(dir $($(1)_fetched)); \
    46+        $(build_TAR) -c --sort=name --mtime=0 --owner=0 --group=0 --numeric-owner -f $($(1)_source) -C $($(1)_local_dir) . && \
    


    Sjors commented at 10:58 am on January 29, 2025:
    01452f8c0da58db549bc46d0cfa7de715344efc4: the Apple version of tar doesn’t have --sort, see https://github.com/chaincodelabs/libmultiprocess/issues/139#issuecomment-2621080394

    ryanofsky commented at 1:56 pm on January 29, 2025:

    re: #31741 (review)

    01452f8: the Apple version of tar doesn’t have --sort, see chaincodelabs/libmultiprocess#139 (comment)

    Thanks, this is dropped

  23. TheCharlatan commented at 12:07 pm on January 29, 2025: contributor
    As an alternative to reaching into depends or committing generated files, could it be feasible to set native compilers in the toolchain file instead for compiling the code generators?
  24. fanquake referenced this in commit ad2f9324c6 on Jan 29, 2025
  25. hebasto commented at 12:33 pm on January 29, 2025: member
    Concept ACK.
  26. ryanofsky force-pushed on Jan 29, 2025
  27. ryanofsky marked this as ready for review on Jan 29, 2025
  28. ryanofsky commented at 2:21 pm on January 29, 2025: contributor

    Rebased 93ba80b3d78fa1c4e7f614df6517b9cb492d05d7 -> ab041937289f4939758c1896f5cb57f556ae0615 (pr/subtree.3 -> pr/subtree.4, compare) after base pr was merged, also updated documentation as suggested, and fixed compatibility with gnu make 3.81 and bsd tar.


    re: #31741#pullrequestreview-2577588200

    It would be good to remind people in the PR description how one verifies a subtree

    Thanks! Added your notes about this.

    It would be good to have an additional CI machine cover multiprocess, using the non-depends -DBUILD_MULTIPROCESS=1 and with BITCOIND=bitcoin-node. For easy rebasing in #30975, ci/test/00_setup_env_mac_native.sh would have my preference.

    I feel like CI change should be a separate PR because there is already a lot to review and discuss here, but let me know if you think this would be important to include.

    The other followup change I hope we can make soon is turning the cmake option on by default

    0-option(BUILD_MULTIPROCESS "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    1+cmake_dependent_option(BUILD_MULTIPROCESS "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables." ON "NOT WIN32" OFF)
    

    re: #31741 (comment)

    As an alternative to reaching into depends or committing generated files, could it be feasible to set native compilers in the toolchain file instead for compiling the code generators?

    It might be technically possible. When I was looking to implement this I read a bunch of things about people using ExternalProject_Add to juggle multiple toolchains in cmake. But it looked very messy and seemed like cmake wasn’t designed to handle multiple toolchains. Luckily, we have system that is great at handling multiple toolchains called depends (and other package managers also support this). So with a simple, clean change to depends to support local directories, cross compiling the subtree is easy and works well.

    With some additional work implementing Cory’s EXTERNAL_MPGEN option, add_subdirectory could be used instead of depends to build the library (not the code generator). This way cmake could be used to control libmultiprocess compile flags even in depends builds, if we want that.

  29. theuni commented at 2:49 pm on January 29, 2025: member

    With some additional work implementing Cory’s EXTERNAL_MPGEN option, add_subdirectory could be used instead of depends to build the library (not the code generator). This way cmake could be used to control libmultiprocess compile flags even in depends builds, if we want that.

    I have this hacked up to work, though I’m still investigating some subdirectory leakage (ugh) that I don’t quite understand. I pretty strongly prefer this approach as it means everyone’s compiling in-tree. I’ll try to get a minimal POC pushed up today.

    As I mentioned above, that means we get all of our hardening/warning flags. My current hacked-up branch is full of warnings that would’ve gone unnoticed otherwise. They appear harmless, but still..

  30. Sjors commented at 3:04 pm on January 29, 2025: member

    I feel like CI change should be a separate PR because there is already a lot to review and discuss here, but let me know if you think this would be important to include.

    I’ll be testing this with #30975 so we’ll get some CI runs regardless.

  31. ryanofsky commented at 3:07 pm on January 29, 2025: contributor
    re: #31741 (comment) @theuni can the change your are working on be a followup? It doesn’t seem like this change will have any effect for developers unless they are using depends for development which I think few do normally. And even then the main effect is that the library is built with cmake flags instead of depends flags. The change would also increase complexity (would only increase not decrease changes in this PR) and would be easier to review and discuss separately.
  32. Sjors commented at 3:26 pm on January 29, 2025: member

    I’m now able to build depends with make MULTIPROCES=1 on macOS without using the gnu versions.

    However when using the result cmake complains about OpenSSL missing:

     0% cmake -B build --toolchain /Users/sjors/dev/bitcoin/depends/aarch64-apple-darwin24.3.0/toolchain.cmake 
     1-- The CXX compiler identification is AppleClang 16.0.0.16000026
     2-- Detecting CXX compiler ABI info
     3-- Detecting CXX compiler ABI info - done
     4-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ - skipped
     5-- Detecting CXX compile features
     6-- Detecting CXX compile features - done
     7CMake Error at /opt/homebrew/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
     8  Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
     9  system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
    10  OPENSSL_INCLUDE_DIR Crypto SSL)
    11Call Stack (most recent call first):
    12  /opt/homebrew/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
    13  /opt/homebrew/share/cmake/Modules/FindOpenSSL.cmake:691 (find_package_handle_standard_args)
    14  /opt/homebrew/share/cmake/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
    15  depends/aarch64-apple-darwin24.3.0/lib/cmake/CapnProto/CapnProtoConfig.cmake:85 (find_dependency)
    16  /opt/homebrew/share/cmake/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
    17  depends/aarch64-apple-darwin24.3.0/lib/cmake/Libmultiprocess/LibmultiprocessConfig.cmake:41 (find_dependency)
    18  CMakeLists.txt:147 (find_package)
    19
    20
    21-- Configuring incomplete, errors occurred!
    

    It does work when I build depends with GNU Make 4.4.1 (via homebrew), so there’s some other subtle detail left to chase…

    Verbose log with Apple GNU Make 3.81: https://gist.githubusercontent.com/Sjors/29733993ac2ef470611911d69f842095/raw/7a8d653562e34f90af0285ec858d028e1f479b1c/gistfile1.txt

    And cmake –trace: https://gist.github.com/Sjors/e9c7e7e45e0bb2a627c9c52fc4a9caf8

  33. BrandonOdiwuor commented at 4:07 pm on January 29, 2025: contributor
    Concept ACK
  34. ryanofsky commented at 4:23 pm on January 29, 2025: contributor

    However when using the result cmake complains about OpenSSL missing:

    Openssh issue you are seeing is puzzling because I don’t know what change here could cause it, and I wonder if you see the same error without this PR? When you are building with depends capnproto is built without ssl support should should not be looking for ssl at all.

    If you are not seeing this issue happen before this PR, but are seeing this issue happen after this PR, I think it would probably most likely be the changes in commit 36db25f0bcaab6af9358faaad9214d5b77a71cb2 and you could try to revert that commit. I don’t think any of the other commits could cause it because they don’t touch the cmake build.

  35. Sjors commented at 4:46 pm on January 29, 2025: member

    wonder if you see the same error without this PR?

    I tried with master @ ad2f9324c619ae608c434ee85ad9ae1b2812877d and indeed got the same error. So it seems unrelated.

  36. Sjors commented at 8:11 am on January 30, 2025: member

    I found several (build) issues while unleashing the CI on these changes: #30975 (comment)

    Let me know which ones you want to address here, and which can be dealt with later.

  37. Sjors commented at 11:50 am on January 30, 2025: member

    If I cherry-pick #31763 on top of this, I’m unable to make a Guix build with multiprocess, at least not with the hosts I tried (x86_64-linux-gnu, aarch64-linux-gnu and arm64-apple-darwin).

     0% DEP_OPTS="MULTIPROCESS=1" HOSTS="x86_64-linux-gnu" contrib/guix/guix-build
     1Checking that we can connect to the guix-daemon...
     2
     3Hint: If this hangs, you may want to try turning your guix-daemon off and on
     4      again.
     5
     6make: Entering directory '/home/sjors/bitcoin/depends'
     7make[1]: Entering directory '/home/sjors/bitcoin/depends'
     8Checksum missing or mismatched for native_libmultiprocess source. Forcing re-download.
     9Fetching from
    10curl: (3) URL rejected: No host part in the URL
    11Fetching from https://bitcoincore.org/depends-sources
    12  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
    13                                 Dload  Upload   Total   Spent    Left  Speed
    14  0  308k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    15curl: (22) The requested URL returned error: 403
    16make[1]: *** [funcs.mk:320: /home/sjors/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 22
    17make[1]: Leaving directory '/home/sjors/bitcoin/depends'
    18make: *** [Makefile:282: download-linux] Error 2
    19make: Leaving directory '/home/sjors/bitcoin/depends'
    

    (it does work on master)

  38. Sjors commented at 4:47 pm on January 31, 2025: member

    #30975 now passes, using a bunch of patches that will make their way here.

    One of those patches fixes the above guix build issue too (no, it doesn’t, will try again later).

  39. theuni commented at 5:50 pm on January 31, 2025: member

    @theuni can the change your are working on be a followup? It doesn’t seem like this change will have any effect for developers unless they are using depends for development which I think few do normally. And even then the main effect is that the library is built with cmake flags instead of depends flags. The change would also increase complexity (would only increase not decrease changes in this PR) and would be easier to review and discuss separately.

    Sorry, I just don’t think this makes sense. The PR here is to include a vendored multiprocess, but it includes a circuitous and unnecessary way of building it for cross/depends that is unnecessary after a few upstream changes. https://github.com/chaincodelabs/libmultiprocess/pull/142 and https://github.com/chaincodelabs/libmultiprocess/pull/140 are enough to enable a much simpler and unified approach for us.

    Devs use depends (I use nothing BUT depends), ci uses depends, and guix uses depends. So I don’t know why we’d work on an approach here that we plan to change significantly later, as opposed to just fixing it up and using the simpler approach to begin with.

  40. Sjors commented at 6:18 pm on January 31, 2025: member

    enable a much simpler and unified approach

    This sounds appealing…

  41. ryanofsky commented at 6:21 pm on January 31, 2025: contributor

    So I don’t know why we’d work on an approach here that we plan to change significantly later, as opposed to just fixing it up and using the simpler approach to begin with.

    I might be misunderstanding but from everything I’ve seen, the change you are implementing is a superset of this change. That is why I am suggesting it could be a followup. No part of this change would be reverted or simplified by the additional work you are doing so it is a natural follow-up.

    I don’t think devs are typically using depends when building locally, but if they are, this PR supports that perfectly well. The only significant differences with your additional change is that flags will be shared between the Bitcoin and libmultiprocess cmake builds, instead of being specified separately for each build. But separate flags are perfectly fine (obviously) because the flags have been separate since the libmultiprocess was originally added to depends in 2019. Again, as I understand it, your change is a superset of my change. It is builds on top of this change. It does not undo things being done in this change.

    Specifically, my changes (f23c3c47ee4db6fb0ee6a4b4378a2734fd95fb03, f23c3c47ee4db6fb0ee6a4b4378a2734fd95fb03, 19eef973cb4ae37d202c9f584b6ed913482aab4a, 36db25f0bcaab6af9358faaad9214d5b77a71cb2, ab041937289f4939758c1896f5cb57f556ae0615) are all pretty small. If your changes are also small, I am happy to add them here when they are ready. Otherwise, I think they should be a followup. Either way they definitely could be a followup.

  42. ryanofsky commented at 6:29 pm on January 31, 2025: contributor

    enable a much simpler and unified approach

    This sounds appealing…

    To be sure, it is good change, and should be adopted. But unless you are using depends locally it will not affect you at all, and even if you are using depends locally it’s just gives some minor quality-of-life improvements, such as you only have to run one make command instead of two if you modify the libmultiprocess subtree locally, and you only have to change flags one place instead of two places if you are changing flags and care whether the flags are apply to libmultiprocess runtime library.

    I don’t these things are worth holding up this PR significantly or making this PR significantly more complicated, but assuming the proposed changes work well and are reasonably small, we should totally add them here.

  43. theuni commented at 6:30 pm on January 31, 2025: member

    I think we do have some miscommunication going on, because yes, I don’t think we’re understanding each other.

    I’m suggesting that the WITH_LIBMULTIPROCESS option can go away entirely. That means that there are no longer 2 ways to build the multiprocess code. Meaning that everyone gets the same build, same flags, etc. No need to pass tsan for tsan builds, for example. It will just work.

    The only significant differences with your additional change is that flags will be shared between the Bitcoin and libmultiprocess cmake builds, instead of being specified separately for each build. But separate flags are perfectly fine (obviously) because the flags have been separate since the libmultiprocess was originally added to depends in 2019.

    This is the point of vendoring the code heh. I don’t want to review what tidy/ci thinks of this until I’m comfortable knowing that we’re all (including CI and guix) building the same way.

    Again, as I understand it, your change is a superset of my change. It is builds on top of this change. It does not undo things being done in this change.

    Again, I’m suggesting that we would get rid of one of the weird build mechanisms here.. never add it in the first place because it’s unnecessary. I don’t see how that could be described as a superset, so I think we must be talking past each other :(

  44. theuni commented at 6:41 pm on January 31, 2025: member

    To be sure, it is good change, and should be adopted. But unless you are using depends locally it will not affect you at all, and even if you are using depends locally it’s just gives some minor quality-of-life improvements, such as you only have to run one make command instead of two if you modify the libmultiprocess subtree locally, and you only have to change flags one place instead of two places if you are changing flags and care whether the flags are apply to libmultiprocess runtime library.

    Among other things, libmultiprocess as-built here is not hardened. When we (the build nerds) asked to have it vendored, we’re asking for it to get the same treatment as the rest of our code. This PR was opened to do the logistical code movement, but it doesn’t demonstrate the benefits of living within our source tree. And further, it gets built differently depending on whether you’re a dev doing local hacking, or ci, or doing a release.

    So I’m sorry, but I don’t want to spend time reviewing that approach. Especially when it’s so easily avoidable. I don’t consider this to be in a state to be merged until those benefits can be demonstrated. I’m really trying to help with that though! :)

    Edit: See https://github.com/chaincodelabs/libmultiprocess/pull/143 as an example of a benefit of building like the rest of our code. In that case it’s dumb and trivial, but we’re likely to find (or guard against) actual issues by building this way across the board.

  45. ryanofsky commented at 7:27 pm on January 31, 2025: contributor

    I’m suggesting that the WITH_LIBMULTIPROCESS option can go away entirely.

    If you look at the PR, you can see that WITH_LIBMULTIPROCESS option adds 5 lines of code and 2 of them are endif(). So I wouldn’t consider that a real coup. Also I don’t think WITH_LIBMULTIPROCESS should be dropped just because depends will not use it, because having it makes it easier to develop and submit changes to the upstream repository, and makes sense to support other packaging systems (see https://github.com/NixOS/nixpkgs/issues/359902) that are more flexible and less monolithic than the depends system.

    This is the point of vendoring the code heh.

    No offense (really), but this strikes me as a crazy thing to say. I didn’t open this PR and spend the last entire week fixing CI bugs because I was concerned setting compiler flags in the Bitcoin cmake build didn’t automatically set the same flags in the libmultiprocess cmake build. Libmultiprocess has been using cmake since 2019, bitcoin just started using cmake last year, so the approach you are favoring literally only became possible a few months ago. I understand sharing compiler flags is a very important thing to you, and I think it is a nice idea, and I am happy to review or accept additional changes in this PR to support it. But libmultiprocess has had separate compiler flags for as long as it existed and since it was added as a depends package in 2019.

    I can’t remember it ever happening to me, but probably, sometime over the last 5 years I did have to set some compiler flags in two builds instead of one. So it’s great I may no longer have to do this. But this is a small quality of life issue in a corner case that will never affect developers unless they are trying to build in an unusual way, and requires a fairly obvious extra step if they are. It is a strange thing to fixate on. But again, I think it’s great to share flags and would like to do that.

    The actual point of vendoring this code as I wrote earlier in #31741 (comment) is:

    My main goal here is to make it trivial for developers to build multiprocess binaries by running “apt install libcapnp-dev capnprotoand-DBUILD_MULTIPROCESS=ON` and not having to do anything else. (At some point in a followup I would also like to change default to ON so even the latter is not necessary.) I also want the depends system to use the git subtree, so I am tweaking the depends system minimally to do that.

    That is why this PR exists. That is why I’ve been working on it the past week.

    re: #31741 (comment)

    When we (the build nerds) asked to have it vendored, we’re asking for it to get the same treatment as the rest of our code.

    The phrasing here is ambiguous, but if you just wanted libmultiprocess to be compiled with the same flags, we could pass the same flags to depends, or specify the same flags in depends. Either of these approaches would be easier than adding a subtree.

    #31741 (comment)

    Again, I’m suggesting that we would get rid of one of the weird build mechanisms here.. never add it in the first place because it’s unnecessary.

    Which one? We are definitely talking past each each other. I think the problem might be (1) your changes might be incomplete, so you think there are parts of this PR that are unnecessary when they are not (2) you haven’t reviewed this PR so you think it is doing unnecessary things it is not doing or (3) which is very possible, that this PR is actually doing something unnecessary and your changes will simplify it.

    In any case, I am looking forward to see your changes, and appreciate all the feedback and improvements you’ve made so far, which have been very helpful.

  46. theuni commented at 8:48 pm on January 31, 2025: member

    Also I don’t think WITH_LIBMULTIPROCESS should be dropped just because depends will not use it, because having it makes it easier to develop and submit changes to the upstream repository, and makes sense to support other packaging systems (see NixOS/nixpkgs#359902) that are more flexible and less monolithic than the depends system.

    Perhaps this is the source of our breakdown. I would consider this an anti-feature. Or AT LEAST, as you’ve been saying, one that can be added on top of the changes here once it’s been discussed. We don’t provide this option for secp or leveldb or crc32 or univalue or minisketch.

    Anyway, I’ve pushed a demo of my changes here: https://github.com/theuni/bitcoin/commits/libmultiprocess-vendor/

    The internal libmultiprocess is now the only the option, and the non-native version has been removed from depends.

    It contains https://github.com/chaincodelabs/libmultiprocess/pull/141, https://github.com/chaincodelabs/libmultiprocess/pull/142, and https://github.com/chaincodelabs/libmultiprocess/pull/143. I understand that 141 probably has a better solution and 142 hasn’t been reviewed or acked yet.

  47. in depends/packages/native_libmultiprocess.mk:5 in 19eef973cb outdated
    0@@ -1,8 +1,5 @@
    1 package=native_libmultiprocess
    2-$(package)_version=07c917f7ca910d66abc6d3873162fc9061704074
    3-$(package)_download_path=https://github.com/chaincodelabs/libmultiprocess/archive
    4-$(package)_file_name=$($(package)_version).tar.gz
    5-$(package)_sha256_hash=ac9db311e3b22aac3c7b7b7b3f6b7fee5cf3043ebb3c3bf412049e8b17166de8
    


    theuni commented at 9:03 pm on January 31, 2025:
    This unfortunately leaves us open to a supply-chain attack as it depends on the contents of the builders’s environment. We’ll need to commit to the hash of a deterministic tarball.

    ryanofsky commented at 9:14 pm on January 31, 2025:
    I am not following. Can you describe the concern?

    ryanofsky commented at 9:43 pm on January 31, 2025:

    I am not following. Can you describe the concern?

    I would think if anything we are more vulnerable to supply-chain attack without this change, because, for example if a few bitcoin core developers conspired and made a version of libmultiprocess with a backdoor and pushed it to github, it wouldn’t be obvious to other developers that the hashes pointed to a backdoor version. It also wouldn’t be obvious to external observers diffing one version of bitcoin to the next to check for introduction of malicious code.

    By contrast, with this change depends system would no longer be downloading a second copy of sources already sit in the local git repository. With this change, in order to introduce a backdoor into libmultiprocess, that backdoor would have be visible in src/ipc/libmultiprocess, show up in normal diffs, and sitting in plain sight.


    theuni commented at 10:26 pm on January 31, 2025:

    Ok, you’re right. I should not have framed this as a potential attack.

    We maintain hard-coded hashes in depends for 2 reasons: To protect against real supply-chain attacks (Someone hijacks github and replaces tarballs with malicious ones) and for determinism (our sources can be described exactly).

    The output of a depends build is locally deterministic based on the build id, which is comprised of the environment, toolchain, and source hashes. After this change, as far as I can tell, that’s no longer true because it also depends on the contents of your hard drive, which may have been modified after checkout.

    In order to maintain the determinism guarantee, we’d need to pin the tarball to an agreed-up hash, same as our other sources.

    So if I’ve been hacking in my local libmultiprocess dir (as I have today), and I do a depends build, it should fail and yell at me that the sources don’t match the agreed-upon hash.


    ryanofsky commented at 6:28 pm on February 3, 2025:

    re: #31741 (review)

    for determinism (our sources can be described exactly).

    The sources are described by the git hash. The tarball could be generated deterministically too, but should be ok to just use default tar options. If during a deterministic build, you put a directory into a intermediate tarball and extract it and compile it, whether the intermediate tarball was deterministic should not effect whether build outputs are deterministic.

    So if I’ve been hacking in my local libmultiprocess dir (as I have today), and I do a depends build, it should fail and yell at me that the sources don’t match the agreed-upon hash.

    Current version now refreshes the tarball in this case. The previous caching behavior was documented and consistent with how URL sources were treated. If a URL contents change after a URL source downloaded, you don’t yelled at in that case either. But refreshing the tarball is more developer-friendly, so it’s good to implement that change.


    vasild commented at 8:16 am on February 5, 2025:

    The output of a depends build is locally deterministic based on the build id, which is comprised of the environment, toolchain, and source hashes. After this change, as far as I can tell, that’s no longer true because it also depends on the contents of your hard drive, which may have been modified after checkout.

    Even without this PR the output of depends build could change based on the contents of the hard drive, no? E.g. you can checkout github/bitcoin/bitcoin, then modify something in src/ and do the depends build.

    Isn’t the hash of an external tarball more or less equivalent in this aspect to the git commit hash? In both cases the sources can be modified after the tarball checksum is verified and extracted or after git clone/checkout.

    So if I’ve been hacking in my local libmultiprocess dir (as I have today), and I do a depends build, it should fail and yell at me that the sources don’t match the agreed-upon hash.

    Doesn’t that apply to any source as well, not only libmultiprocess dir? Should depends build yell if the tree has local mods?

  48. ryanofsky commented at 9:13 pm on January 31, 2025: contributor

    I would consider this an anti-feature. Or AT LEAST, as you’ve been saying, one that can be added on top of the changes here once it’s been discussed.

    Well, if it is an anti-feature, it is a 5 line anti-feature and I am happy to drop it here. If I do this though I will probably make a separate PR to add it later, because I think it will be painful to develop and test upstream libmultiprocess changes without it.

    Anyway, I’ve pushed a demo of my changes here: https://github.com/theuni/bitcoin/commits/libmultiprocess-vendor/

    Thanks, will take a look and incorporate it here.

  49. theuni commented at 9:32 pm on January 31, 2025: member

    @ryanofsky Thanks.

    I think we might just have been disagreeing on the importance of a unified build for everyone for something that will presumably (in the future) become mission-critical. I can see how from your POV it might seems like I’m just being persnickety about flags, but to me it’s about making sure we’re being clear (internally and externally) about ownership/canon/etc… the flags were just illustrative (to me) of the lack of clarity there.

    Thanks for your patience with me on this :)

  50. ryanofsky commented at 9:53 pm on January 31, 2025: contributor

    disagreeing on the importance of a unified build for everyone for something that will presumably (in the future) become mission-critical.

    I really don’t think this is true. I think you have made number a lot of claims that sound great, don’t stand up to scrutiny. I probably have too. These issues can be hard to reason about, so it’s good that we are going through this exercise.

    But I don’t take security or hardening less seriously than anyone else, and I believe this approach has been 100% compatible with both.

    I was balking at the idea that the main reason for adding a subtree would to be add some hardening flags. There are much easier ways to add hardening flags, and much more important benefits to having a subtree.

  51. depends: Update libmultiprocess library before converting to subtree
    Bring in a few cmake changes to improve cmake support
    
    https://github.com/chaincodelabs/libmultiprocess/pull/140 build: don't clobber user/superproject c++ version
    https://github.com/chaincodelabs/libmultiprocess/pull/142 build: add option for external mpgen binary
    https://github.com/chaincodelabs/libmultiprocess/pull/143 cleanup: initialize vars in the EventLoop constructor in the correct order
    https://github.com/chaincodelabs/libmultiprocess/pull/146 cmake: Suppress compiler warnings from capnproto headers
    https://github.com/chaincodelabs/libmultiprocess/pull/147 cmake: EXTERNAL_MPGEN cleanups
    https://github.com/chaincodelabs/libmultiprocess/pull/148 util: fix -Wpessimizing-move warning
    https://github.com/chaincodelabs/libmultiprocess/pull/145 CTest: Module must be included at the top level
    https://github.com/chaincodelabs/libmultiprocess/pull/149 Avoid `-Wundef` compiler warnings
    https://github.com/chaincodelabs/libmultiprocess/pull/152 refactor: Fix compiler and clang-tidy warnings
    1d75538a32
  52. ryanofsky force-pushed on Feb 3, 2025
  53. ryanofsky commented at 6:48 pm on February 3, 2025: contributor
    Rebased ab041937289f4939758c1896f5cb57f556ae0615 -> 1c10bfc167b0f3c17780b2ae3747999ee56c4e1c (pr/subtree.4 -> pr/subtree.5, compare) incorporating changes from https://github.com/theuni/bitcoin/commits/libmultiprocess-vendor/ and making various build fixes to let multiprocess CI jobs succeed in PR #30975 when libmultiprocess is built as a cmake subdirectory instead of as a cmake module.
  54. DrahtBot added the label CI failed on Feb 3, 2025
  55. DrahtBot commented at 7:51 pm on February 3, 2025: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/36604138479

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  56. ryanofsky force-pushed on Feb 3, 2025
  57. ryanofsky commented at 9:18 pm on February 3, 2025: contributor
    Updated 1c10bfc167b0f3c17780b2ae3747999ee56c4e1c -> e4289ddc3cf361cfc2ec8c5c64b96e57950d17ab (pr/subtree.5 -> pr/subtree.6, compare) to fix ci link error https://cirrus-ci.com/task/6281383380779008 caused by applying core_interface debug flags to multiprocess target but not mptest target depending on it Rebased e4289ddc3cf361cfc2ec8c5c64b96e57950d17ab -> b4d397903d652bae1497140775591f755d857e65 (pr/subtree.6 -> pr/subtree.7, compare) to fix more errors from core_interface
  58. ryanofsky force-pushed on Feb 4, 2025
  59. DrahtBot removed the label CI failed on Feb 4, 2025
  60. Sjors commented at 10:05 am on February 4, 2025: member

    I’m unable to make a Guix build. I’m using a branch sjors/2025/02/ipc-guix which enables multiprocess in the simplest possible way, without modifying the depends system, by setting MULTIPROCESS=1 in the pre-download and depends make step.

    It downloads and builds capnp as expected, but in the build says:

     0make: Entering directory '/bitcoin/depends'
     1Checksum missing or mismatched for native_libmultiprocess source. Forcing re-download.
     2...
     3Extracting native_capnp..
     4...
     5Caching native_capnp...
     6Fetching from
     7/gnu/store/7k364lx15sq567ivd79pp9dmnallcxz1-bash-minimal-5.1.16/bin/sh: line 1: curl: command not found
     8Fetching from https://bitcoincore.org/depends-sources
     9/gnu/store/7k364lx15sq567ivd79pp9dmnallcxz1-bash-minimal-5.1.16/bin/sh: line 1: curl: command not found
    10make: *** [funcs.mk:324: /bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 127
    

    I can add more debugging if you can’t reproduce.

  61. ryanofsky commented at 1:11 pm on February 4, 2025: contributor

    re: #31741 (comment)

    I’m unable to make a Guix build.

    Thanks, created https://github.com/chaincodelabs/libmultiprocess/issues/150 to track this

  62. luke-jr commented at 3:57 am on February 5, 2025: member
    Concept NACK: there’s no rationale for why this can’t always be external.
  63. ryanofsky commented at 4:50 am on February 5, 2025: contributor

    Concept NACK: there’s no rationale for why this can’t always be external.

    FWIW, I agree with you and want this to always be external. I personally find it more convenient to work on the library in its standalone git repository, and think cmake’s find_package mechanism is a cleaner and easier to maintain form of integration than the add_subdirectory trick used in this PR. So this PR intentionally does not remove any ability to use the library externally, and it makes git subtree a pure mirror of the external repository with no internal changes.

    The point of this PR, at least for me, is not to change anything about releases or packaging. It is purely to (1) make IPC features easier for developers to test, so they don’t have to install a local dependency, and to (2) get more eyes on the code, so when changes are merged in they look like normal diffs and not like checksum updates. I think both of these things are real pure benefits and should not have any downsides. Practically speaking this PR just makes it so if developer wants to enable IPC features all they have to do is toggle the ENABLE_IPC cmake option without downloading or installing anything separately.

    The only possible downside of this PR is the last commit which does change the depends release build to integrate libmultiprocess differently than before. I feel like it makes the build more fragile and could be a maintenance burden, but theuni thought it was important to make sure libmultiprocess code was automatically compiled with same flags and options as Bitcoin core code. So last commit also seems reasonable to me, even if it wasn’t my initial preference.

  64. Squashed 'src/ipc/libmultiprocess/' content from commit 9558ceb0d47a
    git-subtree-dir: src/ipc/libmultiprocess
    git-subtree-split: 9558ceb0d47ac1f62f88d29ec55f8f3099e32b20
    6342bb4652
  65. Merge commit '6342bb4652174c6c921eae8dbd992d32407e89fb' as 'src/ipc/libmultiprocess' 79b528b576
  66. scripted-diff: s/WITH_MULTIPROCESS/ENABLE_IPC/ in cmake
    Rename WITH_MULTIPROCESS to ENABLE_IPC, because ENABLE_IPC is a more accurate
    name for the feature. It controls whether the src/ipc/ directory is built and
    whether IPC features like -ipcbind, -ipcconnect, and -ipcfd are available. It
    does NOT currently enable multiprocess features which are implemented in #10102
    building on top of the IPC features. It will also no longer (as of the next
    commit), control whether a find_package is call is made so the "WITH_" prefix
    is also inappropriate.
    
    -BEGIN VERIFY SCRIPT-
    git grep -l WITH_MULTIPROCESS | xargs sed -i s/WITH_MULTIPROCESS/ENABLE_IPC/g
    -END VERIFY SCRIPT-
    b75e13e6ec
  67. cmake: Support building with libmultiprocess subtree
    When ENABLE_IPC option is on, build with libmultiprocess subtree and
    `add_subdirectory(src/ipc/libmultiprocess)` instead of external package
    and `find_package(Libmultiprocess)` by default.
    
    Behavior can be toggled with `WITH_LIBMULTIPROCESS` option. Using a subtree
    should be more convenient for most bitcoin developers, but using an external
    package is more convenient for developing in the libmultiprocess repository.
    
    The `WITH_LIBMULTIPROCESS` option is also used to avoid needing to changing the
    depends build here. But in later commits, the depends build is switched to use
    the add_subdirectory build as well.
    
    Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
    9f33034a6f
  68. cmake: Fix ctest mptest "Unable to find executable" errors
    This change is technically not needed to add libmultiprocess as a subtree, but
    it avoids a CI failure in followup PR #30975 which enables multiprocess build
    option in more CI jobs. In that PR, several jobs fail due to the mptest
    executable not being built by default, as reported
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2623801480
    710a0156e5
  69. cmake: Fix warnings from boost headers
    This change is technically not needed to add libmultiprocess as a subtree, but
    it avoids a CI failure in followup PR #30975 which enables multiprocess build
    option in more CI jobs. In that PR, the "macOS 14 native no depends job" fails
    due to warnings in boost headers treated as errors, reported in
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2623801480 and
    https://github.com/chaincodelabs/libmultiprocess/issues/138
    5b73974a9a
  70. cmake: Fix clang-tidy "no input files" errors
    This change is technically not needed to add libmultiprocess as a subtree, but
    it avoids a CI failure in followup PR #30975 which enables multiprocess build
    option in more CI jobs. In that PR, clang-tidy job fails due to missing
    generated example files as reported
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2627116832
    
    Different fixes were suggested
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2627152623 and
    https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2627403382
    
    Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
    8da8b1f210
  71. doc: Update documentation to explain libmultiprocess subtree
    Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
    77a18d40b1
  72. lint: Add exclusions for libmultiprocess subtree
    Without this change linter produces errors about:
    
    - Use of std::filesystem the libmultiprocess example program.
    - Use of locale-dependent functions in example program, in the build time code
      generator, and in the runtime library for debug logging.
    - Include guards not beginning with BITCOIN_
    17329d17ef
  73. depends, moveonly: split up int_get_build_id function
    Move parts of the int_get_build_id into a new int_get_build_properties
    function. There is no change in behavior. This just organizes assignments
    better so some build properties can be used to help compute build ids in the
    next commit.
    ac2a018027
  74. depends: Switch libmultiprocess packages to use local git subtree
    With newly introduced libmultiprocess subtree, there's no need for depends
    system to download and track changes to the upstream repository.
    
    Note that adding the libmultiprocess subtree does not allow dropping
    libmultiprocess packages from the depends build, because libmultiprocess
    includes a code generation tool called mpgen, and in cross-compiled builds,
    bitcoin core's cmake build system doesn't have access to a native toolchain and
    can't build mpgen itself, so the depends system (or the native environment if
    not using depends) needs to supply it.
    97058b53e8
  75. depends: remove non-native libmultiprocess build
    Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
    99b7b857d3
  76. ryanofsky force-pushed on Feb 5, 2025
  77. ryanofsky commented at 5:20 am on February 5, 2025: contributor
    Rebased b4d397903d652bae1497140775591f755d857e65 -> 99b7b857d3b8b9ce9bd7501e2480583369740c55 (pr/subtree.7 -> pr/subtree.8, compare) with some more fixes for downstream PR #30975 needed after switching depends build from find_package to add_subdirectory. Also includes a fix for guix multiprocess build issue https://github.com/chaincodelabs/libmultiprocess/issues/150
  78. vasild commented at 7:29 am on February 5, 2025: contributor
    @luke-jr, some pros of this PR in @ryanofsky’s comment above. What do you think are the cons?
  79. in CMakeLists.txt:143 in 99b7b857d3
    140@@ -141,8 +141,9 @@ endif()
    141 
    142 cmake_dependent_option(WITH_DBUS "Enable DBus support." ON "CMAKE_SYSTEM_NAME STREQUAL \"Linux\" AND BUILD_GUI" OFF)
    143 
    


    vasild commented at 8:53 am on February 5, 2025:

    pico nit: in the commit message of b75e13e6ecce1b2b406ef83099200df42acf37e0 scripted-diff: s/WITH_MULTIPROCESS/ENABLE_IPC/ in cmake:

    0- a find_package is call is made
    1+ a find_package call is made
    
  80. in CMakeLists.txt:145 in 99b7b857d3
    140@@ -141,8 +141,9 @@ endif()
    141 
    142 cmake_dependent_option(WITH_DBUS "Enable DBus support." ON "CMAKE_SYSTEM_NAME STREQUAL \"Linux\" AND BUILD_GUI" OFF)
    143 
    144-option(WITH_MULTIPROCESS "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    145-if(WITH_MULTIPROCESS)
    146+option(ENABLE_IPC "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    147+option(WITH_LIBMULTIPROCESS "Build with external libmultiprocess library instead of with local git submodule when ENABLE_IPC is enabled. This is not normally recommended, but can be useful when cross-compiling or making changes to the upstream project." OFF)
    


    vasild commented at 8:56 am on February 5, 2025:
    s/git submodule/git subtree/
  81. in CMakeLists.txt:146 in 99b7b857d3
    140@@ -141,8 +141,9 @@ endif()
    141 
    142 cmake_dependent_option(WITH_DBUS "Enable DBus support." ON "CMAKE_SYSTEM_NAME STREQUAL \"Linux\" AND BUILD_GUI" OFF)
    143 
    144-option(WITH_MULTIPROCESS "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    145-if(WITH_MULTIPROCESS)
    146+option(ENABLE_IPC "Build multiprocess bitcoin-node and bitcoin-gui executables in addition to monolithic bitcoind and bitcoin-qt executables. Requires libmultiprocess library. Experimental." OFF)
    147+option(WITH_LIBMULTIPROCESS "Build with external libmultiprocess library instead of with local git submodule when ENABLE_IPC is enabled. This is not normally recommended, but can be useful when cross-compiling or making changes to the upstream project." OFF)
    148+if(WITH_LIBMULTIPROCESS)
    


    vasild commented at 8:56 am on February 5, 2025:
    I find it could be confusing to have the same option WITH_LIBMULTIPROCESS before and after this PR with different semantics. Maybe WITH_EXTERNAL_LIBMULTIPROCESS is a better name for the new one?

    vasild commented at 8:58 am on February 5, 2025:
    The comment says “when ENABLE_IPC is enabled” but the code is if(WITH_LIBMULTIPROCESS). Shouldn’t that be if(ENABLE_IPC AND WITH_LIBMULTIPROCESS)?
  82. in test/lint/lint-locale-dependence.py:54 in 99b7b857d3
    50@@ -51,6 +51,7 @@
    51 
    52 REGEXP_EXTERNAL_DEPENDENCIES_EXCLUSIONS = [
    53     "src/crypto/ctaes/",
    54+    "src/ipc/libmultiprocess/",
    


    vasild commented at 10:24 am on February 5, 2025:

    Commit 17329d17ef2f640c5f03e36351f1e19f43acf108 lint: Add exclusions for libmultiprocess subtree looks suboptimal because it adds exceptions for the entire src/ipc/libmultiprocess/ subtree for a whole class of issues. Because there are a few specific issues currently. Even if they are ok, it means new issues will sneak in src/ipc/libmultiprocess/ unnoticed.

    It would be better to either resolve the issues and not exclude src/ipc/libmultiprocess/ from linters or add exceptions for the particular issues mentioned in the commit message, so that newly added issues in the future will be detected.

    Here is a patch to resolve the locale-dependent issues:

      0diff --git i/src/ipc/libmultiprocess/example/calculator.cpp w/src/ipc/libmultiprocess/example/calculator.cpp
      1index 4290d68733..6926fb8b57 100644
      2--- i/src/ipc/libmultiprocess/example/calculator.cpp
      3+++ w/src/ipc/libmultiprocess/example/calculator.cpp
      4@@ -1,11 +1,12 @@
      5 // Copyright (c) 2021 The Bitcoin Core developers
      6 // Distributed under the MIT software license, see the accompanying
      7 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
      8 
      9 #include <calculator.h>
     10+#include <charconv>
     11 #include <fstream>
     12 #include <init.capnp.h>
     13 #include <init.capnp.proxy.h> // NOLINT(misc-include-cleaner)
     14 #include <init.h>
     15 #include <iostream>
     16 #include <memory>
     17@@ -41,13 +42,17 @@ static void LogPrint(bool raise, const std::string& message)
     18 int main(int argc, char** argv)
     19 {
     20     if (argc != 2) {
     21         std::cout << "Usage: mpcalculator <fd>\n";
     22         return 1;
     23     }
     24+    int fd;
     25+    if (std::from_chars(argv[1], argv[1] + strlen(argv[1]), fd).ec != std::errc{}) {
     26+        std::cout << argv[1] << " is not a number or is larger than an int\n";
     27+        return 1;
     28+    }
     29     mp::EventLoop loop("mpcalculator", LogPrint);
     30-    const int fd = std::stoi(argv[1]);
     31     std::unique_ptr<Init> init = std::make_unique<InitImpl>();
     32     mp::ServeStream<InitInterface>(loop, fd, *init);
     33     loop.loop();
     34     return 0;
     35 }
     36diff --git i/src/ipc/libmultiprocess/example/example.cpp w/src/ipc/libmultiprocess/example/example.cpp
     37index a4f84c55a7..d0e9458f5c 100644
     38--- i/src/ipc/libmultiprocess/example/example.cpp
     39+++ w/src/ipc/libmultiprocess/example/example.cpp
     40@@ -22,13 +22,13 @@ static auto Spawn(mp::EventLoop& loop, const std::string& process_argv0, const s
     41 {
     42     int pid;
     43     const int fd = mp::SpawnProcess(pid, [&](int fd) -> std::vector<std::string> {
     44         fs::path path = process_argv0;
     45         path.remove_filename();
     46         path.append(new_exe_name);
     47-        return {path.string(), std::to_string(fd)};
     48+        return {path.string(), std::format("{:d}", fd)};
     49     });
     50     return std::make_tuple(mp::ConnectStream<InitInterface>(loop, fd), pid);
     51 }
     52 
     53 static void LogPrint(bool raise, const std::string& message)
     54 {
     55diff --git i/src/ipc/libmultiprocess/example/printer.cpp w/src/ipc/libmultiprocess/example/printer.cpp
     56index ccaed6890c..924c651784 100644
     57--- i/src/ipc/libmultiprocess/example/printer.cpp
     58+++ w/src/ipc/libmultiprocess/example/printer.cpp
     59@@ -1,11 +1,12 @@
     60 // Copyright (c) 2021 The Bitcoin Core developers
     61 // Distributed under the MIT software license, see the accompanying
     62 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
     63 
     64 #include <fstream>
     65+#include <charconv>
     66 #include <init.capnp.h>
     67 #include <init.capnp.proxy.h> // NOLINT(misc-include-cleaner)
     68 #include <init.h>
     69 #include <iostream>
     70 #include <memory>
     71 #include <mp/proxy-io.h>
     72@@ -34,13 +35,17 @@ static void LogPrint(bool raise, const std::string& message)
     73 int main(int argc, char** argv)
     74 {
     75     if (argc != 2) {
     76         std::cout << "Usage: mpprinter <fd>\n";
     77         return 1;
     78     }
     79+    int fd;
     80+    if (std::from_chars(argv[1], argv[1] + strlen(argv[1]), fd).ec != std::errc{}) {
     81+        std::cout << argv[1] << " is not a number or is larger than an int\n";
     82+        return 1;
     83+    }
     84     mp::EventLoop loop("mpprinter", LogPrint);
     85-    const int fd = std::stoi(argv[1]);
     86     std::unique_ptr<Init> init = std::make_unique<InitImpl>();
     87     mp::ServeStream<InitInterface>(loop, fd, *init);
     88     loop.loop();
     89     return 0;
     90 }
     91diff --git i/src/ipc/libmultiprocess/src/mp/gen.cpp w/src/ipc/libmultiprocess/src/mp/gen.cpp
     92index c4d9a51675..267c2837a6 100644
     93--- i/src/ipc/libmultiprocess/src/mp/gen.cpp
     94+++ w/src/ipc/libmultiprocess/src/mp/gen.cpp
     95@@ -10,12 +10,13 @@
     96 #include <cstdint>
     97 #include <cstdio>
     98 #include <cstdlib>
     99 #include <errno.h>
    100 #include <fstream>
    101 #include <functional>
    102+#include <iostream>
    103 #include <kj/array.h>
    104 #include <kj/common.h>
    105 #include <kj/filesystem.h>
    106 #include <kj/memory.h>
    107 #include <kj/string.h>
    108 #include <map>
    109@@ -634,13 +635,13 @@ static void Generate(kj::StringPtr src_prefix,
    110     h << "#endif\n";
    111 }
    112 
    113 int main(int argc, char** argv)
    114 {
    115     if (argc < 3) {
    116-        fprintf(stderr, "Usage: " PROXY_BIN " SRC_PREFIX INCLUDE_PREFIX SRC_FILE [IMPORT_PATH...]\n");
    117+        std::cerr << "Usage: " << PROXY_BIN << " SRC_PREFIX INCLUDE_PREFIX SRC_FILE [IMPORT_PATH...]\n";
    118         exit(1);
    119     }
    120     std::vector<kj::StringPtr> import_paths;
    121     std::vector<kj::Own<const kj::ReadableDirectory>> import_dirs;
    122     auto fs = kj::newDiskFilesystem();
    123     auto cwd = fs->getCurrentPath();
    124diff --git i/src/ipc/libmultiprocess/src/mp/util.cpp w/src/ipc/libmultiprocess/src/mp/util.cpp
    125index 691ae0b34b..15c40b3373 100644
    126--- i/src/ipc/libmultiprocess/src/mp/util.cpp
    127+++ w/src/ipc/libmultiprocess/src/mp/util.cpp
    128@@ -3,12 +3,13 @@
    129 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
    130 
    131 #include <mp/config.h>
    132 #include <mp/util.h>
    133 
    134 #include <errno.h>
    135+#include <format>
    136 #include <kj/common.h>
    137 #include <kj/string-tree.h>
    138 #include <pthread.h>
    139 #include <sstream>
    140 #include <stdio.h>
    141 #include <string>
    142@@ -82,15 +83,13 @@ std::string LogEscape(const kj::StringTree& string)
    143     string.visit([&](const kj::ArrayPtr<const char>& piece) {
    144         if (result.size() > MAX_SIZE) return;
    145         for (const char c : piece) {
    146             if (c == '\\') {
    147                 result.append("\\\\");
    148             } else if (c < 0x20 || c > 0x7e) {
    149-                char escape[4];
    150-                snprintf(escape, 4, "\\%02x", c);
    151-                result.append(escape);
    152+                result.append(std::format("\\{:02x}", c));
    153             } else {
    154                 result.push_back(c);
    155             }
    156             if (result.size() > MAX_SIZE) {
    157                 result += "...";
    158                 break;
    159diff --git i/src/ipc/libmultiprocess/test/mp/test/test.cpp w/src/ipc/libmultiprocess/test/mp/test/test.cpp
    160index 7721d41ff6..7fc64f6741 100644
    161--- i/src/ipc/libmultiprocess/test/mp/test/test.cpp
    162+++ w/src/ipc/libmultiprocess/test/mp/test/test.cpp
    163@@ -8,12 +8,13 @@
    164 #include <mp/test/foo.h>
    165 
    166 #include <capnp/capability.h>
    167 #include <cstdio>
    168 #include <future>
    169 #include <functional>
    170+#include <iostream>
    171 #include <memory>
    172 #include <kj/common.h>
    173 #include <kj/memory.h>
    174 #include <kj/test.h>
    175 #include <string>
    176 #include <thread>
    177@@ -24,13 +25,15 @@ namespace test {
    178 
    179 KJ_TEST("Call FooInterface methods")
    180 {
    181     std::promise<std::unique_ptr<ProxyClient<messages::FooInterface>>> foo_promise;
    182     std::function<void()> disconnect_client;
    183     std::thread thread([&]() {
    184-        EventLoop loop("mptest", [](bool raise, const std::string& log) { printf("LOG%i: %s\n", raise, log.c_str()); });
    185+        EventLoop loop("mptest", [](bool raise, const std::string& log) {
    186+            std::cout << "LOG" << raise << ": " << log << "\n";
    187+        });
    188         auto pipe = loop.m_io_context.provider->newTwoWayPipe();
    189 
    190         auto connection_client = std::make_unique<Connection>(loop, kj::mv(pipe.ends[0]));
    191         auto foo_client = std::make_unique<ProxyClient<messages::FooInterface>>(
    192             connection_client->m_rpc_system->bootstrap(ServerVatId().vat_id).castAs<messages::FooInterface>(),
    193             connection_client.get(), /* destroy_connection= */ false);
    194diff --git i/test/lint/lint-locale-dependence.py w/test/lint/lint-locale-dependence.py
    195index 1b1aeffdcb..d84e458bb1 100755
    196--- i/test/lint/lint-locale-dependence.py
    197+++ w/test/lint/lint-locale-dependence.py
    198@@ -48,13 +48,12 @@ KNOWN_VIOLATIONS = [
    199     "src/wallet/bdb.cpp:.*DbEnv::strerror",  # False positive
    200     "src/util/syserror.cpp:.*strerror",      # Outside this function use `SysErrorString`
    201 ]
    202 
    203 REGEXP_EXTERNAL_DEPENDENCIES_EXCLUSIONS = [
    204     "src/crypto/ctaes/",
    205-    "src/ipc/libmultiprocess/",
    206     "src/leveldb/",
    207     "src/secp256k1/",
    208     "src/minisketch/",
    209     "src/tinyformat.h",
    210 ]
    211 
    

    And also another one to still run test_runner on src/ipc/libmultiprocess/ but only skip std::filesystem check for example.cpp and the include guard test for the entire src/ipc/libmultiprocess/:

     0diff --git i/test/lint/lint-include-guards.py w/test/lint/lint-include-guards.py
     1index 77af05c1c2..ed10deaa90 100755
     2--- i/test/lint/lint-include-guards.py
     3+++ w/test/lint/lint-include-guards.py
     4@@ -17,12 +17,13 @@ from lint_ignore_dirs import SHARED_EXCLUDED_SUBTREES
     5 
     6 HEADER_ID_PREFIX = 'BITCOIN_'
     7 HEADER_ID_SUFFIX = '_H'
     8 
     9 EXCLUDE_FILES_WITH_PREFIX = ['contrib/devtools/bitcoin-tidy',
    10                              'src/crypto/ctaes',
    11+                             'src/ipc/libmultiprocess',
    12                              'src/tinyformat.h',
    13                              'src/bench/nanobench.h',
    14                              'src/test/fuzz/FuzzedDataProvider.h'] + SHARED_EXCLUDED_SUBTREES
    15 
    16 
    17 def _get_header_file_lst() -> list[str]:
    18diff --git i/test/lint/lint_ignore_dirs.py w/test/lint/lint_ignore_dirs.py
    19index 7525eac341..af9ee7ef6b 100644
    20--- i/test/lint/lint_ignore_dirs.py
    21+++ w/test/lint/lint_ignore_dirs.py
    22@@ -1,6 +1,5 @@
    23 SHARED_EXCLUDED_SUBTREES = ["src/leveldb/",
    24                  "src/crc32c/",
    25                  "src/secp256k1/",
    26                  "src/minisketch/",
    27-                 "src/ipc/libmultiprocess/",
    28                 ]
    29diff --git i/test/lint/test_runner/src/main.rs w/test/lint/test_runner/src/main.rs
    30index fe77a98c29..a633f5d43d 100644
    31--- i/test/lint/test_runner/src/main.rs
    32+++ w/test/lint/test_runner/src/main.rs
    33@@ -284,13 +284,13 @@ fn lint_std_filesystem() -> LintResult {
    34         .args([
    35             "grep",
    36             "--line-number",
    37             "std::filesystem",
    38             "--",
    39             "./src/",
    40-            ":(exclude)src/ipc/libmultiprocess/",
    41+            ":(exclude)src/ipc/libmultiprocess/example/example.cpp",
    42             ":(exclude)src/util/fs.h",
    43         ])
    44         .status()
    45         .expect("command error")
    46         .success();
    47     if found {
    
  83. in depends/packages.md:65 in 99b7b857d3
    60+build id, so if the directory contents change, the package and everything
    61+depending on it will be rebuilt. For efficiency, the tarball is cached once it
    62+has been created, but if the local directory is touched, it will be rebuilt.
    63+
    64+Local packages can be useful for using git submodules or subtrees to manage
    65+package sources, or for  testing local changes that are not available to
    


    vasild commented at 11:04 am on February 5, 2025:
    0package sources, or for testing local changes that are not available to
    
  84. vasild commented at 11:11 am on February 5, 2025: contributor

    Approach ACK 99b7b857d3b8b9ce9bd7501e2480583369740c55

    I think it would be best if all sources are build with the same flags and the same build mechanism is used by devs, CI and depends. I also do not see a harm of an extra option which is off by default, to use an external libmultiprocess, if that makes libmultiprocess’ devs life easier.

    PS I find 97058b53e8 depends: Switch libmultiprocess packages to use local git subtree hard to review.

  85. luke-jr commented at 2:21 pm on February 5, 2025: member
    I think all those benefits could be achieved by adding the glue yet having the developer who wants to test checkout the upstream repo into a subdirectory manually to use it (cmake can complain clearly if missing)
  86. ryanofsky commented at 2:44 pm on February 5, 2025: contributor

    I think all those benefits could be achieved by adding the glue yet having the developer who wants to test checkout the upstream repo into a subdirectory manually to use it (cmake can complain clearly if missing) @luke-jr, Yes, we could provide a glue script to download the git repository instead of mirroring it in a git subtree, pinning it to a hash to avoid bringing in unreviewed changes. This is essentially what depends system does now. Do you think a glue script would be better than a subtree though? That just seems more awkward all around.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-02-05 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me