depends: fix multiprocess build on OpenBSD (apply capnp patch, correct SHA256SUM command) #32690

pull theStack wants to merge 2 commits into bitcoin:master from theStack:202506-depends-fix_openbsd_multiprocess_build changing 5 files +151 −2
  1. theStack commented at 0:21 am on June 6, 2025: contributor

    This PR fixes the multiprocess depends build for OpenBSD by applying upstream patch https://github.com/capnproto/capnproto/pull/2308 and switching the SHA256SUM command to output hash sums in the expected format (the default is BSD format [1], but we need GNU format [2], see commit message for details). Note that the hashing issue is only prevailing for packages defining the $(package)_local_dir variable (introduced in 5d105fb8c3ffa39c3e716c3147ee74795b129113, part of #31741), where the following line of the fetch_local_dir_sha256 function leads to the wrong output: https://github.com/bitcoin/bitcoin/blob/ae024137bda9fe189f4e7ccf26dbaffd44cbbeb6/depends/funcs.mk#L57

    The first commit can be replaced with a simple capnp version bump once this is available in a release.

    Tested on OpenBSD 7.7 (x86_64) via

    0$ gmake -C depends MULTIPROCESS=1 NO_BOOST=1 NO_LIBEVENT=1 NO_QT=1 NO_QR=1 NO_WALLET=1 NO_ZMQ=1 NO_USDT=1

    [1] example output: SHA256 (/home/thestack/.vimrc) = 6ba69d100e8c5ca0488ded6293d4e5f740a6a5d5ace96cbcf0599c18d27389e4 [2] example output: 6ba69d100e8c5ca0488ded6293d4e5f740a6a5d5ace96cbcf0599c18d27389e4 /home/thestack/.vimrc

  2. depends: add patch to fix capnp build on OpenBSD 
    see https://github.com/capnproto/capnproto/pull/2308
    2d938720bd
  3. depends: fix SHA256SUM command on OpenBSD (use GNU mode output) 
    On OpenBSD, the `sha256` command by default outputs hashsums on files in
"BSD" mode, looking like this:

$ sha256 ~/.vimrc
SHA256 (/home/thestack/.vimrc) = 6ba69d100e8c5ca0488ded6293d4e5f740a6a5d5ace96cbcf0599c18d27389e4

This is not compatible with our depends commands, which expect the
hashes to be on the first column (to be extracted via `cut -d" " -f1`).
Fix this by switching to GNU mode output, looking like this:

$ sha256 -r ~/.vimrc
6ba69d100e8c5ca0488ded6293d4e5f740a6a5d5ace96cbcf0599c18d27389e4 /home/thestack/.vimrc

Without this change, the multiprocess depends build fails with the following output:

$ gmake -C depends MULTIPROCESS=1 NO_BOOST=1 NO_LIBEVENT=1 NO_QT=1 NO_QR=1 NO_WALLET=1 NO_ZMQ=1 NO_USDT=1
[ ..... ]
Extracting native_libmultiprocess...
sha256: /home/thestack/bitcoin/depends/work/build/x86_64-unknown-openbsd7.7/native_libmultiprocess/-2bc902f4693/.src-ipc-libmultiprocess.tar.hash: no properly formatted checksum lines found
gmake: *** [funcs.mk:342: /home/thestack/bitcoin/depends/work/build/x86_64-unknown-openbsd7.7/native_libmultiprocess/-2bc902f4693/.stamp_extracted] Error 1
    8713e8060d
  4. DrahtBot commented at 0:22 am on June 6, 2025: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32690.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK Sjors, hebasto, fanquake

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    LLM Linter (✨ experimental)

    Possible typos and grammar issues:

    • may also solves -> may also solve [correct verb form]

    drahtbot_id_4_m

  5. DrahtBot added the label Build system on Jun 6, 2025
  6. hebasto commented at 8:13 am on June 6, 2025: member

    Concept ACK 8713e8060d504f561fed705b4aa5af7b96c36e75.

    https://github.com/hebasto/bitcoin-core-nightly/actions/runs/15485455235/job/43599018182 looks good.

  7. hebasto commented at 8:25 am on June 6, 2025: member
    Why 8713e8060d504f561fed705b4aa5af7b96c36e75 is necessary for libmultiprocess, but not for other packages?
  8. Sjors commented at 9:04 am on June 6, 2025: member

    I tested this on top of #31802 on a VM running OpenBSD 7.7 (dropping NO_IPC for OpenBSB).

    Without the patch I get a bunch of errors like:

    0[  2%] Building CXX object src/kj/CMakeFiles/kj.dir/cidr.c++.o
1/root/src/bitcoin/depends/work/build/aarch64-unknown-openbsd7.7/native_capnp/1.1.0-f5dbdc0b2f0/src/kj/cidr.c++:116:71: error: member access into incomplete type 'const struct sockaddr_in6'
2        otherBits = reinterpret_cast<const struct sockaddr_in6*>(addr)->sin6_addr.s6_addr;

    With the patch the build still fails:

     0gmake -C depends NO_QT=1 NO_QR=1 NO_ZMQ=1 NO_USDT=1
 1...
 2[ 95%] Linking CXX executable capnp
 3ld: warning: parser.c++(parser.c++.o:(kj::Maybe<decltype(apply(instance<capnp::compiler::CapnpParser::CapnpParser(capnp::Orphanage, capnp::compiler::ErrorReporter&)::'lambda38'(capnp::compiler::(anonymous namespace)::Located<capnp::Text::Reader>&&, kj::Maybe<capnp::Orphan<capnp::compiler::LocatedInteger>>&&, capnp::compiler::(anonymous namespace)::Located<kj::Array<kj::Maybe<capnp::compiler::(anonymous namespace)::Located<capnp::Text::Reader>>>>&&, capnp::Orphan<capnp::compiler::Expression>&&, kj::Array<capnp::Orphan<capnp::compiler::Declaration::AnnotationApplication>>&&)&>(), instance<kj::parse::OutputType_<decltype(instance<kj::parse::Sequence_<kj::parse::TransformOrReject_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)0, &capnp::compiler::Token::Reader::getIdentifier() const>> const&, capnp::compiler::(anonymous namespace)::ExactString>, kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)0, &capnp::compiler::Token::Reader::getIdentifier() const>> const&, kj::parse::Optional_<kj::parse::ParserRef<kj::parse::IteratorInput<capnp::compiler::Token::Reader, capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>, capnp::Orphan<capnp::compiler::LocatedInteger>>&>, kj::parse::Transform_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::List<capnp::List<capnp::compiler::Token, (capnp::Kind)3>, (capnp::Kind)6>::Reader, (capnp::compiler::Token::Which)5, &capnp::compiler::Token::Reader::getParenthesizedList() const>> const&, capnp::compiler::(anonymous namespace)::ParseListItems<kj::parse::OneOf_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)0, &capnp::compiler::Token::Reader::getIdentifier() const>> const&, kj::parse::TransformWithLocation_<kj::parse::TransformOrReject_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)4, &capnp::compiler::Token::Reader::getOperator() const>> const&, capnp::compiler::(anonymous namespace)::ExactString>, capnp::compiler::CapnpParser::CapnpParser(capnp::Orphanage, capnp::compiler::ErrorReporter&)::'lambda37'(kj::parse::Span<capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>)>>&>>, kj::parse::TransformOrReject_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)4, &capnp::compiler::Token::Reader::getOperator() const>> const&, capnp::compiler::(anonymous namespace)::ExactString>, kj::parse::ParserRef<kj::parse::IteratorInput<capnp::compiler::Token::Reader, capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>, capnp::Orphan<capnp::compiler::Expression>>&, kj::parse::Many_<kj::parse::ParserRef<kj::parse::IteratorInput<capnp::compiler::Token::Reader, capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>, capnp::Orphan<capnp::compiler::Declaration::AnnotationApplication>>&, false>>&>()(instance<kj::parse::IteratorInput<capnp::compiler::Token::Reader, capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>&>()))>::Type&&>()))> kj::parse::Transform_<kj::parse::Sequence_<kj::parse::TransformOrReject_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)0, &capnp::compiler::Token::Reader::getIdentifier() const>> const&, capnp::compiler::(anonymous namespace)::ExactString>, kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)0, &capnp::compiler::Token::Reader::getIdentifier() const>> const&, kj::parse::Optional_<kj::parse::ParserRef<kj::parse::IteratorInput<capnp::compiler::Token::Reader, capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>, capnp::Orphan<capnp::compiler::LocatedInteger>>&>, kj::parse::Transform_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::List<capnp::List<capnp::compiler::Token, (capnp::Kind)3>, (capnp::Kind)6>::Reader, (capnp::compiler::Token::Which)5, &capnp::compiler::Token::Reader::getParenthesizedList() const>> const&, capnp::compiler::(anonymous namespace)::ParseListItems<kj::parse::OneOf_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)0, &capnp::compiler::Token::Reader::getIdentifier() const>> const&, kj::parse::TransformWithLocation_<kj::parse::TransformOrReject_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)4, &capnp::compiler::Token::Reader::getOperator() const>> const&, capnp::compiler::(anonymous namespace)::ExactString>, capnp::compiler::CapnpParser::CapnpParser(capnp::Orphanage, capnp::compiler::ErrorReporter&)::'lambda37'(kj::parse::Span<capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>)>>&>>, kj::parse::TransformOrReject_<kj::parse::TransformOrReject_<kj::parse::Any_ const&, capnp::compiler::(anonymous namespace)::MatchTokenType<capnp::Text::Reader, (capnp::compiler::Token::Which)4, &capnp::compiler::Token::Reader::getOperator() const>> const&, capnp::compiler::(anonymous namespace)::ExactString>, kj::parse::ParserRef<kj::parse::IteratorInput<capnp::compiler::Token::Reader, capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>, capnp::Orphan<capnp::compiler::Expression>>&, kj::parse::Many_<kj::parse::ParserRef<kj::parse::IteratorInput<capnp::compiler::Token::Reader, capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>, capnp::Orphan<capnp::compiler::Declaration::AnnotationApplication>>&, false>>, capnp::compiler::CapnpParser::CapnpParser(capnp::Orphanage, capnp::compiler::ErrorReporter&)::'lambda38'(capnp::compiler::(anonymous namespace)::Located<capnp::Text::Reader>&&, kj::Maybe<capnp::Orphan<capnp::compiler::LocatedInteger>>&&, capnp::compiler::(anonymous namespace)::Located<kj::Array<kj::Maybe<capnp::compiler::(anonymous namespace)::Located<capnp::Text::Reader>>>>&&, capnp::Orphan<capnp::compiler::Expression>&&, kj::Array<capnp::Orphan<capnp::compiler::Declaration::AnnotationApplication>>&&)>::operator()<kj::parse::IteratorInput<capnp::compiler::Token::Reader, capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>>(kj::parse::IteratorInput<capnp::compiler::Token::Reader, capnp::_::IndexingIterator<capnp::List<capnp::compiler::Token, (capnp::Kind)3>::Reader const, capnp::compiler::Token::Reader>>&) const) in archive libcapnpc.a): warning: strcpy() is almost always misused, please use strlcpy()
 4ld: error: undefined symbol: std::__throw_bad_array_new_length()
 5>>> referenced by module-loader.c++
 6>>>               CMakeFiles/capnp_tool.dir/compiler/module-loader.c++.o:(capnp::compiler::ModuleLoader::Impl::loadModule(kj::ReadableDirectory const&, kj::PathPtr))
 7>>> referenced by compiler.c++
 8>>>               compiler.c++.o:(std::_Hashtable<capnp::compiler::Compiler::Node*, std::pair<capnp::compiler::Compiler::Node* const, unsigned int>, std::allocator<std::pair<capnp::compiler::Compiler::Node* const, unsigned int>>, std::__detail::_Select1st, std::equal_to<capnp::compiler::Compiler::Node*>, std::hash<capnp::compiler::Compiler::Node*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>>::_M_rehash(unsigned long, unsigned long const&)) in archive libcapnpc.a
 9>>> referenced by compiler.c++
10>>>               compiler.c++.o:(std::_Hashtable<capnp::compiler::Module*, std::pair<capnp::compiler::Module* const, kj::Own<capnp::compiler::Compiler::CompiledModule, std::nullptr_t>>, std::allocator<std::pair<capnp::compiler::Module* const, kj::Own<capnp::compiler::Compiler::CompiledModule, std::nullptr_t>>>, std::__detail::_Select1st, std::equal_to<capnp::compiler::Module*>, std::hash<capnp::compiler::Module*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>>::_M_rehash(unsigned long, unsigned long const&)) in archive libcapnpc.a
11>>> referenced 2 more times
12collect2: error: ld returned 1 exit status
13gmake[3]: *** [src/capnp/CMakeFiles/capnp_tool.dir/build.make:122: src/capnp/capnp] Error 1
14gmake[2]: *** [CMakeFiles/Makefile2:480: src/capnp/CMakeFiles/capnp_tool.dir/all] Error 2
15gmake[1]: *** [Makefile:136: all] Error 2
16gmake[1]: Leaving directory '/home/sjors/src/bitcoin/depends/work/build/aarch64-unknown-openbsd7.7/capnp/1.1.0-adf16ebd094'
17gmake: *** [funcs.mk:342: /home/sjors/src/bitcoin/depends/work/build/aarch64-unknown-openbsd7.7/capnp/1.1.0-adf16ebd094/./.stamp_built] Error 2
18gmake: Leaving directory '/home/sjors/src/bitcoin/depends'

    (this also happens with just this PR at 8713e8060d504f561fed705b4aa5af7b96c36e75)

    Though maybe there’s something wrong with my VM? #32691 In particular, I symlinked egcc and eg++ to /usr/local/bin/g{cc,++}.

  9. in depends/builders/openbsd.mk:4 in 8713e8060d 
    0@@ -1,7 +1,7 @@
1 build_openbsd_CC = clang
2 build_openbsd_CXX = clang++
3 
4-build_openbsd_SHA256SUM = sha256
5+build_openbsd_SHA256SUM = sha256 -r
    hebasto commented at 11:23 am on June 6, 2025:
    I believe the same applies to FreeBSD as well.
    Sjors commented at 2:11 pm on June 6, 2025:
    I’ll spin up a FreeBSD 14.1 VM to try that.
    Sjors commented at 2:30 pm on June 6, 2025:

    FreeBSD depends instruction is also missing gmake and cmake in the list of things to install.

    build_freebsd_SHA256SUM = sha256sum already outputs the format we need.

    hebasto commented at 3:11 pm on June 6, 2025:

    build_freebsd_SHA256SUM = sha256sum already outputs the format we need.

    0$ freebsd-version 
114.2-RELEASE
2$ sha256 ~/.cshrc
3SHA256 (/home/hebasto/.cshrc) = 523d96d41067f380469d4f3d68b540edb63b127e91a9955fb7f6a01c53545258
    Sjors commented at 3:30 pm on June 6, 2025:

    sha256sum != sha256 != shasum -a 256

    0$ freebsd-version
114.1-RELEASE
2$ sha256 README.md
3SHA256 (README.md) = 2c98d4dd80b0b50e6668f518ef947135528f076c655a6fa6593111a592898e63
4$ sha256sum README.md
52c98d4dd80b0b50e6668f518ef947135528f076c655a6fa6593111a592898e63  README.md
6$ shasum -a 256 README.md
72c98d4dd80b0b50e6668f518ef947135528f076c655a6fa6593111a592898e63  README.md
    hebasto commented at 3:45 pm on June 6, 2025:
    I was blind. Sorry for the noise.
  10. Sjors commented at 3:41 pm on June 6, 2025: member

    ACK 8713e8060d504f561fed705b4aa5af7b96c36e75

    Switched to an x86_64 based VM and it builds fine. I also ran the Bitcoin Core test suite.

    Can you add a note to the OpenBSD instruction in depends/README.md to also install cmake?

  11. DrahtBot requested review from hebasto on Jun 6, 2025
  12. theStack commented at 3:48 pm on June 6, 2025: contributor

    Why 8713e80 is necessary for libmultiprocess, but not for other packages?

    The libmultiprocess package is the only one which sets the $(package)_local_dir variable, causing execution of the fetch_local_dir_sha256 shell script where the hash is extracted from the file manually, expecting the hash in the first row (=GNU format): https://github.com/bitcoin/bitcoin/blob/ae024137bda9fe189f4e7ccf26dbaffd44cbbeb6/depends/funcs.mk#L57

    Another instance where we do this manual extraction is in the calculation of $(1)_all_file_checksums: https://github.com/bitcoin/bitcoin/blob/ae024137bda9fe189f4e7ccf26dbaffd44cbbeb6/depends/funcs.mk#L62

    But this variable is only a temporary for calculating $(1)_recipe_hash, which is done by feeding the hashing tool from stdin, where the output is as expected [1]: https://github.com/bitcoin/bitcoin/blob/ae024137bda9fe189f4e7ccf26dbaffd44cbbeb6/depends/funcs.mk#L69

    So this was unnoticed, but the recipe hash calculation was not entirely correct, so I’d suggest to backport this commit.

    Note that instances where a hash list is verified via -c are not affected, as this options accepts both the BSD and the GNU formats. Also, if the input to the hashing command is piped in (i.e. coming from stdin), the output is also in the expected format even without specifying -r.

    [1] e.g.

    0$ echo "foobar" | sha256
1aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f
  13. theStack commented at 3:57 pm on June 6, 2025: contributor

    @Sjors:

    Can you add a note to the OpenBSD instruction in depends/README.md to also install cmake?

    Seems like material for a different PR, considering this affects other BSDs as well, and at first glance it seems there are even other packages missing, like e.g. curl (which AFAIR, is also not part of OpenBSD’s base system). Will open a follow-up later.

  14. hebasto approved
  15. hebasto commented at 5:00 pm on June 6, 2025: member
    ACK 8713e8060d504f561fed705b4aa5af7b96c36e75.
  16. fanquake referenced this in commit c1d4253d31 on Jun 10, 2025
  17. fanquake approved
  18. fanquake commented at 3:04 pm on June 11, 2025: member
    ACK 8713e8060d504f561fed705b4aa5af7b96c36e75
  19. fanquake merged this on Jun 11, 2025
  20. fanquake closed this on Jun 11, 2025
  21. hebasto referenced this in commit 6fec181427 on Jun 11, 2025
  22. hebasto referenced this in commit 0b11d88e37 on Jun 11, 2025
  23. fanquake commented at 4:25 pm on June 11, 2025: member
    Backported to 29.x in #32690.
  24. fanquake referenced this in commit e8a5b9561c on Jun 11, 2025
  25. fanquake referenced this in commit fe4e8c28d6 on Jun 11, 2025


theStack DrahtBot hebasto Sjors fanquake

Labels
Build system

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2025-06-15 06:13 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me