build: indefinite mpgen hang on Ubuntu 22.04

fanquake commented at 8:52 am on August 12, 2025: member

Noticed this while looking at #31802, but the same happens on master (273e600e65c2e31a6e9a0bd72b40672aaa503b08).

0# podman run -it ubuntu:22.04 /bin/bash
1
2apt update && apt upgrade -y && apt install build-essential cmake pkgconf python3 git libevent-dev libboost-dev libsqlite3-dev libcapnp-dev capnproto -y
3
4git clone https://github.com/bitcoin/bitcoin
5
6cd bitcoin
7cmake -B build -DENABLE_IPC=ON
8cmake --build build

The build never completes. It hangs indefinitely doing capnp generation:

 0cmake --build build -j10
 1[  1%] Built target crc32c
 2[  1%] Generating bitcoin-build-info.h
 3[  1%] Built target secp256k1_precomputed
 4[  1%] Built target mputil
 5[  2%] Built target univalue
 6[  4%] Built target bitcoin_consensus
 7[  7%] Built target bitcoin_crypto
 8[ 10%] Built target minisketch
 9[ 10%] Built target bitcoin_cli
10[ 10%] Built target exhaustive_tests
11[ 16%] Built target leveldb
12[ 26%] Built target unitester
13[ 26%] Built target mpgen
14[ 26%] Built target object
15[ 27%] Built target multiprocess
16[ 28%] Built target secp256k1
17[ 28%] Built target noverify_tests
18[ 28%] Generating capnp/mining.capnp.c++, capnp/mining.capnp.h, capnp/mining.capnp.proxy-client.c++, capnp/mining.capnp.proxy-types.h, capnp/mining.capnp.proxy-server.c++, capnp/mining.capnp.proxy-types.c++, capnp/mining.capnp.proxy.h
19[ 28%] Generating capnp/init.capnp.c++, capnp/init.capnp.h, capnp/init.capnp.proxy-client.c++, capnp/init.capnp.proxy-types.h, capnp/init.capnp.proxy-server.c++, capnp/init.capnp.proxy-types.c++, capnp/init.capnp.proxy.h
20[ 28%] Generating capnp/echo.capnp.c++, capnp/echo.capnp.h, capnp/echo.capnp.proxy-client.c++, capnp/echo.capnp.proxy-types.h, capnp/echo.capnp.proxy-server.c++, capnp/echo.capnp.proxy-types.c++, capnp/echo.capnp.proxy.h
21[ 28%] Generating capnp/common.capnp.c++, capnp/common.capnp.h, capnp/common.capnp.proxy-client.c++, capnp/common.capnp.proxy-types.h, capnp/common.capnp.proxy-server.c++, capnp/common.capnp.proxy-types.c++, capnp/common.capnp.proxy.h
22[ 28%] Generating mp/test/foo.capnp.c++, mp/test/foo.capnp.h, mp/test/foo.capnp.proxy-client.c++, mp/test/foo.capnp.proxy-types.h, mp/test/foo.capnp.proxy-server.c++, mp/test/foo.capnp.proxy-types.c++, mp/test/foo.capnp.proxy.h
23[ 28%] Built target tests
24[ 28%] Generating capnp/echo.capnp.c++, capnp/echo.capnp.h, capnp/echo.capnp.proxy-client.c++, capnp/echo.capnp.proxy-types.h, capnp/echo.capnp.proxy-server.c++, capnp/echo.capnp.proxy-types.c++, capnp/echo.capnp.proxy.h
25[ 28%] Generating capnp/common.capnp.c++, capnp/common.capnp.h, capnp/common.capnp.proxy-client.c++, capnp/common.capnp.proxy-types.h, capnp/common.capnp.proxy-server.c++, capnp/common.capnp.proxy-types.c++, capnp/common.capnp.proxy.h
26[ 32%] Built target test_util
27[ 32%] Generating capnp/init.capnp.c++, capnp/init.capnp.h, capnp/init.capnp.proxy-client.c++, capnp/init.capnp.proxy-types.h, capnp/init.capnp.proxy-server.c++, capnp/init.capnp.proxy-types.c++, capnp/init.capnp.proxy.h
28[ 32%] Generating capnp/mining.capnp.c++, capnp/mining.capnp.h, capnp/mining.capnp.proxy-client.c++, capnp/mining.capnp.proxy-types.h, capnp/mining.capnp.proxy-server.c++, capnp/mining.capnp.proxy-types.c++, capnp/mining.capnp.proxy.h
29[ 32%] Built target generate_build_info
30[ 37%] Built target bitcoin_wallet
31[ 37%] Built target bitcoin_clientversion
32[ 46%] Built target bitcoin_common
33[ 52%] Built target bitcoin_util
34[ 67%] Built target bitcoin_node
35[ 67%] Built target bitcoin-cli
36[ 67%] Built target bitcoin-tx
37[ 67%] Built target bitcoin-util
38[ 68%] Built target bitcoin-wallet
39[ 68%] Built target bitcoin
40[ 68%] Built target bitcoind

fanquake added the label Build system on Aug 12, 2025

willcl-ark commented at 10:11 am on August 12, 2025: member

I also see the sameinfinite hang:

 0[ 33%] Generating mp/test/foo.capnp.c++, mp/test/foo.capnp.h, mp/test/foo.capnp.proxy-client.c++, mp/test/foo.capnp.proxy-types.h, mp/test/foo.capnp.proxy-server.c++, mp/test/foo.capnp.proxy-types.c++, mp/test/foo.capnp.proxy.h
 1[ 33%] Generating capnp/mining.capnp.c++, capnp/mining.capnp.h, capnp/mining.capnp.proxy-client.c++, capnp/mining.capnp.proxy-types.h, capnp/mining.capnp.proxy-server.c++, capnp/mining.capnp.proxy-types.c++, capnp/mining.capnp.proxy.h
 2[ 33%] Generating capnp/echo.capnp.c++, capnp/echo.capnp.h, capnp/echo.capnp.proxy-client.c++, capnp/echo.capnp.proxy-types.h, capnp/echo.capnp.proxy-server.c++, capnp/echo.capnp.proxy-types.c++, capnp/echo.capnp.proxy.h
 3[ 33%] Generating capnp/common.capnp.c++, capnp/common.capnp.h, capnp/common.capnp.proxy-client.c++, capnp/common.capnp.proxy-types.h, capnp/common.capnp.proxy-server.c++, capnp/common.capnp.proxy-types.c++, capnp/common.capnp.proxy.h
 4[ 33%] Generating capnp/init.capnp.c++, capnp/init.capnp.h, capnp/init.capnp.proxy-client.c++, capnp/init.capnp.proxy-types.h, capnp/init.capnp.proxy-server.c++, capnp/init.capnp.proxy-types.c++, capnp/init.capnp.proxy.h
 5[ 33%] Generating capnp/mining.capnp.c++, capnp/mining.capnp.h, capnp/mining.capnp.proxy-client.c++, capnp/mining.capnp.proxy-types.h, capnp/mining.capnp.proxy-server.c++, capnp/mining.capnp.proxy-types.c++, capnp/mining.capnp.proxy.h
 6[ 33%] Built target generate_build_info
 7Consolidate compiler generated dependencies of target bitcoin_clientversion
 8Consolidate compiler generated dependencies of target bitcoin_util
 9[ 33%] Built target bitcoin_clientversion
10Consolidate compiler generated dependencies of target bitcoin_common
11Consolidate compiler generated dependencies of target bitcoin_wallet
12[ 38%] Built target bitcoin_util
13Consolidate compiler generated dependencies of target bitcoin_node
14Consolidate compiler generated dependencies of target bitcoin
15[ 38%] Built target bitcoin
16[ 47%] Built target bitcoin_common
17[ 47%] Building CXX object src/CMakeFiles/bitcoin-cli.dir/bitcoin-cli.cpp.o
18[ 47%] Building CXX object src/CMakeFiles/bitcoin-util.dir/bitcoin-util.cpp.o
19[ 47%] Building CXX object src/CMakeFiles/bitcoin-tx.dir/bitcoin-tx.cpp.o
20[ 52%] Built target bitcoin_wallet
21[ 52%] Building CXX object src/CMakeFiles/bitcoin-wallet.dir/bitcoin-wallet.cpp.o
22[ 67%] Built target bitcoin_node
23[ 67%] Building CXX object src/CMakeFiles/bitcoin-wallet.dir/init/bitcoin-wallet.cpp.o
24[ 67%] Linking CXX executable ../bin/bitcoin-util
25[ 67%] Building CXX object src/CMakeFiles/bitcoind.dir/bitcoind.cpp.o
26[ 67%] Built target bitcoin-util
27[ 68%] Building CXX object src/CMakeFiles/bitcoin-wallet.dir/wallet/wallettool.cpp.o
28[ 70%] Building CXX object src/CMakeFiles/bitcoind.dir/init/bitcoind.cpp.o
29[ 70%] Linking CXX executable ../bin/bitcoin-cli
30[ 70%] Linking CXX executable ../bin/bitcoin-tx
31[ 70%] Built target bitcoin-cli
32[ 70%] Linking CXX executable ../bin/bitcoind
33[ 70%] Built target bitcoin-tx
34[ 70%] Linking CXX executable ../bin/bitcoin-wallet
35[ 70%] Built target bitcoind
36[ 70%] Built target bitcoin-wallet
37^Cgmake[2]: *** [src/ipc/CMakeFiles/bitcoin_ipc.dir/build.make:96: src/ipc/capnp/echo.capnp.c++] Interrupt
38gmake[2]: *** [src/ipc/CMakeFiles/bitcoin_ipc_headers.dir/build.make:98: src/ipc/capnp/echo.capnp.c++] Interrupt
39gmake[2]: *** [src/ipc/CMakeFiles/bitcoin_ipc_headers.dir/build.make:120: src/ipc/capnp/init.capnp.c++] Interrupt
40gmake[2]: *** [src/ipc/CMakeFiles/bitcoin_ipc.dir/build.make:74: src/ipc/capnp/common.capnp.c++] Interrupt
41gmake[2]: *** [src/ipc/CMakeFiles/bitcoin_ipc_headers.dir/build.make:76: src/ipc/capnp/common.capnp.c++] Interrupt
42gmake[2]: *** [src/ipc/CMakeFiles/bitcoin_ipc.dir/build.make:140: src/ipc/capnp/mining.capnp.c++] Interrupt
43gmake[2]: *** [src/ipc/CMakeFiles/bitcoin_ipc_headers.dir/build.make:142: src/ipc/capnp/mining.capnp.c++] Interrupt
44gmake[2]: *** [src/ipc/libmultiprocess/test/CMakeFiles/mptest.dir/build.make:74: src/ipc/libmultiprocess/test/mp/test/foo.capnp.c++] Interrupt
45gmake[2]: *** [src/ipc/CMakeFiles/bitcoin_ipc.dir/build.make:118: src/ipc/capnp/init.capnp.c++] Interrupt
46gmake[1]: *** [CMakeFiles/Makefile2:1502: src/ipc/libmultiprocess/test/CMakeFiles/mptest.dir/all] Interrupt
47gmake[1]: *** [CMakeFiles/Makefile2:1737: src/ipc/CMakeFiles/bitcoin_ipc.dir/all] Interrupt
48gmake[1]: *** [CMakeFiles/Makefile2:1764: src/ipc/CMakeFiles/bitcoin_ipc_headers.dir/all] Interrupt
49gmake: *** [Makefile:146: all] Interrupt

ryanofsky commented at 2:04 pm on August 12, 2025: contributor

Thanks, instructions make this very easy to reproduce. Error is strange. If I attach with gdb it shows program is hung in a self-loop instruction:

0=> 0x0000563e5f583f7f <+1103>:	jmp    0x563e5f583f7f <main(int, char**)+1103>

Some more context is:

0   0x0000563e5f583f6b <+1083>:	call   0x563e5f584450 <_ZN2kj3OwnIKNS_17ReadableDirectoryEE7disposeEv>
1   0x0000563e5f583f70 <+1088>:	add    $0x8,%r13
2   0x0000563e5f583f74 <+1092>:	cmp    %r13,0x8(%rsp)
3   0x0000563e5f583f79 <+1097>:	jne    0x563e5f583e6d <main(int, char**)+829>
4=> 0x0000563e5f583f7f <+1103>:	jmp    0x563e5f583f7f <main(int, char**)+1103>
5   0x0000563e5f583f81 <+1105>:	nopl   0x0(%rax)
6   0x0000563e5f583f88 <+1112>:	lea    0x70(%rsp),%r15
7   0x0000563e5f583f8d <+1117>:	lea    0xd0(%rsp),%rdi
8   0x0000563e5f583f95 <+1125>:	mov    %r15,%rdx
9   0x0000563e5f583f98 <+1128>:	call   0x563e5f584690 <_ZNSt6vectorIN2kj3OwnIKNS0_17ReadableDirectoryEEESaIS4_EE17_M_realloc_insertIJS4_EEEvN9__gnu_cxx17__normal_iteratorIPS4_S6_EEDpOT_>

GDB says corresponding source code is

 0(gdb) info line *$pc
 1Line 669 of "./ipc/libmultiprocess/src/mp/gen.cpp" starts at address 0x563e5f583f74 <main(int, char**)+1092> and ends at 0x563e5f583f88 <main(int, char**)+1112>.
 2(gdb) info line *$pc
 3Line 669 of "./ipc/libmultiprocess/src/mp/gen.cpp" starts at address 0x563e5f583f74 <main(int, char**)+1092> and ends at 0x563e5f583f88 <main(int, char**)+1112>.
 4(gdb) list
 5664	            import_dirs.emplace_back(kj::mv(*dir));
 6665	        } else {
 7666	            throw std::runtime_error(std::string("Failed to open import directory: ") + argv[i]);
 8667	        }
 9668	    }
10669	    for (const char* path : {CMAKE_INSTALL_PREFIX "/include", capnp_PREFIX "/include"}) {
11670	        KJ_IF_MAYBE(dir, fs->getRoot().tryOpenSubdir(cwd.evalNative(path))) {
12671	            import_paths.emplace_back(path);
13672	            import_dirs.emplace_back(kj::mv(*dir));
14673	        }

But I’m not sure I trust assembly to source mapping because if I add std::cerr prints, program seems to be hanging at a later point ( 146 if (src_prefix == ".") { in Generate()

This is very confusing and I’m not sure what is causing it. I haven’t encountered a jmp instruction just looping to itself before but I guess a compiler might output one as a trap for unreachable code or something like that. Looking at the code, I don’t think I can identify any bugs, so I wonder if cause might be that program is being compiled with bad flags or something like that. Will look into this more.

ryanofsky commented at 2:46 pm on August 12, 2025: contributor

Found the problem, and a workaround below. The bug is kind of crazy. Ubuntu 22.04 installs capnproto 0.8.0 which defines

0inline bool operator==(const char* a, const StringPtr& b) { return b == a; }

Which seems to call itself recursively when compiled with c++20. This doesn’t happen with newer versions of capnproto because they include a change https://github.com/capnproto/capnproto/pull/1170 https://github.com/capnproto/capnproto/pull/1235/commits/4db91df614be91db82833ec1b7cef18ef97711cb dropping it when a newer c++ version is used.

Following change seems to successfully work around the problem though need to test more.

 0--- a/src/ipc/libmultiprocess/src/mp/gen.cpp
 1+++ b/src/ipc/libmultiprocess/src/mp/gen.cpp
 2@@ -143,7 +143,7 @@ static void Generate(kj::StringPtr src_prefix,
 3     const std::vector<kj::Own<const kj::ReadableDirectory>>& import_dirs)
 4 {
 5     std::string output_path;
 6-    if (src_prefix == ".") {
 7+    if (src_prefix == kj::StringPtr{"."}) {
 8         output_path = src_file;
 9     } else if (!src_file.startsWith(src_prefix) || src_file.size() <= src_prefix.size() ||
10                src_file[src_prefix.size()] != '/') {
11@@ -153,7 +153,7 @@ static void Generate(kj::StringPtr src_prefix,
12     }
13 
14     std::string include_path;
15-    if (include_prefix == ".") {
16+    if (include_prefix == kj::StringPtr{"."}) {
17         include_path = src_file;
18     } else if (!src_file.startsWith(include_prefix) || src_file.size() <= include_prefix.size() ||
19                src_file[include_prefix.size()] != '/') {
20@@ -394,8 +394,8 @@ static void Generate(kj::StringPtr src_prefix,
21 
22                 const std::string method_prefix = Format() << message_namespace << "::" << method_interface.getShortDisplayName()
23                                                            << "::" << Cap(method_name);
24-                const bool is_construct = method_name == "construct";
25-                const bool is_destroy = method_name == "destroy";
26+                const bool is_construct = method_name == kj::StringPtr{"construct"};
27+                const bool is_destroy = method_name == kj::StringPtr{"destroy"};
28 
29                 struct Field
30                 {
31@@ -434,7 +434,7 @@ static void Generate(kj::StringPtr src_prefix,
32                         field.result_is_set = true;
33                     }
34 
35-                    if (!param && field_name == "result") {
36+                    if (!param && field_name == kj::StringPtr{"result"}) {
37                         field.retval = true;
38                         has_result = true;
39                     }

ryanofsky commented at 4:37 pm on August 12, 2025: contributor

IPC functionality seems to work fine on ubuntu 22.04 apart from this bug, so I guess I will create a PR with the workaround from #33176 (comment).

I posted a comment about this upstream https://github.com/capnproto/capnproto/pull/1170#discussion_r2270385770.

This issue does not occur with Cap’n Proto 0.9.0 and later due to the changes in that PR which added c++20 support. In theory a fix could be backported, but ubuntu 22.04 is still using 0.8.0, not even 0.8.1 which fixes a CVE, so it seems best not to rely on ubuntu packaging.

1BitcoinBoWP1FZ4xwTNkq6XksKidmgYYw commented at 5:41 pm on August 12, 2025: none

@ryanofsky

This issue does not occur with Cap’n Proto 0.9.0 and later

I oppose merging any workaround for this issue since:

The root cause has been resolved upstream (no need for Bitcoin-specific fixes)
We should instead implement a targeted warning advising users to upgrade their Ubuntu distribution

Any additional workaround would:

Unnecessarily complicate the codebase
Provide no benefit for modern Ubuntu versions
Introduce maintenance burden without solving core issues

Let’s keep Bitcoin Core clean by only merging changes that:

Address actual problems (not upstream-fixed issues)
Provide clear benefits across environments
Avoid version-specific pollution

Sjors commented at 6:22 pm on August 12, 2025: member

I’m not sure if we should bother with a workaround for a version that also has a CVE. The compiler failure might be a “feature” here.

ryanofsky commented at 8:40 pm on August 12, 2025: contributor

I guess it seems ok to drop support for old versions, but this also seems like an easy problem to fix so I went ahead and implemented a fix in https://github.com/bitcoin-core/libmultiprocess/pull/194 along with a CI job that tests for compatibility.

I’m not sure if we should bother with a workaround for a version that also has a CVE. The compiler failure might be a “feature” here.

To be sure, the fact that the stable ubuntu package might have a CVE is eyebrow raising, but I’m not it actually does. I don’t know very much about ubuntu and debian packaging and I just looked at the tarballs and patches at the bottom of https://packages.ubuntu.com/source/jammy/capnproto and noticed they seemed to be using an unpatched 0.8.0. The fix in https://github.com/bitcoin-core/libmultiprocess/pull/194 also lets mpgen work with 0.8.1 and 0.7.1 and is not specifically targeted at 0.8.0.

1BitcoinBoWP1FZ4xwTNkq6XksKidmgYYw commented at 9:05 pm on August 12, 2025: none

@ryanofsky,

Regarding the Cap’n Proto dependency, the correct source link is:
https://launchpad.net/ubuntu/+source/capnproto

After reviewing PR #194 (https://github.com/bitcoin-core/libmultiprocess/pull/194), I have some concerns:

Maintenance Policy:
We generally shouldn’t maintain patches for older OS versions unless there’s a compelling security reason. Supporting outdated systems creates long-term maintenance burden.
Code Origin:
Some changes appear to be copied directly from upstream.
Better Alternatives:
Users running older systems should either:
- Upgrade their OS to supported versions
- Build dependencies from source

Or, download and install binaries directly from https://launchpad.net/ubuntu/+source/capnproto:

ryanofsky commented at 10:06 pm on August 12, 2025: contributor

Regarding the Cap’n Proto dependency, the correct source link is: https://launchpad.net/ubuntu/+source/capnproto

Thanks! If I follow that link to https://launchpad.net/ubuntu/+source/capnproto/0.8.0-2ubuntu2 I see same tarballs that I checked previously at https://packages.ubuntu.com/source/jammy/capnproto, so this may be additional confirmation.

Re your concerns:

1. **Maintenance Policy**:

To be sure https://github.com/bitcoin-core/libmultiprocess/pull/194 isn’t adding a patch that needs to be maintained, it’s just comparing strings in a different way that’s more verbose but more backwards compatible. I don’t think the changes there will create a maintenance burden. I actually think to the contrary, the CI improvements there will help with long term maintenance, by making it easier to swap capnproto versions for testing and by catching any changes that are unintentionally backwards compatible.

2. **Code Origin**:
   Some changes appear to be copied directly from upstream.

I don’t think this is true, but I’d welcome you to review the PR and point out any problems there that I might be missing.

3. **Better Alternatives**:

I do agree updating the OS or building cap’nproto from source could be good alternatives if anyone runs across this bug. Setting -DENABLE_IPC=off could be another good alternative in many cases. I still think fixing the bug has merits though so would welcome review in the PR.

Sjors commented at 9:13 am on August 13, 2025: member

The fix in https://github.com/bitcoin-core/libmultiprocess/pull/194 also lets mpgen work with 0.8.1 and 0.7.1 and is not specifically targeted at 0.8.0.

Ok, that seems a good enough reason, because we might want to support those versions (even if not immediately). @1BitcoinBoWP1FZ4xwTNkq6XksKidmgYYw wrote:

We generally shouldn’t maintain patches for older OS versions unless there’s a compelling security reason.

Bitcoin Core generally supports fairly old OS versions. There’s pros and cons to that (and it’s certainly a pain), but that’s a more general discussion.

ryanofsky commented at 12:23 pm on August 13, 2025: contributor

Latest version of https://github.com/bitcoin-core/libmultiprocess/pull/194 should trigger an error on ubuntu 22.04 instead of hanging:

0CMake Error at CMakeLists.txt:32 (message):
1  Cap'n Proto 0.8.0 is affected by CVE-2022-46149.
2
3  Please install an updated package.
4
5  Details: https://github.com/advisories/GHSA-qqff-4vw4-f6hx

ryanofsky commented at 0:37 am on August 18, 2025: contributor

I didn’t dig into this but it seems like the root cause of infinite recursion might be a gcc bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53499 so using clang could be another workaround for this. The gcc bug is also fixed in gcc 14 according the issue. Ubuntu 22.04 uses gcc 11.4.0.

(I stumbled across this seeing a headline about a gcc bug https://osec.io/blog/2025-08-11-compiler-bug-causes-compiler-bug https://news.ycombinator.com/item?id=44872584)

build: indefinite mpgen hang on Ubuntu 22.04 #33176