[POC] build: static musl libc based bitcoind #23203

pull fanquake wants to merge 1 commits into bitcoin:master from fanquake:depends_musl_cross_make changing 6 files +78 −0
  1. fanquake commented at 7:21 am on October 6, 2021: member

    This proof of concept is one approach to static musl libc based builds. It leverages musl-cross-make to compile a musl libc targeting toolchain, made up of GCC 10.3.0 + binutils 2.37 + musl libc 1.2.3, and then uses that toolchain to build our depends libraries, and ultimately bitcoind.

    More than likely we’d take a different approach in Guix for (potentially) release builds, however, something like this could exist in depends, and provide an alternative to building against glibc outside Guix. Or, this may well end up being something we just throw away in favour of a better approach; still lots to be thought about here.

    Only tested compiling on, for x86_64-pc-linux-gnu so far.

    You can use it as follows:

    0make -C depends/ -j9 NO_QT=1 NO_WALLET=1 NO_UPNP=1 NO_ZMQ=1 NO_NATPMP=1 USE_MUSL_LIBC=1 HOST=x86_64-linux-musl
    1./autogen.sh
    2CONFIG_SITE=/path/to/your/bitcoin/depends/x86_64-pc-linux-musl/share/config.site ./configure --enable-reduce-exports --enable-lto LDFLAGS="-Wl,-O2" LIBTOOL_APP_LDFLAGS="-all-static"
    3make src/bitcoind
    

    The end result (once stripped) is a ~10mb static bitcoind:

    0ldd src/bitcoind
    1	statically linked
    

    If you see:

    0terminate called after throwing an instance of 'std::runtime_error'
    1  what():  locale::facet::_S_create_c_locale name not valid
    2Aborted (core dumped)
    

    when running, pass LC_ALL=C. i.e LC_ALL=C src/bitcoind.

    I’ve taken the binary onto a few different system / inside VMs, and am running a full sync.

    Note that all dependencies do compile, except for Qt (build issue with FreeType I have not investigated), I’ve just excluded all optional dependencies from the depends build above.

    When building, you may see warnings about libraries, such as libstdc++, as having been moved, during linking, and/or other warnings, from dependencies, such as sqlite, due to -flto usage.

    Related to #18110.

  2. fanquake added the label Brainstorming on Oct 6, 2021
  3. fanquake added the label Build system on Oct 6, 2021
  4. laanwj commented at 11:04 am on October 6, 2021: member

    Cocnept ACK, this is great!

    Note that all dependencies do compile, except for Qt (build issue with FreeType I have not investigated), I’ve just excluded all optional dependencies from the depends build above.

    Qt is going to be a big challenge for full-static linking (due to system dependencies for X, FreeType etc). It’s ok to skip that for now, IMO, would be nice to be able to ship a statically linked bitcoind and utilities at least.

  5. DrahtBot commented at 11:15 am on October 6, 2021: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #16545 (refactor: Implement missing error checking for ArgsManager flags by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  6. laanwj commented at 1:23 pm on October 6, 2021: member
    Looking at the depends build log, I think the custom toolchain isn’t correctly fed into the boost build. It’s using my system compiler: (snip—this was wrong)
  7. fanquake commented at 1:37 pm on October 6, 2021: member

    It’s using my system compiler:

    That excerpt is for the native b2 build, where using the system compiler is expected.

  8. laanwj commented at 1:56 pm on October 6, 2021: member

    I might have not properly cleaned my tree. Will retry a full build.

    Edit: whoops, that did it. Testing note make sure you start from a clean tree. If you get a linking error about boost::system::generic_category(), this is caused by pollution from previous builds.

  9. laanwj commented at 2:20 pm on October 6, 2021: member

    Great, the functional tests pass and so does test_bitcoin make check fails on the secp256k1 tests, but not because of any error, simply because they’re not built.

     0FAIL: tests
     1===========
     2
     3/home/user/src/bitcoin/src/secp256k1/build-aux/test-driver: line 107: ./tests: No such file or directory
     4FAIL tests (exit status: 127)
     5
     6FAIL: exhaustive_tests
     7======================
     8
     9/home/user/src/bitcoin/src/secp256k1/build-aux/test-driver: line 107: ./exhaustive_tests: No such file or directory
    10FAIL exhaustive_tests (exit status: 127)
    

    They do exist, but it’s unable to run them somehow;

    0~/src/bitcoin$ ls -al build/src/secp256k1/exhaustive_tests build/src/secp256k1/tests
    1-rwxr-xr-x 1 user user 128480 Oct  6 16:15 build/src/secp256k1/exhaustive_tests
    2-rwxr-xr-x 1 user user 466624 Oct  6 16:15 build/src/secp256k1/tests
    3~/src/bitcoin$ build/src/secp256k1/tests
    4bash: build/src/secp256k1/tests: No such file or directory
    5~/src/bitcoin$ file build/src/secp256k1/tests
    6build/src/secp256k1/tests: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, not stripped
    

    I think this is the problem: “interpreter /lib/ld-musl-x86_64.so.1”

  10. laanwj commented at 6:20 pm on October 6, 2021: member

    The build works as-is on RISC-V! Except I had to add --without-asm to prevent secp256k1 from including x86 assembly (failing while linking bitcoind etc). I think the problem is that the instruction set detection in secp256k1’s configure compiles a C file with __asm__ in it but doesn’t link it: the IR files likely contain the assembly code as text, and only in the link phase assembling is actually attempted. So it doesn’t detect that x86 assembly isn’t valid for the platform.

    Edit: also the tests segfault on RISC-V, no idea why yet. Ok, built with debug information, this is the traceback:

     0(gdb) bt
     1[#0](/bitcoin-bitcoin/0/)  0x0000000000000000 in ?? ()
     2[#1](/bitcoin-bitcoin/1/)  0x0000000000519e74 in std::__glibcxx_rwlock_rdlock (__rwlock=0xaebdd8 <_ZN12_GLOBAL__N_1L14signatureCacheE.lto_priv.0+304>) at /usr/include/c++/10/shared_mutex:73
     3[#2](/bitcoin-bitcoin/2/)  std::__shared_mutex_pthread::lock_shared (this=0xaebdd8 <_ZN12_GLOBAL__N_1L14signatureCacheE.lto_priv.0+304>) at /usr/include/c++/10/shared_mutex:224
     4[#3](/bitcoin-bitcoin/3/)  std::shared_mutex::lock_shared (this=<optimized out>, this=<optimized out>) at /usr/include/c++/10/shared_mutex:421
     5[#4](/bitcoin-bitcoin/4/)  std::shared_lock<std::shared_mutex>::shared_lock (__m=..., this=<optimized out>, this=<optimized out>, __m=...) at /usr/include/c++/10/shared_mutex:722
     6[#5](/bitcoin-bitcoin/5/)  (anonymous namespace)::CSignatureCache::Get (this=<optimized out>, erase=true, entry=...) at script/sigcache.cpp:69
     7[#6](/bitcoin-bitcoin/6/)  CachingTransactionSignatureChecker::VerifyECDSASignature (this=0x3ff5049998, vchSig=..., pubkey=..., sighash=...) at script/sigcache.cpp:109
     8[#7](/bitcoin-bitcoin/7/)  0x00000000005c6082 in GenericTransactionSignatureChecker<CTransaction>::CheckECDSASignature (this=0x3ff5049998, vchSigIn=..., vchPubKey=..., scriptCode=..., sigversion=<optimized out>) at script/interpreter.cpp:1688
     9[#8](/bitcoin-bitcoin/8/)  0x00000000006c49e2 in EvalChecksigPreTapscript (fSuccess=@0x3ff5049179: true, serror=0x3ff0000bcc, sigversion=<optimized out>, checker=..., flags=134673, pend=..., pbegincodehash=..., vchPubKey=..., vchSig=...) at script/interpreter.cpp:363
    10[#9](/bitcoin-bitcoin/9/)  EvalChecksig(std::vector<unsigned char, std::allocator<unsigned char> > const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, prevector<28u, unsigned char, unsigned int, int>::const_iterator, prevector<28u, unsigned char, unsigned int, int>::const_iterator, ScriptExecutionData&, unsigned int, BaseSignatureChecker const&, SigVersion, ScriptError_t*, bool&) [clone .constprop.0] (sig=..., pubkey=..., pbegincodehash=..., pend=..., execdata=..., flags=<optimized out>, checker=..., sigversion=<optimized out>,
    11    serror=0x3ff0000bcc, success=@0x3ff5049179: true) at script/interpreter.cpp:421
    12[#10](/bitcoin-bitcoin/10/) 0x00000000005cf5de in EvalScript (stack=..., script=..., flags=<optimized out>, checker=..., sigversion=<optimized out>, execdata=..., serror=0x3ff0000bcc) at script/interpreter.cpp:1094
    

    I’m not sure what this is. But I strongly suspect it’s not a bug in our code. It could be some kind of C++ ABI conflict (does it matter what C++ libarary version is on the system?), but also simply a RISC-V specific compilation / library /thread handling bug. We’ll know more once we try on for other platforms.

  11. yanmaani commented at 3:06 pm on October 8, 2021: none
    Concept ACK, always great to see people doing static builds.
  12. theStack commented at 10:38 am on October 11, 2021: contributor
    Concept ACK
  13. laanwj commented at 1:37 pm on October 11, 2021: member

    does it matter what C++ libarary version is on the system?

    FWIW I checked and there’s another copy of the C++ library headers under depends/riscv64-unknown-linux-gnu/native/x86_64-linux-musl/include/c++/10.3.0. I think it’s supposed to get shared_mutex from there. But also looked at the diff with my local one under /usr/include and there’s only a comment difference. So not sure this is the source of the issue.

  14. fanquake commented at 11:07 am on October 14, 2021: member

    I’ve reworked this PR a bit. It’s been rebased, and now builds on #20744 (using std::filesystem over boost::filesystem) and #23152 (--enable-lto). bitcoind is no longer linking against any Boost libs, only libevent, which should lessen the potential for build issues. I’ve also changed the native musl package to point to my fork, where I’ve added support for GCC 11.2 and binutils 2.37 (will PR upstream soon).

    Given that, I’ve updated the config here so that the toolchain we’re now using is GCC 11.2 + binutils 2.37 + musl 1.2.2.

    Currently the link invocation for a minimal bitcoind should be (removed duplicate options & shortened lib paths):

     0
     1# compiler
     2<path to>/x86_64-linux-musl-g++
     3
     4# compile options
     5-m64
     6-std=c++17
     7-fdebug-prefix-map=/home/ubuntu/bitcoin=.
     8-fstack-reuse=none
     9-Wstack-protector
    10-fstack-protector-all
    11-fcf-protection=full
    12-fstack-clash-protection
    13-flto
    14-flto-odr-type-merging
    15-fPIE
    16-pipe
    17-O2
    18-fno-extended-identifiers
    19-fvisibility=hidden
    20
    21# link options
    22-Wl,--exclude-libs -Wl,ALL
    23-Wl,-z -Wl,relro
    24-Wl,-z -Wl,now
    25-Wl,-z -Wl,separate-code
    26-pie
    27-static
    28-Wl,-O2 
    29-o bitcoind bitcoind-bitcoind.o init/bitcoind-bitcoind.o
    30
    31# depends lib search path
    32-L/home/ubuntu/bitcoin/depends/x86_64-pc-linux-gnu/lib
    33
    34# internal libs
    35libbitcoin_common.a
    36libbitcoin_consensus.a
    37libbitcoin_server.a
    38libbitcoin_util.a 
    39
    40crypto/libbitcoin_crypto_base.a
    41crypto/libbitcoin_crypto_sse41.a
    42crypto/libbitcoin_crypto_avx2.a 
    43crypto/libbitcoin_crypto_shani.a
    44
    45leveldb/libleveldb.a
    46leveldb/libmemenv.a
    47
    48crc32c/libcrc32c.a
    49crc32c/libcrc32c_sse42.a
    50
    51secp256k1/.libs/libsecp256k1.a
    52
    53univalue/.libs/libunivalue.a 
    54
    55# libstdc++
    56path/to/libstdc++/in/depends/libstdc++.a
    57
    58# external libs
    59-levent_pthreads
    60-levent
    61-pthread
    
  15. fanquake force-pushed on Oct 14, 2021
  16. practicalswift commented at 11:23 am on October 14, 2021: contributor

    @fanquake

    Nice work!

    It would be interesting to see the output of ldd src/bitcoind :)

  17. fanquake commented at 11:25 am on October 14, 2021: member

    @practicalswift

    It would be interesting to see the output of ldd src/bitcoind :)

    It’s:

    0$ ldd src/bitcoind
    1	statically linked
    
  18. DrahtBot added the label Needs rebase on Oct 15, 2021
  19. dongcarl commented at 2:19 pm on October 20, 2021: contributor
    It seems like musl-cross-make automagically downloads a bunch of repositories (gcc, etc)… Any way we can “pre-seed” that?
  20. theuni commented at 7:05 pm on October 20, 2021: member

    Nice! Playing around with this. Got it to build, great work.

    First observation (could be a red herring, take with a grain of salt) is the link line:

    x86_64-linux-musl-g++ ... /home/cory/dev/bitcoin2/depends/x86_64-pc-linux-gnu/native/bin/../lib/gcc/x86_64-linux-musl/11.2.0/../../../../x86_64-linux-musl/lib/libstdc++.a libbitcoin_consensus.a crypto/libbitcoin_crypto_base.a crypto/libbitcoin_crypto_sse41.a crypto/libbitcoin_crypto_avx2.a crypto/libbitcoin_crypto_shani.a leveldb/libleveldb.a crc32c/libcrc32c.a crc32c/libcrc32c_sse42.a leveldb/libmemenv.a secp256k1/.libs/libsecp256k1.a ...

    That static libstdc++.a seems weird and out of place, I assume libtool is throwing it in there.

    If omitted, g++ will add -lstdc++ in the correct place. Note that above it falls before a few c++ libs on the link-line, so I assume it’s the added -lstdc++ that’s doing the heavy lifting anyway.

    I’m mostly mentioning this because the two resulting binaries do differ. Though I haven’t looked into how or whether the difference is at all significant.

  21. fanquake force-pushed on Nov 25, 2021
  22. DrahtBot removed the label Needs rebase on Nov 25, 2021
  23. fanquake force-pushed on Feb 4, 2022
  24. fanquake commented at 1:36 pm on February 4, 2022: member

    Rebased this on master now that we are no-longer linking Boost libs, and changed to using GCC 10.3.0, so it, and binutils 2.37 match what we use for Guix builds. Also changed the depends variable to USE_MUSL_LIBC.

    Building with -Os, a fully static bitcoind is 8.6mb.

  25. jamesob commented at 4:46 pm on February 4, 2022: member
    Awesome, concept ACK! Will test soon.
  26. jamesob commented at 5:16 pm on February 4, 2022: member

    Hah, amazing.

    0src/bitcoin2 (efcf935) % du -hs ./src/bitcoind
    130M     ./src/bitcoind
    2
    3src/bitcoin2 (efcf935) % ldd ./src/bitcoind
    4        statically linked
    5
    6src/bitcoin2 (efcf935) % podman run --rm -it -v ./src/bitcoind:/bitcoind docker.io/library/alpine /bitcoind -version
    7Bitcoin Core version v22.99.0-efcf9355339f
    
  27. JeremyRubin commented at 6:01 pm on February 4, 2022: contributor

    Concept ACK!

    Can this work for clang too?

  28. DrahtBot added the label Needs rebase on Feb 14, 2022
  29. fanquake force-pushed on Feb 15, 2022
  30. DrahtBot removed the label Needs rebase on Feb 15, 2022
  31. fanquake force-pushed on Apr 28, 2022
  32. fanquake force-pushed on Apr 28, 2022
  33. fanquake force-pushed on Apr 28, 2022
  34. fanquake commented at 11:42 am on April 28, 2022: member

    A couple changes. Switched to upstream + a patch for binutils 2.37. Now using musl 1.2.3. Improved support for cross-compiling. Explicitly specified Linux headers (currently 4.19.88-1) in config.mak. We can add a patch here and sync up with Guix, likely to 5.15.28 post #25006.

    NOTE: you now need to set HOST when doing the depends build. i.e HOST=x86_64-linux-musl.

    Can this work for clang too?

    We should be able to do something similar with Clang.

  35. fanquake force-pushed on Jul 27, 2022
  36. fanquake marked this as a draft on Jul 27, 2022
  37. fanquake commented at 9:41 am on July 27, 2022: member
    Converting this to a draft. I still think using musl libc / this depends based approach is interesting, but will be focussing on #25573 and it’s dependant PRs for now.
  38. fanquake force-pushed on Sep 13, 2022
  39. fanquake renamed this:
    [POC] build: static musl libc based bitcoind (with LTO)
    [POC] build: static musl libc based bitcoind
    on Sep 13, 2022
  40. [POC] build: static, musl based bitcoind 79cb4f6b35
  41. fanquake force-pushed on Oct 2, 2022
  42. fanquake commented at 3:34 pm on October 2, 2022: member
    I will probably maintain this branch going forward, but going to close for now re my comment above.
  43. fanquake closed this on Oct 2, 2022

  44. bitcoin locked this on Oct 2, 2023

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-09-28 22:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me