build: Explore building with musl as libc #18110

issue dongcarl openend this issue on February 10, 2020
  1. dongcarl commented at 8:37 pm on February 10, 2020: contributor

    This has been discussed briefly with theuni and fanquake in the past, but I want to put it out there as an issue for more general discussion. I recommend reading through musl’s project introduction here. Reproduced:

    musl provides consistent quality and implementation behavior from tiny embedded systems to full-fledged servers. Minimal machine-specific code means less chance of breakage on minority architectures and better success with “write once run everywhere” C development.

    musl’s efficiency is unparalleled in Linux libc implementations. Designed from the ground up for static linking, musl carefully avoids pulling in large amounts of code or data that the application will not use. Dynamic linking is also efficient; by integrating the entire standard library implementation, including threads, math, and even the dynamic linker itself into a single shared object, most of the startup time and memory overhead of dynamic linking have been eliminated.

    musl features the first post-NPTL implementation of POSIX threads for Linux, and the first aimed at complete conformance and robustness. Thread cancellation has been re-designed to avoid serious race conditions in the original NPTL design. As for efficiency, the whole threads implementation weighs in at around 10-20k depending on target architecture and compiler settings.

    Not only the threads implementation, but all code in musl has been designed for realtime-quality robustness. Low-memory or resource exhaustion conditions are never fatal. musl has no unnecessary dynamic allocation and no unrecoverable late failures. All error conditions can be detected and handled by applications; interfaces for which an application could not reasonably handle failure do not fail.

    Using musl maximizes application deployability. Its permissive MIT license is compatible with all FOSS licenses, static-linking-friendly, and makes commercial use painless too. Binaries statically linked with musl have no external dependencies, even for features like DNS lookups or character set conversions that are implemented with dynamic loading on glibc. An application can really be deployed as a single binary file and run on any machine with the appropriate instruction set architecture and Linux kernel or Linux syscall ABI emulation layer.

    Finally, musl has simple source code and source tree layout, so it’s easy to customize or track down the cause of unexpected behavior or bugs, or simply learn how the library works.

    A few more resources for people to consider:

    In-depth discussion of some design decisions: https://wiki.musl-libc.org/design-concepts.html A comparison of different libcs: http://www.etalabs.net/compare_libcs.html Comprehensive compatibility information: https://wiki.musl-libc.org/compatibility.html FAQ: https://www.musl-libc.org/faq.html

    Perhaps being able to build fully-static Bitcoin Core binaries is a worthwhile features, as I’ve heard that glibc supports static builds in theory, but has many quirks in practice.

  2. dongcarl added the label Feature on Feb 10, 2020
  3. dongcarl added the label Brainstorming on Feb 10, 2020
  4. dongcarl added the label Build system on Feb 10, 2020
  5. maflcko commented at 9:02 pm on February 10, 2020: member

    See also http://gnusha.org/bitcoin-builds/2019-11-20.log

     005:00 < MarcoFalke> So how would I build Bitcoin Core against musl?
     105:00 < fanquake> Spin up an Alpine container?
     205:02 < fanquake> https://github.com/fanquake/core-review/blob/master/docker/alpine.dockerfile
     305:02 < MarcoFalke> Ok, I guess I should make the ci system work with alpine
     405:02 < MarcoFalke> Can I do it in Ubuntu as well?
     505:02 < MarcoFalke> [08:02] <wumpus> I suppose it's pretty much impossible to do that on ubuntu ,though
     605:03 < fanquake> Heh, maybe using that wrapper
     705:04 < MarcoFalke> Yeah
     805:04 < wumpus> TIL there's a wrapper, this is very cool
     905:14 < wumpus> it looks like the wrapper only works for C, not C++
    1005:16 < wumpus> it's super easy to get to work, installing from source takes 45 seconds or so on my old desktop; but it's no help for bitcoin if it's C only, there's no musl-g++ wrapper
    1105:23 < wumpus> "Be aware that, "out of the box", the wrapper only supports C applications, not C++. This is because the C++ libraries and headers are missing from the musl include/library path. The existing libstdc++ is actually compatible with musl in most cases and could be used by copying it into the musl library path, but the C++ header files are usually not compatible. One option may be rebuilding just
    1205:23 < wumpus> libstdc++ against musl; however, if C++ support is needed, it's recommended just to build a native toolchain targetting musl. "
    1305:35 < wumpus> I guess there's no way around using something like alpine linux
    1406:22 < wumpus> effetctively, building C++ for musl requires a full cross toolchain; using a cross-compilation toolchain for the native platform, might be advantageous from a determinism perspective though
    1506:26 < wumpus> insulated the build from what happens to be installed on the system
    1607:04 < wumpus> dongcarl cfields what do you think about switching linux builds to musl libc static builds?
    1707:10 < wumpus> it has the advantage that the binaries work pretty much everywhere instead of only on a limited range of libc versions, and it's starting to be untenable to maintain libc compatibility this far back
    1807:17 < dongcarl> not opposed... with Guix we won't have to maintain libc compatibility, since we won't be forced to upgrade libc version by ubuntu... but I do see the binary works everywhere advantage
    1907:18 < wumpus> sure, you can build against a old libc version instead of the libc-compat workaround, but you'd have just as much trouble supporting getauxval
    2007:19 < wumpus> it doesn't exist in 2.11 no matter how you build against it
    
  6. laanwj commented at 12:41 pm on February 11, 2020: member
    I’m not sure how this is going to interact with, say, the GUI (desktop/system integration is pretty much at odds with static executables). But definitely worth investigating.
  7. maflcko commented at 12:51 pm on February 11, 2020: member
    Similar to the CentOS build in the ci system, we could have an alpine build
  8. dongcarl commented at 8:43 pm on February 11, 2020: contributor

    Similar to the CentOS build in the ci system, we could have an alpine build

    That’s probably a good first step if it doesn’t slow down the ci system. Are you thinking just add another runner that builds with alpine and runs the tests?

  9. dongcarl commented at 8:45 pm on February 11, 2020: contributor

    I’m not sure how this is going to interact with, say, the GUI (desktop/system integration is pretty much at odds with static executables). But definitely worth investigating. @laanwj Ah yes good point. I don’t think it’d be too weird if we end up supporting building everything other than bitcoin-qt statically with musl?

  10. dongcarl commented at 8:51 pm on December 25, 2020: contributor

    Did an experimental x86_64-native build of v0.20.1 (bitcoind-only) with alpine’s musl-based toolchain…

    The configure line that worked:

    0env CONFIG_SITE=$PWD/depends/x86_64-pc-linux-musl/share/config.site \
    1    ./configure --disable-ccache \
    2                --disable-maintainer-mode \
    3                --disable-dependency-tracking \
    4                --enable-reduce-exports --disable-bench \
    5                --disable-gui-tests \
    6                CC=clang CXX=clang++ \
    7                CFLAGS="-O2 -g0 --static -static -fPIC" \
    8                CXXFLAGS="-O2 -g0 --static -static -fPIC" \
    9                LDFLAGS="-s -static-libgcc -static-libstdc++ -Wl,-O2"
    
    0~/bitcoin # file src/bitcoind
    1src/bitcoind: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
    

    Size-wise, it looks doable (at least for bitcoind itself):

    • 10,132 KiB: musl-built fully-static bitcoind v0.20.1
    • 10,072 KiB: release bitcoind v0.20.1
  11. laanwj commented at 11:34 pm on January 2, 2021: member

    Nice work!

    It would be interesting to see benchmarks of sync performance. I don’t expect that much of a difference, but it could go either way. The glibc allocator is quite efficient, I think musl’s is somewhat simpler and slower in some cases. On the other hand, static linking can reduce call overhead (especially wih LTO, but that’s a matter for another time).

    @laanwj Ah yes good point. I don’t think it’d be too weird if we end up supporting building everything other than bitcoin-qt statically with musl?

    I don’t see a strong reason why not. Having some more requirements on the system (with regard to installed libraries, and e.g. fonts) for the GUI than for the other tools is not unreasonable. Thinking of it it’s very little different from how things are now.

  12. laanwj commented at 6:34 pm on January 28, 2021: member

    I was thinking about this a bit again in context of #21022 and I think having a static bitcoind that runs directly on top of the kernel and doesn’t depend on any libraries that happen to be on the system would be a good aim, at least for the release binaries. It removes a aource of variability and possible bugs. It would allow getting rid of all the glibc compatibility hacks.

    Also it would also make packaging into containers, app images, and VMs and such easier. Basically you’d need only the binary and that’s that.

  13. lestephane commented at 10:17 am on January 30, 2021: none

    This would help running bitcoind on a 512M VPS KVM slice, which I’ve been trying (and failing) for a while. Every little bit helps. I’ve started looking into Guix to build an image iso that does nothing but run bitcoind with all non-essential features disabled. I’ll be watching this issue, and would love to know if anyone else is trying to do the same as me. We need more full nodes.

    On Thu, Jan 28, 2021 at 8:35 PM Wladimir J. van der Laan < notifications@github.com> wrote:

    I was thinking about this a bit again in context of #21022 https://github.com/bitcoin/bitcoin/pull/21022 and I think having a static bitcoind that runs directly on top of the kernel and doesn’t depend on any libraries that happen to be on the system would be a good aim, at least for the release binaries. It removes a aource of variability and possible bugs. It would allow getting rid of all the glibc compatibility hacks.

    Also it would also make packaging into containers, app images, and VMs and such easier. Basically you’d need only the binary and that’s that.

    — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/18110#issuecomment-769289338, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJVHFATSD4CI7YY3XEGOGTS4GU53ANCNFSM4KSVFCWA .

  14. laanwj added this to the milestone 23.0 on Jun 21, 2021
  15. laanwj commented at 9:56 am on June 21, 2021: member
    Adding 23.0 milestone optimistically after discussion on IRC.
  16. dongcarl commented at 6:30 pm on June 21, 2021: contributor

    Looking at the Guix mailing lists… There was some considerable work done for a musl-cross toolchain by @daym as part of a project to add Heads to Guix: https://issues.guix.gnu.org/37466

    Would be great to hear from @daym about the stumbling blocks.

  17. maflcko removed this from the milestone 23.0 on Jan 20, 2022
  18. maflcko commented at 7:20 pm on January 20, 2022: member
    Removing milestone again, as #23203 is POC and not a blocker for 23.0
  19. pinheadmz commented at 2:29 pm on April 27, 2023: member
    This issue is unlikely to be fixed in Bitcoin Core. We’ll close for now, but feel free to open another issue or pull request with a fix.
  20. pinheadmz closed this on Apr 27, 2023

  21. fanquake commented at 1:48 pm on May 1, 2023: member
    I think the intial route we’ll take for static builds will be to use glibc.
  22. bitcoin locked this on Apr 30, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-11-23 09:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me