Consider making 27.x Long-Term Support (LTS)

ArmchairCryptologist commented at 5:56 pm on October 10, 2024: none

As mentioned in the patch notes for 28.x, since this release drops support for glibc older than 2.31, it will not run on several widely used Linux distros, including RHEL 8 and the RHELatives that base off of this, such as AlmaLinux 8 and Rocky Linux 8.

In case anyone is unaware, RHEL 8 and its derivatives are still in active support, and will in fact not be EOL until 2029. While I don’t have exact install numbers, I would not be surprised if more than half of Bitcoin Core installations that run on dedicated Linux servers will not be able to update to 28.x as a result of this, without first updating to a new OS.

As such, while I’m aware that older release branches usually have important updates backported for some time, I feel it would make sense to “officially” designate 27.x as a LTS with an extended window of backported updates - maybe not for the full four years and change until these OSes are EOL, but at least for a good part of it. Otherwise, considering the new security disclosure policy, I fear there will be a lot of clients with published exploits still running when 27.x advisories start getting published.

ArmchairCryptologist added the label Feature on Oct 10, 2024

maflcko commented at 6:05 pm on October 10, 2024: member

My understanding is that RHEL-8-like are still supported, if you manage to run a recent enough compiler on them and use that to compile Bitcoin Core yourself.

bitcoin deleted a comment on Oct 11, 2024

willcl-ark commented at 9:22 am on October 11, 2024: member

RHEL 8 and the RHELatives that base off of this, such as AlmaLinux 8 and Rocky Linux 8.

I would not be surprised if more than half of Bitcoin Core installations that run on dedicated Linux servers will not be able to update to 28.x as a result of this, without first updating to a new OS.

Do you have any further information on how you conclude that more than half of Bitcoin Core installations run on these OSes (and their derivatives)? This number doesn’t feel right to me, even as a (high) estimate.

This report: https://repology.org/project/glibc/badges shows that glibc >=2.31 is supported by:

Debian 11, 12, 13
Fedora 32 - 40
Gentoo
Homebrew
Manjaro stable
nixpkgs
openSUSE > 15.3
Slackware current
Ubuntu >= 20.04

Whilst it is a shame that RHEL 8 doesn’t support it, it also doesn’t feel practical for this project to support a LTS version for so long (and only for this distro) because of the nature of the patches that may require backport.

For example, whilst backporting critical security vulnerabilites may be practical in most cases, as these are often logic bugs or edge cases, backporting things like (potential) soft fork activation logic could prove to be much more work. And, unless someone volunteers to take this on, then, in my opinion, I don’t think we have the resources to commit to that as a project.

Incidentally, 27.x will be in maintenance support for another ~12 months or so already.

I will leave my self open to being corrected on the usage metrics of RHEL 8 as a node OS, so in the case that there are, as you claim, a very large number of users using that OS, then I would happily reconsider my above opinion.

Another approach would be to find a volunteer who would commit to maintaining a seperate “Long Term Support source tree”, providing LTS for RHEL 8. But with this approach you begin to lose some aspects of the security/review process which the mainline Bitcoin Core project brings to the table.

maflcko commented at 11:15 am on October 11, 2024: member

If this really was an issue, adding back the build patches would probably be easier than backporting all bugfixes for years: https://github.com/bitcoin/bitcoin/commit/0c57a798b50bf03a0b69f632c691932d608254ef#diff-42e88f6b2dc3f513ccde184c74d1853a01b2bfc3ee64debf98c52efe316b04c1L456

However, I still don’t understand why this is an issue.

Maybe I am missing something obvious, but at least locally I can call ./configure on the 28.x branch just fine on Alma8:

0dnf install gcc-toolset-13 -y
1( cd depends && make CC=/opt/rh/gcc-toolset-13/root/usr/bin/gcc CXX=/opt/rh/gcc-toolset-13/root/usr/bin/g++ NO_QT=1 NO_BDB=1 NO_ZMQ=1 NO_UPNP=1 NO_NATPMP=1 -j $(nproc) )
2./autogen.sh && CONFIG_SITE="$PWD/depends/x86_64-pc-linux-gnu/share/config.site" ./configure
3make clean && make -j$( nproc )

I didn’t compile and run the unit and functional tests, but I don’t see why that shouldn’t work.

ArmchairCryptologist commented at 12:45 pm on October 11, 2024: none

Do you have any further information on how you conclude that more than half of Bitcoin Core installations run on these OSes (and their derivatives)? This number doesn’t feel right to me, even as a (high) estimate. This report: https://repology.org/project/glibc/badges shows that glibc >=2.31 is supported by:

It is mostly a hunch based on my own experience with the server hosting industry. Most of the distros mentioned here are not generally considered stable for production/server usage, and are only offered as unsupported custom installs. I rarely see free Linux distros except for AlmaLinux, Rocky Linux, Ubuntu and Debian offered by default, and being the default, ~99% of servers will be deployed using it. While Ubuntu 18 and Debian 10 are now EOL, that still leaves RHEL and the derivatives.

I don’t have hard installation numbers between the four of course, but after Red Hat was eaten by IBM and CentOS 8 was gutted, I have mostly seen the server hosting industry defaulting to AlmaLinux 8 and Rocky Linux 8 in the last few years, and only fairly recently begun defaulting to AlmaLinux 9 and Rocky Linux 9. Considering that in-place upgrades from (RHEL/AlmaLinux/Rocky Linux) 8 to 9 are not generally supported, I would expect the majority of servers and VPSes that were installed with any of the RHELatives to be running on the 8 branch for the foreseeable future.

However, I still don’t understand why this is an issue. Maybe I am missing something obvious, but at least locally I can call ./configure on the 28.x branch just fine on Alma8:

I’m not really sure why glibc 2.28 support was dropped from the official binaries if it still trivially compiles against it. I looked through the commit that changed this, and it mostly seems to be referring to future improvements.

sipa commented at 12:49 pm on October 11, 2024: member

If there is significant demand for running Bitcoin Core on RH8, I think it would make sense for us to consider adding the glibc workarounds necessary so that guix-built binaries support glibc 2.28. I suspect that is easier than backporting fixes to 27.x for longer, and doesn’t preclude RH8 users from getting the features added in 28.x and later.

I don’t think the possibility of self-compiling on RH8 is a full replacement for project binaries that just work. Presumably people who use RH8 want to benefit from the assurance reproducible builds have, just like other users.

fanquake commented at 12:49 pm on October 11, 2024: member

if it still trivially compiles against it.

Having to maintain support for compiling well EOL glibcs is not trivial, and continues to get harder as the compilers and toolchains used by the project modernise. It’s already the case that on the next update to our release (Guix) build environment, our current glibc (2.31), will stop compiling without further patching, because the newer binutils (2.41), can’t compile it when targeting riscv.

Having to compile our release binaries against old glibcs also prevents us from properly implementing/using modern hardening features.

sipa commented at 1:01 pm on October 11, 2024: member

@ArmchairCryptologist There is a difference between (1) our source code being able to be compiled against an old glibc, and then working on that old glibc and (2) the release process being able to run on modern platforms (and making use of the features that offers, including targetting modern hardware) and the resulting binaries working on old systems.

Of course, one possibility would be to have multiple release builds, for example a separate one targetting old-glibc x86_64 systems only, but that has the downside of reducing the testing our release binaries get (especially if such an old-glibc build is only used by a small minority of users). So far, we have instead just had a single binary per hardware platform, compiled against modern glibc, but with specific workarounds to permit the binary to still support old glibc. It is a balancing question on how much work is warranted to test and maintain these workarounds.

maflcko commented at 1:14 pm on October 11, 2024: member

for example a separate one targetting old-glibc x86_64

Looking at the patches, it may be possible to do a x86_64 guix build (maybe even aarch64) with glibc2.28, without adding any patches back, as they relate to other architectures. However, I haven’t tried this and don’t know how involved it would be.

edit: However, someone may also complain that support for Ubuntu 18.04 was dropped. With “Ubuntu Pro” support will end in a few years only. So if an old-glibc build is added, may as well go with 2.27?

sipa commented at 1:24 pm on October 11, 2024: member

@maflcko I assume you mean glibc 2.28 and 2.27? What would be the downside of only having glibc 2.27 release binaries for x86_64 (and/or aarch64), as opposed to having a separate old-glibc build?

fanquake commented at 2:24 pm on October 11, 2024: member

What would be the downside of only having glibc 2.27 release binaries for x86_64 (and/or aarch64), as opposed to having a separate old-glibc build?

If what you’re suggesting is : x86_64-linux-gnu: 2.27 arm-linux-gnueabihf: 2.modern aarch64-linux-gnu: 2.27 riscv64-linux-gnu: 2.modern powerpc64-linux-gnu: 2.modern powerpc64le-linux-gnu: 2.modern (when re-enabled)

Some might be:

We can’t ship modern (hardening) features to users running x86_64 or aarch64. i.e #24123, #30685, because they aren’t compatible with the older glibc (and I don’t think we should be trying to backport these patches to make that work).
We might have to maintain (possibly our own, as may not exist upstream) patches for glibc so that we can continue to compile it using a modern toolchain.
We’ll have to maintain a more-complex Guix environment, as we’ll have multiple versions of glibc, as well as potentially have to keep versions of tools pinned, so that we can build the older glibc as opposed to using the versions that are natively shipping with our time machine.

sipa commented at 2:27 pm on October 11, 2024: member

@fanquake Fair points, though your second and third point also apply to having a separate old-glibc build (in addition to having modern x86_64/aarch64 builds).

Consider making 27.x Long-Term Support (LTS) #31068