gitian: GLIBC_2.29 not found error at bitcoind startup on bionic #21454

issue willyko opened this issue on March 16, 2021

willyko commented at 10:44 PM on March 16, 2021: contributor
bitcoind & bitcoin-qt built from gitian on master branch have not been working on Bionic for the last month or so.
The bitcoind works on Focal though. Following @jonasschnelli 's nightly gitian build, the last working bitcoind binary was dated January 28 on commit d1ddead0

Expected behavior bitcoind to run

Actual behavior
```
$ ./bitcoind
bitcoind: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by bitcoind)
bitcoind: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by bitcoind)
```
To reproduce On Ubuntu Bionic, download any of the bitcoin-*-x86_64-linux-gnu.tar.gz after January 28th and run the bitcoind

System information The binaries are downloaded from https://bitcoin.jonasschnelli.ch/gitian/?show=nightly#nighly I ran the gitian build myself and hit the same error too. The binaries work on Focal but not on Bionic. Did not test on other Ubuntu releases
willyko added the label Bug on Mar 16, 2021
fanquake added the label Build system on Mar 16, 2021
MarcoFalke added this to the milestone 22.0 on Mar 17, 2021
MarcoFalke commented at 1:19 PM on March 17, 2021: member

6a726cb534ed471010bf3f466732874b2a1b22e5 is from the 28th
MarcoFalke commented at 1:25 PM on March 17, 2021: member

guix is also affected
sidhujag referenced this in commit e44fc5fe6a on Mar 17, 2021

hebasto commented at 2:01 PM on March 18, 2021: member

FWIW,

$ lsb_release -d
Description:	Ubuntu 18.04.5 LTS
$ ldd --version | head -1
ldd (Ubuntu GLIBC 2.27-3ubuntu1.4) 2.27

hebasto commented at 5:45 PM on March 18, 2021: member

6a726cb is from the 28th

Just tested the gitian build of 6a726cb534ed471010bf3f466732874b2a1b22e5 on bionic. It works flawlessly.
sipa commented at 5:46 PM on March 18, 2021: member

@jonasschnelli How are your nightlies built? Do they use gitian?
willyko commented at 6:12 PM on March 18, 2021: contributor

There were no nightly build from January 29 - February 10, and it looks like some builds were failing due to No space left on device error according to the log. I was able to gitian build and run on bionic all the way up to https://github.com/bitcoin/bitcoin/commit/c969ab43c3888b1cd8e691484f2dc32a607d050f and that's the commit right before the switch to focal for gitian vm
hebasto commented at 10:23 PM on March 18, 2021: member

I can confirm that issue is caused by ca85449f22cb909b9ca83861aa51fbf8f40a9070 (#21036 merging) -- gitian builds work fine before this commit.

fanquake commented at 12:19 AM on March 19, 2021: member

The offending symbols are:

0000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  log
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  pow
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  exp
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.28  fcntl64

hebasto commented at 1:31 AM on March 19, 2021: member

The offending symbols are:

Should they be substituted in glibc_compat.cpp?

fanquake commented at 1:47 AM on March 19, 2021: member

I was confused as to how this could have happened, because our gitian/guix builds (symbol checks) all should have failed in #21036 if there were newer-than-allowed symbols being introduced. The issue seems to be that Pixie is incorrectly returning None, rather than a symbol version quite often. i.e with:

index 436f179d6..831346c5b 100755
--- a/contrib/devtools/symbol-check.py
+++ b/contrib/devtools/symbol-check.py
@@ -174,6 +174,7 @@ def check_imported_symbols(filename) -> bool:
             continue
         sym = symbol.name.decode()
         version = symbol.version.decode() if symbol.version is not None else None
+        print(sym, version)
         if version and not check_version(MAX_VERSIONS, version, elf.hdr.e_machine):
             print('{}: symbol {} from unsupported version {}'.format(filename, cppfilt(sym), version))
             ok = False

I'm seeing this:

<trim>
fcntl64 None
memmove None
pthread_rwlock_unlock None
realloc None
__fprintf_chk None
fseeko64 None
pthread_once None
_Unwind_GetRegionStart None
__pthread_key_create None
getgrgid None
fchown None
__tls_get_addr GLIBC_2.3
__memcpy_chk None
_Exit None
open64 None
freeifaddrs None
pow None
_ITM_deregisterTMCloneTable None
strpbrk None
pthread_getschedparam None
_Unwind_Resume None
timerfd_create None
getauxval None
getprotobynumber None
strcat None
mmap None
umask None
setsockopt None
getpgid None
exp None
_Unwind_SetIP None
<trim>

when running python3 bitcoin/contrib/devtools/symbol-check.py bitcoin-ee0dc02c6f93/bin/bitcoind. Which is against a recent nightly build downloaded from bitcoin builds.

sidhujag commented at 2:03 AM on March 19, 2021: none
The offending symbols are:

0000000000000000 DF *UND* 0000000000000000 GLIBC_2.29 log 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.29 pow 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.29 exp 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.28 fcntl64
Besides the point but pow on <= 2.28 is 10k times slower (x86-64 and others) if you don't need double precision you can use powf(), I found this seemed to fix the unresolved export for focal in our code. pow() is used in traffic widget, CTxMemPool::GetMinFee and addrman (GetChance) so not particularly sensitive imo.

https://man7.org/linux/man-pages/man3/pow.3.html
laanwj commented at 1:09 PM on June 14, 2021: member

The issue seems to be that Pixie is incorrectly returning None, rather than a symbol version quite often. i.e with:

Interesting. Will take a look at the pixie issue. See #22244.
laanwj referenced this in commit 19e598bab0 on Jun 14, 2021
hebasto commented at 12:08 AM on June 20, 2021: member

bitcoind compatibility fixed in #22281 and #22287. @willyko Do you mind testing suggested fixes?
hebasto referenced this in commit bdaff07b32 on Jun 20, 2021
laanwj referenced this in commit a305a687e7 on Jun 21, 2021
rebroad referenced this in commit 537ea316f9 on Jun 23, 2021
sidhujag referenced this in commit 5c4ff037a8 on Jun 24, 2021
rebroad referenced this in commit ddcc39d4c5 on Jun 28, 2021
hebasto commented at 6:41 PM on July 5, 2021: member

Since #22365 has been merged, this could be closed, no?
laanwj closed this on Jul 5, 2021
josibake referenced this in commit 5ab3b2c499 on Jul 21, 2021
luke-jr referenced this in commit 8991a13c02 on Jul 26, 2021
luke-jr referenced this in commit 78df148816 on Jul 28, 2021
DrahtBot locked this on Aug 18, 2022

Contributors

Labels

Milestone
22.0

Linked (view graph)

#22244 devtools: Correctly extract symbol versions in symbol-check #22281 build: Avoid @GLIBC_2.29 libm symbols when --enable-glibc-back-compat #22287 build: Avoid fcntl64@GLIBC_2.28 symbol when --enable-glibc-back-compat #22305 build: Avoid fcntl64@GLIBC_2.28 in libsqlite3.a