gitian: GLIBC_2.29 not found error at bitcoind startup on bionic

willyko commented at 10:44 pm on March 16, 2021: contributor

bitcoind & bitcoin-qt built from gitian on master branch have not been working on Bionic for the last month or so.
The bitcoind works on Focal though. Following @jonasschnelli ’s nightly gitian build, the last working bitcoind binary was dated January 28 on commit d1ddead0

Expected behavior bitcoind to run

Actual behavior

0$ ./bitcoind
1bitcoind: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by bitcoind)
2bitcoind: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by bitcoind)

To reproduce On Ubuntu Bionic, download any of the bitcoin-*-x86_64-linux-gnu.tar.gz after January 28th and run the bitcoind

System information The binaries are downloaded from https://bitcoin.jonasschnelli.ch/gitian/?show=nightly#nighly I ran the gitian build myself and hit the same error too. The binaries work on Focal but not on Bionic. Did not test on other Ubuntu releases

willyko added the label Bug on Mar 16, 2021

fanquake added the label Build system on Mar 16, 2021

MarcoFalke added this to the milestone 22.0 on Mar 17, 2021

MarcoFalke commented at 1:19 pm on March 17, 2021: member

6a726cb534ed471010bf3f466732874b2a1b22e5 is from the 28th

MarcoFalke commented at 1:25 pm on March 17, 2021: member

guix is also affected

sidhujag referenced this in commit e44fc5fe6a on Mar 17, 2021

hebasto commented at 2:01 pm on March 18, 2021: member

FWIW,

0$ lsb_release -d
1Description:	Ubuntu 18.04.5 LTS
2$ ldd --version | head -1
3ldd (Ubuntu GLIBC 2.27-3ubuntu1.4) 2.27

hebasto commented at 5:45 pm on March 18, 2021: member

6a726cb is from the 28th

Just tested the gitian build of 6a726cb534ed471010bf3f466732874b2a1b22e5 on bionic. It works flawlessly.

sipa commented at 5:46 pm on March 18, 2021: member

@jonasschnelli How are your nightlies built? Do they use gitian?

willyko commented at 6:12 pm on March 18, 2021: contributor

There were no nightly build from January 29 - February 10, and it looks like some builds were failing due to No space left on device error according to the log. I was able to gitian build and run on bionic all the way up to https://github.com/bitcoin/bitcoin/commit/c969ab43c3888b1cd8e691484f2dc32a607d050f and that’s the commit right before the switch to focal for gitian vm

hebasto commented at 10:23 pm on March 18, 2021: member

I can confirm that issue is caused by ca85449f22cb909b9ca83861aa51fbf8f40a9070 (#21036 merging) – gitian builds work fine before this commit.

fanquake commented at 0:19 am on March 19, 2021: member

The offending symbols are:

00000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  log
10000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  pow
20000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  exp
30000000000000000      DF *UND*	0000000000000000  GLIBC_2.28  fcntl64

hebasto commented at 1:31 am on March 19, 2021: member

The offending symbols are:

Should they be substituted in glibc_compat.cpp?

fanquake commented at 1:47 am on March 19, 2021: member

I was confused as to how this could have happened, because our gitian/guix builds (symbol checks) all should have failed in #21036 if there were newer-than-allowed symbols being introduced. The issue seems to be that Pixie is incorrectly returning None, rather than a symbol version quite often. i.e with:

 0index 436f179d6..831346c5b 100755
 1--- a/contrib/devtools/symbol-check.py
 2+++ b/contrib/devtools/symbol-check.py
 3@@ -174,6 +174,7 @@ def check_imported_symbols(filename) -> bool:
 4             continue
 5         sym = symbol.name.decode()
 6         version = symbol.version.decode() if symbol.version is not None else None
 7+        print(sym, version)
 8         if version and not check_version(MAX_VERSIONS, version, elf.hdr.e_machine):
 9             print('{}: symbol {} from unsupported version {}'.format(filename, cppfilt(sym), version))
10             ok = False

I’m seeing this:

 0<trim>
 1fcntl64 None
 2memmove None
 3pthread_rwlock_unlock None
 4realloc None
 5__fprintf_chk None
 6fseeko64 None
 7pthread_once None
 8_Unwind_GetRegionStart None
 9__pthread_key_create None
10getgrgid None
11fchown None
12__tls_get_addr GLIBC_2.3
13__memcpy_chk None
14_Exit None
15open64 None
16freeifaddrs None
17pow None
18_ITM_deregisterTMCloneTable None
19strpbrk None
20pthread_getschedparam None
21_Unwind_Resume None
22timerfd_create None
23getauxval None
24getprotobynumber None
25strcat None
26mmap None
27umask None
28setsockopt None
29getpgid None
30exp None
31_Unwind_SetIP None
32<trim>

when running python3 bitcoin/contrib/devtools/symbol-check.py bitcoin-ee0dc02c6f93/bin/bitcoind. Which is against a recent nightly build downloaded from bitcoin builds.

sidhujag commented at 2:03 am on March 19, 2021: none

The offending symbols are:

00000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  log
10000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  pow
20000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  exp
30000000000000000      DF *UND*	0000000000000000  GLIBC_2.28  fcntl64

Besides the point but pow on <= 2.28 is 10k times slower (x86-64 and others) if you don’t need double precision you can use powf(), I found this seemed to fix the unresolved export for focal in our code. pow() is used in traffic widget, CTxMemPool::GetMinFee and addrman (GetChance) so not particularly sensitive imo.

https://man7.org/linux/man-pages/man3/pow.3.html

laanwj commented at 1:09 pm on June 14, 2021: member

The issue seems to be that Pixie is incorrectly returning None, rather than a symbol version quite often. i.e with:

Interesting. Will take a look at the pixie issue. See #22244.

laanwj referenced this in commit 19e598bab0 on Jun 14, 2021

hebasto commented at 0:08 am on June 20, 2021: member

bitcoind compatibility fixed in #22281 and #22287. @willyko Do you mind testing suggested fixes?

hebasto referenced this in commit bdaff07b32 on Jun 20, 2021

laanwj referenced this in commit a305a687e7 on Jun 21, 2021

rebroad referenced this in commit 537ea316f9 on Jun 23, 2021

sidhujag referenced this in commit 5c4ff037a8 on Jun 24, 2021

rebroad referenced this in commit ddcc39d4c5 on Jun 28, 2021

hebasto commented at 6:41 pm on July 5, 2021: member

Since #22365 has been merged, this could be closed, no?

laanwj closed this on Jul 5, 2021

josibake referenced this in commit 5ab3b2c499 on Jul 21, 2021

luke-jr referenced this in commit 8991a13c02 on Jul 26, 2021

luke-jr referenced this in commit 78df148816 on Jul 28, 2021

DrahtBot locked this on Aug 18, 2022

gitian: GLIBC_2.29 not found error at bitcoind startup on bionic #21454