depends: should linux host should be using -sysroot?

ryanofsky commented at 2:12 pm on May 8, 2020: member

It seems strange that linux host definition in depends just calls the native system compiler with -m32/-m64 flags, instead of supporting cross compilers, and that it doesn’t use a -sysroot value to prevent picking up system libraries:

https://github.com/bitcoin/bitcoin/blob/master/depends/hosts/linux.mk

This is pretty different than the other windows/darwin/android hosts defined in the same directory, and seems like it would prevent make HOST=x86_64-pc-linux-gnu from working on anything other than a matching native platform.

Another side-effect of using the native compiler without a -sysroot value is that the build currently pick ups system libraries like BDB. From #bitcoin-builds:

[19:57:12] <achow101> when I build with the depends system, is it supposed to fallback to my system libs if the depends build misses a dependendency? [22:40:02] <dongcarl> achow101: What’s the full context? [22:45:13] <achow101> dongcarl: I modified depends to build a version of bdb that we didn’t support. configure then ignored that version and used my system lib instead of erroring as i was expecting it to

This is also causing headaches in #18677. I first added new packages there not even using host_CC etc variables, so native builds on linux and mac worked fine, but the cross compiled linux->mac build didn’t work, which fanquake reported. To fix that, I started using the host_CC etc variables and passing along -sysroot values for host packages to fix the reported linux->mac cross-compile problem. But this turned out to break the i686-linux-gnu build on travis, because specifying -sysroot prevents gcc from using include files like stdio.h from /usr/include/ and instead only use /usr/x86_64-linux-gnu/include/stdio.h. This worked fine on my system, but didn’t work on travis because the libc6-dev-amd64-cross package wasn’t installed. So I added that package and travis turned green.

But now hebasto is reporting the same error I was seeing on travis in an local linux mint build #18677 (comment) and it seems like linux mint might not have a libc6-dev-amd64-cross package.

I fixed this by not setting -sysroot for the new packages for the native linux build, and I probably should have done this before to respect the host_CC etc variables more. But it would also be good to understand why the linux host variables are set the way they are to begin with. Are they actually intended to pick up outside system libraries and not support cross compiling?

It’s unclear if they’re set the current way for expediency, or if there’s a different design goal for the linux host compared to the other hosts.

ryanofsky added the label Bug on May 8, 2020

fanquake added the label Build system on May 8, 2020

ryanofsky commented at 3:27 pm on May 8, 2020: member

@theuni @dongcarl @hebasto @achow101 @fanquake (not that I have to tag fanquake) may be interested or know more about this

dongcarl commented at 9:54 pm on August 17, 2020: member

@ryanofsky I’m a little unclear on the specific question you have here, but would very much like to get to the bottom of it :relaxed:

It seems strange that linux host definition in depends just calls the native system compiler with -m32/-m64 flags, instead of supporting cross compilers and that it doesn’t use a -sysroot value to prevent picking up system libraries:

https://github.com/bitcoin/bitcoin/blob/master/depends/hosts/linux.mk

This is pretty different than the other windows/darwin/android hosts defined in the same directory, and seems like it would prevent make HOST=x86_64-pc-linux-gnu from working on anything other than a matching native platform.

I’m not 100% sure what you mean here.

The only host we use a -sysroot value for is darwin

I believe the Makefile logic in default.mk and linux.mk works (but can most likely be improved) for cross compiling. In the specific case you mentioned, this is the current behaviour:

0$ make BUILD=riscv64-linux-gnu HOST=x86_64-pc-linux-gnu print-{build,host}{,_CC,_CXX}
1build = riscv64-unknown-linux-gnu
2host = x86_64-pc-linux-gnu
3build_CC = gcc
4host_CC = x86_64-pc-linux-gnu-gcc -m64
5build_CXX = g++
6host_CXX = x86_64-pc-linux-gnu-g++ -m64

I can see what you’re talking about w/re the sysroot situation. I think the situation currently is:

For HOST=BUILD native builds, we rely on unprefixed-gcc’s search/library path detection routine, which will definitely pick up system libraries: https://github.com/bitcoin/bitcoin/blob/1bc8e8eae2dc4b47ef67afc6880ea48b8e84a086/depends/hosts/default.mk#L1-L13
For cross builds, we either invoke $(HOST)-gcc, or in the case of i686/x86_64 crosses, specify the -m32/64 multilib. This should indicate to gcc’s detection routine to only look for library/headers matching the desired architecture, but it is possible that the user has installed “cross-compiling development” system headers/libraries (other than libc and gang) that get picked up.

I think it is a worthwhile goal to make sure that we pick up the system headers/libraries that we do want (libc, libgcc, libstdc++, kernel headers etc.) and not pick up the system headers/libraries that we don’t want (the system bdb, for example). The macOS SDK allows us to do that very nicely, however, it is difficult on Linux distros to separate the two.

I’ll do some more investigation around this, and feel free to clarify your question if I understood it wrong. Will follow up tomorrow…

ryanofsky commented at 6:19 pm on October 27, 2020: member

Thanks! I think that answers my questions.

IIUC, you’re saying it might make sense in theory to -sysroot when host_CC is the local system compiler (gcc/clang) instead of a cross-compiler $(default_host_CC). Using -sysroot in this case would avoid unintentionally picking up libraries and headers that happen to be installed locally. But things are messy so this doesn’t work well in practice, and we aren’t doing it right now on linux. We are for macos.

We could probably also use -sysroot when host_CC is set to a cross-compiler $(default_host_CC) rather than the local system compiler. But we don’t because cross-compilers already know to look in their own toolchains and ignore local libraries and headers, so it would be mostly redundant and only serve to avoid picking up “cross-compiling development” libraries.

The other thing I was confused by is hosts/linux.mk choosing the native compiler instead of the cross compiler in cases where using the native compiler would be wrong. You’re right to point out that it actually does choose the cross compiler in the linux->linux case:

0$ make BUILD=riscv64-linux-gnu HOST=x86_64-pc-linux-gnu print-host_CC
1host_CC = x86_64-pc-linux-gnu-gcc -m64

But I was thinking more about nonlinux->linux cases:

0$ make BUILD=x86_64-apple-darwin16 HOST=x86_64-pc-linux-gnu print-host_CC
1host_CC = gcc -m64
2
3$ make BUILD=x86_64-w64-mingw32 HOST=x86_64-pc-linux-gnu print-host_CC
4make: *** No rule to make target 'builders/mingw32.mk'.  Stop.

But I guess these cases are so broken there’d be no point trying to set the variables more correctly

ryanofsky closed this on Oct 27, 2020

ryanofsky commented at 7:38 pm on June 2, 2021: member

re: #18915#issue-614768292

[19:57:12] when I build with the depends system, is it supposed to fallback to my system libs if the depends build misses a dependendency?

Just to link a related issue, #21629 seems to add a way of reliably not falling back to system BDB

DrahtBot locked this on Aug 18, 2022

depends: should linux host should be using -sysroot? #18915