depends: Purge libtool archives #15844

We use pkg-config where we can, which generally replaces libtool at a
higher level and does not have the same downsides as libtool. These
archives sit in our depends tree with no purpose and pollute the final
bitcoin build with massive overlinking.

See here for an explanation of the various problems libtool archives can cause.

Unrelated in every way except in spirit: -D__LIBTOOL_IS_A_FOOL__!!

This PR is based on #16041, and therefore should be merged after #16041.

I had to do this once for an embedded project, because the la files were confusing things (as they were generated in a different environment). But I'm not sure here, do you have evidence of "massive overlinking", what is the result of that? Does the build become slower, does the executable become larger? If so, can you show results please?

@laanwj

The result is usually -lstdc++ and various dynamic linking flags we don't want ending up in the .la files, and so when we do the final bitcoin build, we're getting extra symbols that aren't necessary in our ELF table. This makes make -C src check-symbols fail.

You can expand "Old content no. 1" here for an example.

This comes at my request, so concept ACK :)

I suspect it may flesh out a few actual bugs, where we had assumed pkg-config was doing its job.

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #16041 (depends: Build secondary deps statically. by dongcarl)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@dongcarl Could you try this https://github.com/theuni/bitcoin/commit/f643224b4ca147f8c127804d1c03237eee345d03 as a test fix for the new failure?

@theuni Looking at the travis failures it's similar to something I've encountered before related to _XEatDataWords which is fixed by https://github.com/freedesktop/libXext/commit/bb24f2970f2e425f4df90c9b73d078ad15a73fbb. On my local builds I've just upgraded the package to the earliest version that contained that fix: https://github.com/freedesktop/libXext/releases/tag/libXext-1.3.3

I can just push another commit that bumps libXext up one minor version if that's okay with people?

@dongcarl Hah, yes, thanks for noticing. I just assumed ours wasn't ancient. Please include the bump with this PR, since it breaks the build without it.

Looking at the Travis build logs for https://github.com/bitcoin/bitcoin/pull/15844/commits/27ce43dcd512d1158c7400ac03afbdd1af2d7031, it would seem that cross builds are running into another problem that I assumed to be specific to my particular configuration. Namely, that we need to give the config flag --enable-malloc0returnsnull.

I've included that config flag in https://github.com/bitcoin/bitcoin/pull/15844/commits/e5444bbd13a30448f3d811d4f1faef058f18e65a, this seems to be a common problem for cross builds of X things: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861073

Concept ACK

Looks like there is still at least one issue with the 32-bit + dash [GOAL: install] [GUI: no BIP70] build here:

checking for QTACCESSIBILITY... yes
checking for QTFB... yes
checking for X11XCB... yes
checking for QTXCBQPA... yes
checking for static Qt plugins: -lqminimal... no
configure: error: Could not resolve: -lqminimal

This is the diff of /lib for a master depends build, vs this PR on macOS, will test on others:

<details><summary><code>diffoscope x86_64-apple-darwin18.5.0-master/lib x86_64-apple-darwin18.5.0/lib</code>:</summary>

@@ -28,37 +28,28 @@
libboost_unit_test_framework-mt.a
libcrypto.a
libdb-4.8.a
libdb.a
libdb_cxx-4.8.a
libdb_cxx.a
libevent.a
-libevent.la
libevent_core.a
-libevent_core.la
libevent_extra.a
-libevent_extra.la
libevent_pthreads.a
-libevent_pthreads.la
libminiupnpc.a
libprotobuf-lite.a
-libprotobuf-lite.la
libprotobuf.a
-libprotobuf.la
-libprotoc.la
libqrencode.a
-libqrencode.la
libqtharfbuzz.a
libqtlibpng.a
libqtpcre2.a
librapidcheck.a
libssl.a
libz.a
libzmq.a
-libzmq.la
pkgconfig
pkgconfig/Qt5AccessibilitySupport.pc
pkgconfig/Qt5CglSupport.pc
pkgconfig/Qt5ClipboardSupport.pc
pkgconfig/Qt5Core.pc
pkgconfig/Qt5DBus.pc
pkgconfig/Qt5DeviceDiscoverySupport.pc

</details>

After depends building inside a ubuntu:1804 Docker container.

<details><summary><code>diffoscope x86_64-pc-linux-gnu-master/lib x86_64-pc-linux-gnu/lib</code></summary>

@@ -15,25 +15,21 @@
libQt5Network.a
libQt5PrintSupport.a
libQt5ServiceSupport.a
libQt5Test.a
libQt5ThemeSupport.a
libQt5Widgets.a
libQt5XcbQpa.a
-libX11-xcb.la
libX11-xcb.so
libX11-xcb.so.1
libX11-xcb.so.1.0.0
-libX11.la
libX11.so
libX11.so.6
libX11.so.6.3.0
libXau.a
-libXau.la
-libXext.la
libXext.so
libXext.so.6
libXext.so.6.4.0
libboost_chrono-mt.a
libboost_filesystem-mt.a
libboost_prg_exec_monitor-mt.a
libboost_system-mt.a
@@ -42,154 +38,117 @@
libboost_timer-mt.a
libboost_unit_test_framework-mt.a
libcrypto.a
libdb-4.8.a
libdb.a
libdb_cxx-4.8.a
libdb_cxx.a
-libdbus-1.la
libdbus-1.so
libdbus-1.so.3
libdbus-1.so.3.14.11
libevent.a
-libevent.la
libevent_core.a
-libevent_core.la
libevent_extra.a
-libevent_extra.la
libevent_pthreads.a
-libevent_pthreads.la
-libexpat.la
libexpat.so
libexpat.so.1
libexpat.so.1.6.8
-libfontconfig.la
libfontconfig.so
libfontconfig.so.1
libfontconfig.so.1.9.2
-libfreetype.la
libfreetype.so
libfreetype.so.6
libfreetype.so.6.13.0
libminiupnpc.a
libprotobuf-lite.a
-libprotobuf-lite.la
libprotobuf.a
-libprotobuf.la
-libprotoc.la
libqrencode.a
-libqrencode.la
libqtharfbuzz.a
libqtlibpng.a
libqtpcre2.a
librapidcheck.a
libssl.a
-libxcb-composite.la
libxcb-composite.so
libxcb-composite.so.0
libxcb-composite.so.0.0.0
-libxcb-damage.la
libxcb-damage.so
libxcb-damage.so.0
libxcb-damage.so.0.0.0
-libxcb-dpms.la
libxcb-dpms.so
libxcb-dpms.so.0
libxcb-dpms.so.0.0.0
-libxcb-dri2.la
libxcb-dri2.so
libxcb-dri2.so.0
libxcb-dri2.so.0.0.0
-libxcb-dri3.la
libxcb-dri3.so
libxcb-dri3.so.0
libxcb-dri3.so.0.0.0
-libxcb-glx.la
libxcb-glx.so
libxcb-glx.so.0
libxcb-glx.so.0.0.0
-libxcb-present.la
libxcb-present.so
libxcb-present.so.0
libxcb-present.so.0.0.0
-libxcb-randr.la
libxcb-randr.so
libxcb-randr.so.0
libxcb-randr.so.0.1.0
-libxcb-record.la
libxcb-record.so
libxcb-record.so.0
libxcb-record.so.0.0.0
-libxcb-render.la
libxcb-render.so
libxcb-render.so.0
libxcb-render.so.0.0.0
-libxcb-res.la
libxcb-res.so
libxcb-res.so.0
libxcb-res.so.0.0.0
-libxcb-screensaver.la
libxcb-screensaver.so
libxcb-screensaver.so.0
libxcb-screensaver.so.0.0.0
-libxcb-shape.la
libxcb-shape.so
libxcb-shape.so.0
libxcb-shape.so.0.0.0
-libxcb-shm.la
libxcb-shm.so
libxcb-shm.so.0
libxcb-shm.so.0.0.0
libxcb-static.a
-libxcb-sync.la
libxcb-sync.so
libxcb-sync.so.1
libxcb-sync.so.1.0.0
-libxcb-xevie.la
libxcb-xevie.so
libxcb-xevie.so.0
libxcb-xevie.so.0.0.0
-libxcb-xf86dri.la
libxcb-xf86dri.so
libxcb-xf86dri.so.0
libxcb-xf86dri.so.0.0.0
-libxcb-xfixes.la
libxcb-xfixes.so
libxcb-xfixes.so.0
libxcb-xfixes.so.0.0.0
-libxcb-xinerama.la
libxcb-xinerama.so
libxcb-xinerama.so.0
libxcb-xinerama.so.0.0.0
-libxcb-xkb.la
libxcb-xkb.so
libxcb-xkb.so.1
libxcb-xkb.so.1.0.0
-libxcb-xprint.la
libxcb-xprint.so
libxcb-xprint.so.0
libxcb-xprint.so.0.0.0
-libxcb-xtest.la
libxcb-xtest.so
libxcb-xtest.so.0
libxcb-xtest.so.0.0.0
-libxcb-xv.la
libxcb-xv.so
libxcb-xv.so.0
libxcb-xv.so.0.0.0
-libxcb-xvmc.la
libxcb-xvmc.so
libxcb-xvmc.so.0
libxcb-xvmc.so.0.0.0
-libxcb.la
libxcb.so
libxcb.so.1
libxcb.so.1.1.0
libz.a
libzmq.a
-libzmq.la
pkgconfig
pkgconfig/Qt5AccessibilitySupport.pc
pkgconfig/Qt5Core.pc
pkgconfig/Qt5DBus.pc
pkgconfig/Qt5DeviceDiscoverySupport.pc
pkgconfig/Qt5EventDispatcherSupport.pc
pkgconfig/Qt5FbSupport.pc

</details>

Ready for re-review. Checks are passing.

Thanks, having a look at this.

Updated with some commentary to help future cross-compilation target architectures/libcs.

Rebased.

<!--a722867cd34abeea1fadc8d60700f111-->

Gitian builds for commit 3001cc61cf11e016c403ce83c9cbcfd3efcbcfd9 (master):

• 15b471a7b08d6bd0c1cae96f073aef4b... bitcoin-0.18.99-osx-unsigned.dmg
• 45262dff43ecd3eab71c5968210bbd31... bitcoin-0.18.99-osx64.tar.gz
• c7e1076ae7761619b238069eb66f314f... bitcoin-0.18.99-win64-debug.zip
• bb0445848e6e33be45a6dd52cf66780c... bitcoin-0.18.99-win64-setup-unsigned.exe
• 8278d9e77eea04629372152a43a3c356... bitcoin-0.18.99-win64.zip
• f3a20c18807b1426639b7d1fc49c3f98... bitcoin-core-osx-0.19-res.yml
• 519a40aea2793ff44530f1dca4dea241... bitcoin-core-win-0.19-res.yml
• 033a78687856f5dbee10443e0116628c... bitcoin-linux-build.log
• 96837efa33aeebc77116f1269f94bc3d... bitcoin-osx-build.log
• 44d74a0e5956a7fd5823766dcae684b5... bitcoin-win-build.log

Gitian builds for commit fbf6a3e8ceaa2593f6fb169e8c2f520f25588fe4 (master and this pull):

• 3d7ba2d62cc0c732b54224c78210ecaa... bitcoin-0.18.99-osx-unsigned.dmg
• 98aa3c9ee559aa9680eace2916be9b37... bitcoin-0.18.99-osx64.tar.gz
• 200623f365febf180f6256cdc89f5306... bitcoin-0.18.99-win64-debug.zip
• e5f3d0815c9f8e9a3a354b25ab8fe9fd... bitcoin-0.18.99-win64-setup-unsigned.exe
• bc3baf45397d4d06f9ab977883fb67a7... bitcoin-0.18.99-win64.zip
• 8cb2392ad5cb708a1258b8de1473cc99... bitcoin-core-osx-0.19-res.yml
• 719edb1716db9108806a93db83d8e5a8... bitcoin-core-win-0.19-res.yml
• d745af27c134e1ce5e420b19f85a3433... bitcoin-linux-build.log
• ad3b84cb9eb72ea248c87257fa18f7f5... bitcoin-osx-build.log
• 9a38c4a652c05c8ee79198b0a30e9ac3... bitcoin-win-build.log

tACK 716afd6

.las all seem to be removed, and we pick up a libexpat.a because of 716afd6.

Building on macOS 10.14.x:

<details><summary><code>diffoscope x86_64-apple-darwin18.6.0/lib x86_64-apple-darwin18.6.0-716afd6a3/lib</code></summary>

libboost_unit_test_framework-mt.a
libcrypto.a
libdb-4.8.a
libdb.a
libdb_cxx-4.8.a
libdb_cxx.a
libevent.a
-libevent.la
libevent_core.a
-libevent_core.la
libevent_extra.a
-libevent_extra.la
libevent_pthreads.a
-libevent_pthreads.la
libminiupnpc.a
libprotobuf-lite.a
-libprotobuf-lite.la
libprotobuf.a
-libprotobuf.la
-libprotoc.la
libqrencode.a
-libqrencode.la
libqtharfbuzz.a
libqtlibpng.a
libqtpcre2.a
librapidcheck.a
libssl.a
libz.a
libzmq.a
-libzmq.la
pkgconfig
pkgconfig/Qt5AccessibilitySupport.pc
pkgconfig/Qt5CglSupport.pc
pkgconfig/Qt5ClipboardSupport.pc
pkgconfig/Qt5Core.pc
pkgconfig/Qt5DBus.pc
pkgconfig/Qt5DeviceDiscoverySupport.pc

</details>

Building on Debian 9.9:

<details><summary><code>diffscope x86_64-pc-linux-gnu/lib x86_64-pc-linux-gnu-716afd6a3/lib</code></summary>

libQt5Network.a
libQt5PrintSupport.a
libQt5ServiceSupport.a
libQt5Test.a
libQt5ThemeSupport.a
libQt5Widgets.a
libQt5XcbQpa.a
-libX11-xcb.la
libX11-xcb.so
libX11-xcb.so.1
libX11-xcb.so.1.0.0
-libX11.la
libX11.so
libX11.so.6
libX11.so.6.3.0
libXau.a
-libXau.la
-libXext.la
libXext.so
libXext.so.6
libXext.so.6.4.0
libboost_chrono-mt.a
libboost_filesystem-mt.a
libboost_prg_exec_monitor-mt.a
libboost_system-mt.a
@@ -42,155 +38,115 @@
libboost_timer-mt.a
libboost_unit_test_framework-mt.a
libcrypto.a
libdb-4.8.a
libdb.a
libdb_cxx-4.8.a
libdb_cxx.a
-libdbus-1.la
libdbus-1.so
libdbus-1.so.3
libdbus-1.so.3.14.11
libevent.a
-libevent.la
libevent_core.a
-libevent_core.la
libevent_extra.a
-libevent_extra.la
libevent_pthreads.a
-libevent_pthreads.la
-libexpat.la
-libexpat.so
-libexpat.so.1
-libexpat.so.1.6.8
-libfontconfig.la
+libexpat.a
libfontconfig.so
libfontconfig.so.1
libfontconfig.so.1.9.2
-libfreetype.la
libfreetype.so
libfreetype.so.6
libfreetype.so.6.13.0
libminiupnpc.a
libprotobuf-lite.a
-libprotobuf-lite.la
libprotobuf.a
-libprotobuf.la
-libprotoc.la
libqrencode.a
-libqrencode.la
libqtharfbuzz.a
libqtlibpng.a
libqtpcre2.a
librapidcheck.a
libssl.a
-libxcb-composite.la
libxcb-composite.so
libxcb-composite.so.0
libxcb-composite.so.0.0.0
-libxcb-damage.la
libxcb-damage.so
libxcb-damage.so.0
libxcb-damage.so.0.0.0
-libxcb-dpms.la
libxcb-dpms.so
libxcb-dpms.so.0
libxcb-dpms.so.0.0.0
-libxcb-dri2.la
libxcb-dri2.so
libxcb-dri2.so.0
libxcb-dri2.so.0.0.0
-libxcb-dri3.la
libxcb-dri3.so
libxcb-dri3.so.0
libxcb-dri3.so.0.0.0
-libxcb-glx.la
libxcb-glx.so
libxcb-glx.so.0
libxcb-glx.so.0.0.0
-libxcb-present.la
libxcb-present.so
libxcb-present.so.0
libxcb-present.so.0.0.0
-libxcb-randr.la
libxcb-randr.so
libxcb-randr.so.0
libxcb-randr.so.0.1.0
-libxcb-record.la
libxcb-record.so
libxcb-record.so.0
libxcb-record.so.0.0.0
-libxcb-render.la
libxcb-render.so
libxcb-render.so.0
libxcb-render.so.0.0.0
-libxcb-res.la
libxcb-res.so
libxcb-res.so.0
libxcb-res.so.0.0.0
-libxcb-screensaver.la
libxcb-screensaver.so
libxcb-screensaver.so.0
libxcb-screensaver.so.0.0.0
-libxcb-shape.la
libxcb-shape.so
libxcb-shape.so.0
libxcb-shape.so.0.0.0
-libxcb-shm.la
libxcb-shm.so
libxcb-shm.so.0
libxcb-shm.so.0.0.0
libxcb-static.a
-libxcb-sync.la
libxcb-sync.so
libxcb-sync.so.1
libxcb-sync.so.1.0.0
-libxcb-xevie.la
libxcb-xevie.so
libxcb-xevie.so.0
libxcb-xevie.so.0.0.0
-libxcb-xf86dri.la
libxcb-xf86dri.so
libxcb-xf86dri.so.0
libxcb-xf86dri.so.0.0.0
-libxcb-xfixes.la
libxcb-xfixes.so
libxcb-xfixes.so.0
libxcb-xfixes.so.0.0.0
-libxcb-xinerama.la
libxcb-xinerama.so
libxcb-xinerama.so.0
libxcb-xinerama.so.0.0.0
-libxcb-xkb.la
libxcb-xkb.so
libxcb-xkb.so.1
libxcb-xkb.so.1.0.0
-libxcb-xprint.la
libxcb-xprint.so
libxcb-xprint.so.0
libxcb-xprint.so.0.0.0
-libxcb-xtest.la
libxcb-xtest.so
libxcb-xtest.so.0
libxcb-xtest.so.0.0.0
-libxcb-xv.la
libxcb-xv.so
libxcb-xv.so.0
libxcb-xv.so.0.0.0
-libxcb-xvmc.la
libxcb-xvmc.so
libxcb-xvmc.so.0
libxcb-xvmc.so.0.0.0
-libxcb.la
libxcb.so
libxcb.so.1
libxcb.so.1.1.0
libz.a
libzmq.a
-libzmq.la
pkgconfig
pkgconfig/Qt5AccessibilitySupport.pc
pkgconfig/Qt5Core.pc
pkgconfig/Qt5DBus.pc
pkgconfig/Qt5DeviceDiscoverySupport.pc
pkgconfig/Qt5EventDispatcherSupport.pc
pkgconfig/Qt5FbSupport.pc

</details>

Updated to also enforce the stricter --disable-malloc0returnsnull on libX11. Many thanks to @fanquake for his diligent review!

re-tACK 6ef57ad

Update: Rebase on #16041.

Sorry for updating this so often, want to make sure that once people with merge access approve, the whole process is smooth.

ACK 8541cbea2, code changes look good to me