build: Link time garbage collection

fanquake commented at 12:28 pm on April 12, 2020: member

Adds the linker flags required to turn on link time garbage collection to the Linux gitian descriptors and Guix build. After some experimentation I’m not convinced this is something that we should do for Windows builds at this stage.

Size comparison for a gitian build of master (df303ceb650521dc7b1ba91e0eea383c387a5860) vs this PR (8c31ef9ec68549c4105e8115f74f435c71b3901c):

binary	master (df303ceb650521dc7b1ba91e0eea383c387a5860)	PR (8c31ef9ec68549c4105e8115f74f435c71b3901c)	saving
bitcoind	10525336	9775672	7%
bitcoin-cli	2055800	1113624	45%
bitcoin-qt	31896456	31114024	2.5%
bitcoin-tx	2376152	1372536	42%
bitcoin-wallet	5899168	3199808	45%
test_bitcoin	17063328	15354784	10%

fanquake added the label Build system on Apr 12, 2020

fanquake added the label Needs gitian build on Apr 12, 2020

DrahtBot commented at 1:07 pm on April 12, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

laanwj commented at 11:14 am on April 13, 2020: member

Concept ACK.

DrahtBot removed the label Needs gitian build on Apr 13, 2020

fanquake force-pushed on Apr 15, 2020

fanquake commented at 0:27 am on April 16, 2020: member

Rebased for #17929 and added a commit for Guix.

jonasschnelli commented at 6:42 am on April 16, 2020: contributor

Nice. Concept ACK. Is there a clang equivalent for --gc-sections? fdata-sections AFAIK is also supported in clang. Could the clang linker flag be -dead_strip?

in contrib/gitian-descriptors/gitian-linux.yml:45 in 9f2e435761 outdated

43@@ -44,9 +44,9 @@ script: |
44   CONFIGFLAGS="--enable-glibc-back-compat --enable-reduce-exports --disable-bench --disable-gui-tests"
45   FAKETIME_HOST_PROGS="gcc g++"
46   FAKETIME_PROGS="date ar ranlib nm"
47-  HOST_CFLAGS="-O2 -g"
48-  HOST_CXXFLAGS="-O2 -g"
49-  HOST_LDFLAGS_BASE="-static-libstdc++ -Wl,-O2"
50+  HOST_CFLAGS="-O2 -g -ffunction-sections -fdata-sections"

laanwj commented at 4:24 pm on April 16, 2020:

These flags are effective for the depends build too, I hope?

fanquake commented at 9:36 am on April 22, 2020: member

I’ve been doing some experiments with Windows builds, and passing --gc-sections there currently results in an increase in binary size. I still need to test using a more recent version of binutils.

Is there a clang equivalent for –gc-sections?

gc-sections is a linker flag, so as long as Clang forwards it through, and whichever linker you are using understands it, it should work. Note that ld64 on macOS does not.

fdata-sections AFAIK is also supported in clang.

Clang supports -ffunction-sections and -fdata-sections. However only for ELF targets.

0-fdata-sections         Place each data in its own section (ELF Only)
1-ffunction-sections     Place each function in its own section (ELF Only)

Could the clang linker flag be -dead_strip?

We do currently pass dead_strip to ld64 for our macOS builds; and see size reductions. For example, compiling master with and without:

Binary	dead_strip	no dead_strip
bitcoind	10900836	11741428
bitcoin-qt	28593032	32178676
bitcoin-cli	466896	1065220
bitcoin-tx	1196536	1671900
bitcoin-wallet	3054504	5581956

fanquake referenced this in commit ac21090f20 on Apr 28, 2020

sidhujag referenced this in commit d861869e55 on Apr 28, 2020

fanquake renamed this:
~~[WIP] build: Link time garbage collection~~
build: Link time garbage collection
on May 22, 2020

fanquake force-pushed on May 22, 2020

fanquake added the label Needs gitian build on May 22, 2020

fanquake added the label Needs Guix build on May 22, 2020

fanquake marked this as ready for review on May 22, 2020

fanquake force-pushed on May 22, 2020

fanquake commented at 6:44 am on May 22, 2020: member

I’ve updated this to drop the changes for Windows for now. After some more experimenting it wasn’t clear that it was working as we’d like and I was still seeing increased binary sizes with newer binutils.

I have also updated the PR description with a comparison of the size differences for a gitian build of master vs this branch.

fanquake force-pushed on May 22, 2020

DrahtBot removed the label Needs gitian build on May 25, 2020

DrahtBot removed the label Needs Guix build on Jun 18, 2020

MarcoFalke deleted a comment on Jun 18, 2020

practicalswift commented at 5:25 am on September 18, 2020: contributor

Concept ACK

MarcoFalke deleted a comment on Sep 24, 2020

build: perform link time garbage collection in gitian

Adds the linker flag required to turn on link time garbage collection
to the linux gitian descriptors. Combined with `-ffunction-sections` and
`-fdata-sections` this results in a between 5% and 45% reduction in file
size for the binaries we are shipping.

19b20ee9d7

build: perform link time garbage collection in Guix

Adds the linker flags required to turn on link time garbage collection
to the Linux Guix build. Combined with `-ffunction-sections` and
`-fdata-sections` this results in a between ~5% and 45% reduction in file
size for the binaries we are shipping.

d05d56047a

fanquake force-pushed on Nov 22, 2020

fanquake commented at 4:21 am on November 22, 2020: member

I rebased this and ran some gitian builds to compare master (816132e6eb23c06287c1e820cc0990d2317c2f10) and this PR (d05d56047a38686c70dc066d010ed8f06990508b) (also included 0.20.1):

 0# ls -la bitcoin-0.20.1/bin/
 12055800 Jul 31 12:01 bitcoin-cli
 231843176 Jul 31 12:01 bitcoin-qt
 32376152 Jul 31 12:01 bitcoin-tx
 45952832 Jul 31 12:01 bitcoin-wallet
 510312344 Jul 31 12:01 bitcoind
 616702880 Jul 31 12:01 test_bitcoin
 7
 8# ls -la bitcoin-816132e6eb23/bin/
 94689528 Nov 21 06:47 bitcoin-cli
1036844872 Nov 21 06:47 bitcoin-qt
114792088 Nov 21 06:47 bitcoin-tx
1210000192 Nov 21 06:47 bitcoin-wallet
1315277080 Nov 21 06:47 bitcoind
1421103040 Nov 21 06:47 test_bitcoin
15
16# ls -la bitcoin-d05d56047a38/bin
174439576 Apr 15  2020 bitcoin-cli
1834407752 Apr 15  2020 bitcoin-qt
194505304 Apr 15  2020 bitcoin-tx
207214880 Apr 15  2020 bitcoin-wallet
2114953496 Apr 15  2020 bitcoind
2220516928 Apr 15  2020 test_bitcoin

All of our shipped binaries have certainly grown a bit since 0.20.1. There’s still a few mb to be saved in some cases.

dergoegge commented at 1:51 pm on December 9, 2020: member

I confirmed the space savings by running the gitian builds on master and this PR. I also read up on how the used options (functions-sections, data-sections, gc-sections) work and if there any unwanted side effects.

Two questions:

AFAICT the options are not active in the depends build. Why not?

https://github.com/bitcoin/bitcoin/blob/d05d56047a38686c70dc066d010ed8f06990508b/contrib/gitian-descriptors/gitian-linux.yml#L96-L99

The gcc docs warn to only use these options if there is a significant benefit from doing so because certain compiler optimizations wont work while they are enabled. Is it ok to ignore that warning? If so, why?

From https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html:

Only use these options when there are significant benefits from doing so. […] They prevent optimizations by the compiler and assembler using relative locations inside a translation unit since the locations are unknown until link time. An example of such an optimization is relaxing calls to short call instructions.

fanquake closed this on Feb 17, 2021

fanquake referenced this in commit 681b25e3cd on Nov 25, 2021

sidhujag referenced this in commit 67658eec7d on Nov 25, 2021

PastaPastaPasta referenced this in commit 0ec2a3b640 on Jun 19, 2022

PastaPastaPasta referenced this in commit f7a94b58d4 on Jun 19, 2022

DrahtBot locked this on Aug 16, 2022

fanquake deleted the branch on Nov 9, 2022

build: Link time garbage collection #18605

Conflicts