build: add linker optimisation flags to gitian & guix (Linux) #17929

pull fanquake wants to merge 2 commits into bitcoin:master from fanquake:pass_optimizations_to_linker changing 2 files +2 −2
  1. fanquake commented at 12:01 pm on January 15, 2020: member

    This PR adds -Wl,O2 to our gitian and guix LDFLAGS. This makes the linker perform certain optimisations (and is different from LTO).

    Any -O argument will enable optimizations in GNU ld. We can use -O2 here, as this matches our compile flags. Note that this would also enable additional optimizations if using the lld or gold linkers, when compared to -O0.

    A nice writeup + diagrams of some of these optimizations is available here: http://lwn.net/Articles/192624/.

    master

    0# bitcoind
    1Histogram for `.gnu.hash' bucket list length (total of 3 buckets)
    2 Length  Number     % of total  Coverage
    3      0  1          ( 33.3%)       0.0%
    4      1  0          (  0.0%)       0.0%
    5      2  1          ( 33.3%)      40.0%
    6      3  1          ( 33.3%)     100.0%
    
    0# bitcoin-qt 
    1Histogram for `.gnu.hash' bucket list length (total of 3 buckets)
    2 Length  Number     % of total  Coverage
    3      0  0          (  0.0%)       0.0%
    4      1  1          ( 33.3%)      10.0%
    5      2  0          (  0.0%)      10.0%
    6      3  0          (  0.0%)      10.0%
    7      4  1          ( 33.3%)      50.0%
    8      5  1          ( 33.3%)     100.0%
    

    this PR:

    0# bitcoind
    1Histogram for `.gnu.hash' bucket list length (total of 8 buckets)
    2 Length  Number     % of total  Coverage
    3      0  3          ( 37.5%)       0.0%
    4      1  5          ( 62.5%)     100.0%
    
    0# bitcoin-qt
    1Histogram for `.gnu.hash' bucket list length (total of 19 buckets)
    2 Length  Number     % of total  Coverage
    3      0  9          ( 47.4%)       0.0%
    4      1  10         ( 52.6%)     100.0%
    

    GNU ld -O

    If level is a numeric values greater than zero ld optimizes the output. This might take significantly longer and therefore probably should only be enabled for the final binary. At the moment this option only affects ELF shared library generation. Future releases of the linker may make more use of this option. Also currently there is no difference in the linker’s behaviour for different non-zero values of this option. Again this may change with future releases.

    lld -O

    Optimize output file size

  2. fanquake added the label Build system on Jan 15, 2020
  3. fanquake added the label Needs gitian build on Jan 15, 2020
  4. fanquake added the label Needs Conceptual Review on Jan 15, 2020
  5. fanquake requested review from dongcarl on Jan 15, 2020
  6. fanquake requested review from theuni on Jan 15, 2020
  7. MarcoFalke commented at 3:58 pm on January 15, 2020: member
    How is this different from #16791?
  8. fanquake commented at 9:53 pm on January 15, 2020: member

    How is this different from #16791? @MarcoFalke Despite similar naming, this is not the same thing as LTO. This will optimize the symbol hash tables in binaries, whereas LTO does intermodular optimizations. More info on either: Optimizing Linker Load Times and LTO.

  9. dongcarl commented at 0:37 am on January 16, 2020: member

    Concept ACK, good to know that it’s different from LTO.

    Need to look into it more carefully.

  10. DrahtBot removed the label Needs gitian build on Jan 16, 2020
  11. dongcarl commented at 9:14 pm on March 9, 2020: member

    Concept ACK, although would like to see this in Gitian/release builds instead of default configure. At some point we should probably add a doc to the repo of recommended configure flags and what they do.


    It seems to me that, when supplied with -O2:

    • ld optimizes the hash chain size for the symbol hash table, which may lead to larger outputs
    • ld.gold optimizes the output file size

    Perhaps that accounts for the performance differences you’re seeing?

  12. fanquake force-pushed on Mar 12, 2020
  13. fanquake force-pushed on Mar 12, 2020
  14. fanquake commented at 10:47 am on March 12, 2020: member

    Concept ACK, although would like to see this in Gitian/release builds instead of default configure.

    I agree, and I’ve change this to be a configure flag, --enable-linker-optimizations (off by default). Then we can pass it into gitian builds in future if desired. Going to follow up with some more info shortly.

  15. practicalswift commented at 12:11 pm on March 12, 2020: contributor
    Concept ACK
  16. jgarzik commented at 1:48 pm on March 12, 2020: contributor

    Passing optimization flags to the link stage has been the default for decades in autotools projects (autoconf/automake/libtool). Thus,

    1. concept ACK
    2. It should be enabled by default.
  17. fanquake renamed this:
    build: pass optimization flags to the linker
    build: add --enable-linker-optimizations configure flag
    on Mar 13, 2020
  18. fanquake removed the label Needs Conceptual Review on Mar 13, 2020
  19. fanquake marked this as ready for review on Mar 13, 2020
  20. dongcarl commented at 5:41 pm on March 18, 2020: member
    From #bitcoin-builds meeting notes today: We should switch to just adding the -Wl,-O2 to LDFLAGs in Gitian/Guix, and benchmark to see the differences.
  21. MarcoFalke deleted a comment on Apr 10, 2020
  22. MarcoFalke added the label Needs gitian build on Apr 10, 2020
  23. DrahtBot commented at 2:26 pm on April 11, 2020: member

    Gitian builds

    File commit 51e2ce45d6824b4d621f26b2048e6a43672e45dc(master) commit 639244fb1c937317e48cded3153be1f7fc520297(master and this pull)
    bitcoin-0.20.99-aarch64-linux-gnu-debug.tar.gz db1457360b890c88... 4d22d899d297d779...
    bitcoin-0.20.99-aarch64-linux-gnu.tar.gz 75edcb08d0984f24... d2bf92384624ffae...
    bitcoin-0.20.99-arm-linux-gnueabihf-debug.tar.gz 580ba23b791dcb7e... 471266bf15dcf6c8...
    bitcoin-0.20.99-arm-linux-gnueabihf.tar.gz 8e0a426284ac41a8... 5a33f2161668d2bc...
    bitcoin-0.20.99-osx-unsigned.dmg 86461d918affc786... bf07133a87b4c921...
    bitcoin-0.20.99-osx64.tar.gz d4df0c5871fdf98e... 4f73c0eb50e847c6...
    bitcoin-0.20.99-riscv64-linux-gnu-debug.tar.gz b011ecbef10f6ea7... 0b828989bc58b6b1...
    bitcoin-0.20.99-riscv64-linux-gnu.tar.gz 73ba7e9385eac2f3... bedfd6878146beb2...
    bitcoin-0.20.99-win64-debug.zip 2088c0ff7ec0d77a... ee2ba447d38a4ca4...
    bitcoin-0.20.99-win64-setup-unsigned.exe ebfc12dfefda99dd... e75fd279aebb231c...
    bitcoin-0.20.99-win64.zip cbe6ca804ab62842... 20442fad0638d982...
    bitcoin-0.20.99-x86_64-linux-gnu-debug.tar.gz a66564f1e5ef974b... 6225b71f144d521e...
    bitcoin-0.20.99-x86_64-linux-gnu.tar.gz 6322ecbb05f44ce6... 912d64a26bceec6f...
    bitcoin-0.20.99.tar.gz 3df427ca65c2fad5... b5141bd82e118cf5...
    bitcoin-core-linux-0.20-res.yml 4b7423af6bd2c252... 4c0f64b763878c0b...
    bitcoin-core-osx-0.20-res.yml 8cb882f174639da0... 731072bb8a039e60...
    bitcoin-core-win-0.20-res.yml dda02e3247c94f96... ff33eb10350adf4a...
    linux-build.log 9540bc006731d125... e0be785a59b38230...
    osx-build.log 94afb36855d58118... bc443ba6fc99f367...
    win-build.log e508a602a512e046... 5724bcce840d406f...
    bitcoin-core-linux-0.20-res.yml.diff f38b4fcfd32d0bb1...
    bitcoin-core-osx-0.20-res.yml.diff 382bf1cbca3ee4d1...
    bitcoin-core-win-0.20-res.yml.diff 100a1afd1ab10da6...
    linux-build.log.diff efcc41af1758ebe5...
    osx-build.log.diff cf7696c49e770162...
    win-build.log.diff 9496c5d91224b071...
  24. DrahtBot removed the label Needs gitian build on Apr 11, 2020
  25. build: add linker optimization flags to gitian descriptors
    Any -O argument will enable optimizations in GNU ld. We can use -O2
    here, as this matches our compile flags. Note that this would also
    enable additional optimizations if using the lld or gold linkers,
    when compared to -O0.
    b8b050a8d6
  26. build: add linker optimization flags to guix
    Any -O argument will enable optimizations in GNU ld. We can use -O2
    here, as this matches our compile flags. Note that this would also
    enable additional optimizations if using the lld or gold linkers,
    when compared to -O0.
    f2b5b0a3b4
  27. fanquake renamed this:
    build: add --enable-linker-optimizations configure flag
    build: add linker optimisation flags to gitian & guix (Linux)
    on Apr 12, 2020
  28. fanquake force-pushed on Apr 12, 2020
  29. fanquake added the label Needs gitian build on Apr 12, 2020
  30. fanquake commented at 10:48 am on April 12, 2020: member
    I’ve updated this PR as discussed. Add the flags to the guix and gitian descriptors. Also adding additional information to the PR description.
  31. MarcoFalke added the label Needs Guix build on Apr 12, 2020
  32. DrahtBot commented at 4:23 pm on April 12, 2020: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #18605 ([WIP] build: Link time garbage collection by fanquake)
    • #14066 (gitian-linux: Build binaries for 64-bit POWER by luke-jr)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  33. DrahtBot commented at 8:12 pm on April 12, 2020: member

    Gitian builds

    File commit a5623ba89f050182ce9b1f570f3736b272b544b2(master) commit fdf3779ceac34b573a9a062f8b980590598fb251(master and this pull)
    bitcoin-0.20.99-aarch64-linux-gnu-debug.tar.gz 9c82e51c7e47d727... 3be095836c88e741...
    bitcoin-0.20.99-aarch64-linux-gnu.tar.gz edcb63d1eb5495e9... 61fe0f8c404f578a...
    bitcoin-0.20.99-arm-linux-gnueabihf-debug.tar.gz d5d4a5345f86af70... 18ce3cacb989e601...
    bitcoin-0.20.99-arm-linux-gnueabihf.tar.gz bfdbf761a2b9c50e... 0c5da690e9412896...
    bitcoin-0.20.99-osx-unsigned.dmg b5a64eac14885b58... 4be8deaaca3e962e...
    bitcoin-0.20.99-osx64.tar.gz 7c3d65d5be56192d... 6ebfec7c0f379759...
    bitcoin-0.20.99-riscv64-linux-gnu-debug.tar.gz 12b39df1f36d643d... c1ee68e23173fc39...
    bitcoin-0.20.99-riscv64-linux-gnu.tar.gz ab41831e0fe28459... 092054b5d1f4ed80...
    bitcoin-0.20.99-win64-debug.zip 1728364737f21cda... 6d383c0879cc5d3e...
    bitcoin-0.20.99-win64-setup-unsigned.exe 2b0b438a64b4a82a... 141dc3f5fbaee97f...
    bitcoin-0.20.99-win64.zip bc40c8a3cde15085... ac9b912b8a56eb2c...
    bitcoin-0.20.99-x86_64-linux-gnu-debug.tar.gz 02ce0e29960a563b... 6a53e4b0bfe524df...
    bitcoin-0.20.99-x86_64-linux-gnu.tar.gz 6c74ee5691689103... 092f54a5bd16505d...
    bitcoin-0.20.99.tar.gz 45fea2ff2b455258... d07a3a6297e1ab69...
    bitcoin-core-linux-0.21-res.yml 2fde331d1835beaf... 86ca1f773636e702...
    bitcoin-core-osx-0.21-res.yml 3e6d3ca9c4fb21bd... b806b088475a9e10...
    bitcoin-core-win-0.21-res.yml 0ccbdabff23646b2... 89b9544bf03ff929...
    linux-build.log 8b06a9b50657a83b... 650b8277042c4058...
    osx-build.log a6b34b7c4e79ab71... bf17b106a0c97899...
    win-build.log a755498086d9f0c1... 119eb2032702d78e...
    bitcoin-core-linux-0.21-res.yml.diff 1ae6f4414ae60fdc...
    bitcoin-core-osx-0.21-res.yml.diff 7360dcf7511673a3...
    bitcoin-core-win-0.21-res.yml.diff aba92b9ecab612f2...
    linux-build.log.diff 18db88368da187df...
    osx-build.log.diff ca7511978c75c981...
    win-build.log.diff 3f50b969dfa2e357...
  34. DrahtBot removed the label Needs gitian build on Apr 12, 2020
  35. dongcarl commented at 5:46 pm on April 13, 2020: member
    ACK f2b5b0a3b48f9241971c14238040048f8b630811 Thanks for the histogram!
  36. laanwj commented at 6:35 am on April 14, 2020: member
    ACK f2b5b0a3b48f9241971c14238040048f8b630811
  37. fanquake merged this on Apr 15, 2020
  38. fanquake closed this on Apr 15, 2020

  39. fanquake deleted the branch on Apr 15, 2020
  40. sidhujag referenced this in commit 6f3b3dcfde on Apr 15, 2020
  41. MarcoFalke removed the label Needs Guix build on Apr 20, 2020
  42. luke-jr referenced this in commit 7cf7cd4212 on Jun 9, 2020
  43. DrahtBot locked this on Feb 15, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-18 21:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me