depends: expat 2.4.8 & fix building with -flto #25697

1. 
   fanquake commented at 2:34 pm on July 25, 2022: member

   Currently, when building the expat package in depends, using -flto (LTO=1), the configure check can fail, because it cannot determine the system endianess:

   0configure:18718: result: unknown
   1configure:18733: error: unknown endianness
   2 presetting ac_cv_c_bigendian=no (or yes) will help
   

   Fix that by defining _DEFAULT_SOURCE, which in turn defines __USE_MISC (features.h):

   0#if defined _DEFAULT_SOURCE
   1# define __USE_MISC	1
   2#endif
   

   which exposes additional definitions in endian.h:

    0#include <features.h>
    1
    2/* Get the definitions of __*_ENDIAN, __BYTE_ORDER, and __FLOAT_WORD_ORDER.  */
    3#include <bits/endian.h>
    4
    5#ifdef __USE_MISC
    6# define LITTLE_ENDIAN	__LITTLE_ENDIAN
    7# define BIG_ENDIAN	__BIG_ENDIAN
    8# define PDP_ENDIAN	__PDP_ENDIAN
    9# define BYTE_ORDER	__BYTE_ORDER
   10#endif
   

   and gives us a working configure.

   You could test building this change with Guix + LTO with this branch. Note that that build may fail for other reasons (on x86_64), unrelated to this change.

   Some related upstream discussion: https://bugs.gentoo.org/757681 https://forums.gentoo.org/viewtopic-t-1013786.html

2. fanquake added the label Build system on Jul 25, 2022
3. 
   DrahtBot commented at 6:24 pm on July 25, 2022: contributor

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Conflicts

   Reviewers, this pull request conflicts with the following ones:

   • #25391 (guix: Use LTO to build releases by fanquake)
   • #24283 (build: Add show-% target for multi-line variables and debug info by hebasto)
   • #22552 (build: Improve depends build system robustness by hebasto)

   If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

4. 
   jarolrod commented at 7:11 pm on July 25, 2022: member

   GUIX hashes for lto_in_guix

   x86:

    0$ env HOSTS='aarch64-linux-gnu arm-linux-gnueabihf powerpc64-linux-gnu powerpc64le-linux-gnu riscv64-linux-gnu x86_64-linux-gnu' ./contrib/guix/guix-build 
    1$ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
    2
    35004a59842f8f206f4847dde50598670d99874da9fd0d028e99a946f3b99f96d  guix-build-75abb99603cc/output/aarch64-linux-gnu/SHA256SUMS.part
    4ce0028199f70ac1bd9fcb6068e19f0a9c4c4bb29daddcd149513e3a7cc255eca  guix-build-75abb99603cc/output/aarch64-linux-gnu/bitcoin-75abb99603cc-aarch64-linux-gnu-debug.tar.gz
    568c3953c69b28ea6a07fce6fb948dd94d90b3838fc225732a60beca9364a0f0b  guix-build-75abb99603cc/output/aarch64-linux-gnu/bitcoin-75abb99603cc-aarch64-linux-gnu.tar.gz
    6dfef8e0a7ef0fdf0aa6ab45f04268eb7f5fce8013f6e93d3c41f6e79ef46ec34  guix-build-75abb99603cc/output/arm-linux-gnueabihf/SHA256SUMS.part
    71aba24c79463b7ca02b264070cd37e86fc371c7ba1d217f51c537aef887c88b5  guix-build-75abb99603cc/output/arm-linux-gnueabihf/bitcoin-75abb99603cc-arm-linux-gnueabihf-debug.tar.gz
    8efc6aed030f9261c0dddf09f6855236123ff26ec3a741dfea474b78f80831a9e  guix-build-75abb99603cc/output/arm-linux-gnueabihf/bitcoin-75abb99603cc-arm-linux-gnueabihf.tar.gz
    9ca9294f428c3e50fe9db9e3b25a056577956eafd84bbc148e422c1ce421a0540  guix-build-75abb99603cc/output/dist-archive/bitcoin-75abb99603cc.tar.gz
   10a7e52ac891968f0659e32e146b423a0b473a91fed6c103b121934bb014a90f32  guix-build-75abb99603cc/output/powerpc64-linux-gnu/SHA256SUMS.part
   1194b63ff49b908b6202ce78c94213581197df9655fd99631dcfac43d1ba6c7533  guix-build-75abb99603cc/output/powerpc64-linux-gnu/bitcoin-75abb99603cc-powerpc64-linux-gnu-debug.tar.gz
   12e6bb1de2fc4fc37e232b982b69841486355c7fbac603159995b52684a2198e79  guix-build-75abb99603cc/output/powerpc64-linux-gnu/bitcoin-75abb99603cc-powerpc64-linux-gnu.tar.gz
   13bb40740f3d3d7db728f6234e263fb22252176856ef5850b7b8c343664608b6c7  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/SHA256SUMS.part
   14078970d4dd9084f928065074feb52fefcc7795c332d066bdc90a2495a674255c  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/bitcoin-75abb99603cc-powerpc64le-linux-gnu-debug.tar.gz
   157260e272084140e2d1b2c78d3786f363aaaea16f967274bb804289849c8f0351  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/bitcoin-75abb99603cc-powerpc64le-linux-gnu.tar.gz
   167239351d62b1db37ef9d45ca562c165f458f8dd1420cb80473323e4a4d0d0476  guix-build-75abb99603cc/output/riscv64-linux-gnu/SHA256SUMS.part
   17d9db73da46e36dfde19b207f809189e657002b717d2da08a754e7ee2d364b161  guix-build-75abb99603cc/output/riscv64-linux-gnu/bitcoin-75abb99603cc-riscv64-linux-gnu-debug.tar.gz
   1830bd2e0878a602cdfeafe9f3770e8d327cd3357a9b6cd0943f3c03fd5b1ddc02  guix-build-75abb99603cc/output/riscv64-linux-gnu/bitcoin-75abb99603cc-riscv64-linux-gnu.tar.gz
   199eecd92f5964fdcac87f2337c7fa88faa58feedb095c4b07395bffcc1421a627  guix-build-75abb99603cc/output/x86_64-linux-gnu/SHA256SUMS.part
   20241934184478155d1919d80b67db01ce2659f3c9ca5960b1c8a0bee34281b24d  guix-build-75abb99603cc/output/x86_64-linux-gnu/bitcoin-75abb99603cc-x86_64-linux-gnu-debug.tar.gz
   21db57a69879d2b8efedaaf8b9254d61465cdd0606e06a490b599bc56d1d7e3931  guix-build-75abb99603cc/output/x86_64-linux-gnu/bitcoin-75abb99603cc-x86_64-linux-gnu.tar.gz
   

   arm64:

    0$ env HOSTS='arm-linux-gnueabihf powerpc64-linux-gnu powerpc64le-linux-gnu riscv64-linux-gnu x86_64-linux-gnu' ./contrib/guix/guix-build 
    1$ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
    2
    3080237f918c928a2e13c4644075ddcbd1011f3bf243b2096ec16e489e7ea9279  guix-build-75abb99603cc/output/arm-linux-gnueabihf/SHA256SUMS.part
    4106e71d5aaa6569896cf97cfbc98f1f3cdeb9392a2313f2414c3c51012e8d4c9  guix-build-75abb99603cc/output/arm-linux-gnueabihf/bitcoin-75abb99603cc-arm-linux-gnueabihf-debug.tar.gz
    51af28e6fda591c83ee9ab8a35be3ff4c82e46a96fb3ea4afc2ee947b329479c3  guix-build-75abb99603cc/output/arm-linux-gnueabihf/bitcoin-75abb99603cc-arm-linux-gnueabihf.tar.gz
    6ca9294f428c3e50fe9db9e3b25a056577956eafd84bbc148e422c1ce421a0540  guix-build-75abb99603cc/output/dist-archive/bitcoin-75abb99603cc.tar.gz
    7f47d609deda0768a1ae2e69e44b02545d0ae0baff4be7ccad07eeab3d637f298  guix-build-75abb99603cc/output/powerpc64-linux-gnu/SHA256SUMS.part
    872f5eedc3cbea81706a6f47f974760cc9db5be536514dea41d105a049a999959  guix-build-75abb99603cc/output/powerpc64-linux-gnu/bitcoin-75abb99603cc-powerpc64-linux-gnu-debug.tar.gz
    94bea76a954edfecef6b7db0fd27dba4cd272b57b388bb821687c369ff7d6f581  guix-build-75abb99603cc/output/powerpc64-linux-gnu/bitcoin-75abb99603cc-powerpc64-linux-gnu.tar.gz
   105d68c496dee71dfa32c53002dca93eadc33f93cc49ea97d67f65d59a0cb8a287  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/SHA256SUMS.part
   114532e72989706a46b1da733c1624842ad4cc58c73290445c25c02e4166f71ff1  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/bitcoin-75abb99603cc-powerpc64le-linux-gnu-debug.tar.gz
   12cdab8486ea2b294f4fba7a4c8bd89d6fb17acf03b9fa690d7ba3c9d414d7ac31  guix-build-75abb99603cc/output/powerpc64le-linux-gnu/bitcoin-75abb99603cc-powerpc64le-linux-gnu.tar.gz
   13c7b0872824132c0f1d77bf40d9e37ffa9c89cff05ebf4c6e374944f57266daa8  guix-build-75abb99603cc/output/riscv64-linux-gnu/SHA256SUMS.part
   1423a90bc3073366ac8755f92f47f5b7e30a91f3bdcc90b04ccb04dba11a76406c  guix-build-75abb99603cc/output/riscv64-linux-gnu/bitcoin-75abb99603cc-riscv64-linux-gnu-debug.tar.gz
   159a55baaaab65088447654ef7a93b5264d8840cee427944286d7f02eee6e15358  guix-build-75abb99603cc/output/riscv64-linux-gnu/bitcoin-75abb99603cc-riscv64-linux-gnu.tar.gz
   16bd9b2c120df3c73d4c81f4f5016995cff7595b1a928197eba1e54971c8259014  guix-build-75abb99603cc/output/x86_64-linux-gnu/SHA256SUMS.part
   174d93c7aa1ede282eff25bc52942e3317b4f148af4ced7978ac8b15d160ed4dfc  guix-build-75abb99603cc/output/x86_64-linux-gnu/bitcoin-75abb99603cc-x86_64-linux-gnu-debug.tar.gz
   18f3fc20822a49b1372091021efa5e3a8a28a934e5235b4d9d43581f66df1486aa  guix-build-75abb99603cc/output/x86_64-linux-gnu/bitcoin-75abb99603cc-x86_64-linux-gnu.tar.gz
   

5. in depends/packages/expat.mk:15 in 790ae82ded outdated

   13   $(package)_config_opts=--disable-shared --without-docbook --without-tests --without-examples
   14   $(package)_config_opts += --disable-dependency-tracking --enable-option-checking
   15   $(package)_config_opts += --without-xmlwf
   16   $(package)_config_opts_linux=--with-pic
   17-  $(package)_cflags += -fno-lto
   18+  $(package)_cppflags += -D_DEFAULT_SOURCE
   

   ---

   ---

   

   hebasto commented at 10:59 pm on July 25, 2022:

   Should it be limited only for cases when glibc been used?

   ---

   

   fanquake commented at 8:59 am on July 26, 2022:

   Why? I think that over-complicated for what we need here.
6. 
   hebasto commented at 11:10 pm on July 25, 2022: member

   Currently, when building the expat package in depends, using -flto (LTO=1), inside Guix (and only inside Guix from what I’ve seen), the configure check fails…

   It fails not only inside Guix, but for every cross-compiling, e.g.,:

   0$ make -C depends expat_configured LTO=1 HOST=arm-linux-gnueabihf
   

   Considering that the _DEFAULT_SOURCE macro implies a wider feature set than the _POSIX_C_SOURCE one, is it safe to define the former for a dependency while the latter been used in https://github.com/bitcoin/bitcoin/blob/aa22009887eba6b6afdc530cfbcbe98e8aa34434/src/util/system.cpp#L45 ?

7. 
   fanquake commented at 9:00 am on July 26, 2022: member

   Considering that the _DEFAULT_SOURCE macro implies a wider feature set than the _POSIX_C_SOURCE one, is it safe to define the former for a dependency while the latter been used in

   I don’t really think that makes much of a difference here. _DEFAULT_SOURCE is still only used for the expat build.

8. in depends/packages/expat.mk:9 in 790ae82ded outdated

    6-$(package)_sha256_hash=cf032d0dba9b928636548e32b327a2d66b1aab63c4f4a13dd132c2d1d2f2fb6a
    7+$(package)_sha256_hash=f79b8f904b749e3e0d20afeadecf8249c55b2e32d4ebb089ae378df479dcaf25
    8 
    9+# -D_DEFAULT_SOURCE defines __USE_MISC, which exposes additional
   10+# definitions in endian.h, which are required for a working
   11+# endianess check in configure when building with -flto in Guix.
   

   ---

   ---

   

   hebasto commented at 9:26 am on July 26, 2022:

   nit: A comment could be improved basing on this comment.

   ---

   

   fanquake commented at 10:38 am on July 26, 2022:

   Adjusted the comment to remove the mention of Guix.
9. hebasto approved
10. 
    hebasto commented at 9:28 am on July 26, 2022: member

    ACK 790ae82dedafc6a61c285bd856471cfe3c9006ab, I have reviewed the code and it looks OK, I agree it can be merged.

    It’s worth to mention that in the span from v2.4.1 to v2.4.8 a few CVEs have been fixed.

    I’ve verified that no other adjustments of our depends build system required.

    Successfully can cross-compile the expat package in depends with LTO=1.

11. depends: expat 2.4.8 304452558c
12. depends: re-enable using -flto when building expat e838a98475
13. fanquake force-pushed on Jul 26, 2022
14. 
    fanquake commented at 10:39 am on July 26, 2022: member

    Addressed the suggestion and updated the PR description.
15. fanquake renamed this:
    depends: expat 2.4.8 & fix building with -flto when using Guix
    depends: expat 2.4.8 & fix building with -flto
    on Jul 26, 2022
16. hebasto approved
17. 
    hebasto commented at 11:34 am on July 26, 2022: member

    re-ACK e838a9847580527b8321d65e57b1c53cc2af6bf4, only suggested changes since my recent review.
18. 
    jarolrod commented at 5:09 pm on July 26, 2022: member

    code review ACK e838a9847580527b8321d65e57b1c53cc2af6bf4
19. 
    fanquake commented at 11:04 am on July 27, 2022: member

    It fails not only inside Guix, but for every cross-compiling, e.g.,: $ make -C depends expat_configured LTO=1 HOST=arm-linux-gnueabihf

    This doesn’t fail for me. What system is this on? What compiler are you using etc?

20. 
    hebasto commented at 11:30 am on July 27, 2022: member

    It fails not only inside Guix, but for every cross-compiling, e.g.,: $ make -C depends expat_configured LTO=1 HOST=arm-linux-gnueabihf

    This doesn’t fail for me. What system is this on? What compiler are you using etc?

     0$ git rev-parse HEAD
     131c1b14754574b0b0a54587e937fab66b3b85e39
     2$ git diff
     3diff --git a/depends/packages/expat.mk b/depends/packages/expat.mk
     4index 349319138e..50791ebc6e 100644
     5--- a/depends/packages/expat.mk
     6+++ b/depends/packages/expat.mk
     7@@ -9,7 +9,6 @@ define $(package)_set_vars
     8   $(package)_config_opts += --disable-dependency-tracking --enable-option-checking
     9   $(package)_config_opts += --without-xmlwf
    10   $(package)_config_opts_linux=--with-pic
    11-  $(package)_cflags += -fno-lto
    12 endef
    13 
    14 define $(package)_config_cmds
    15$ make -C depends expat_configured LTO=1 HOST=arm-linux-gnueabihf
    16...
    17configure: error: unknown endianness
    18 presetting ac_cv_c_bigendian=no (or yes) will help
    19...
    20$ arm-linux-gnueabihf-g++ --version
    21arm-linux-gnueabihf-g++ (Ubuntu 11.2.0-17ubuntu1) 11.2.0
    22Copyright (C) 2021 Free Software Foundation, Inc.
    23This is free software; see the source for copying conditions.  There is NO
    24warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    

21. 
    fanquake commented at 11:55 am on July 27, 2022: member

    I’ve opened an issue upstream: https://github.com/libexpat/libexpat/issues/618.
22. fanquake merged this on Jul 27, 2022
23. fanquake closed this on Jul 27, 2022

    ---

24. fanquake deleted the branch on Jul 27, 2022
25. 
    fanquake commented at 1:28 pm on July 27, 2022: member

    Not an expat bug, so I’ve submitted a report further upstream to autoconf: https://savannah.gnu.org/support/index.php?110687.
26. sidhujag referenced this in commit 2908b5c566 on Jul 27, 2022
27. 
    fanquake commented at 7:39 pm on July 27, 2022: member

    This has now also been fixed in autoconf upstream: https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=234fc6c86613ed3f366dd1d88996e4d5d85ee222.
28. bitcoin locked this on Jul 27, 2023

Contributors
fanquake DrahtBot jarolrod hebasto

Labels
Build system