configure: Support -fdebug-prefix-map and -fmacro-prefix-map #20353

pull ajtowns wants to merge 1 commits into bitcoin:master from ajtowns:202011-ccache-debug-prefix changing 1 files +4 −0
  1. ajtowns commented at 8:25 am on November 9, 2020: member

    When bitcoin is checked out in two directories (eg via git worktree) object files between the two will differ due to the full path being included in the debug section. -fdebug-prefix-map is used to replace this with “.” to avoid this unnecessary difference and allow ccache to share objects between worktrees (provided the source and compile options are the same).

    Also provide -fmacro-prefix-map if supported so that the working dir is not encoded in __FILE__ macros.

  2. ajtowns added the label Build system on Nov 9, 2020
  3. laanwj commented at 8:43 am on November 9, 2020: member

    TIL.

    Concept ACK.

    Something could be said for also passing it in the non-debug case. After all, even without explicit debug information enabled, gcc adds some level of basic debug information like symbols, and yes this includes file paths:

    0$ strings -a src/bitcoind | grep /home/$USER|wc -l
    11524
    

    Not all of these come from debug information though! They may also be from __FILE__ macro. Is there a flag for that? E.g.:

    0$ objdump --dwarf=info src/bitcoind|grep DW_AT_comp_dir
    1    <1a>   DW_AT_comp_dir    : (indirect string, offset: 0x9f): /home/…/projects/bitcoin/bitcoin/build/src
    2    <282c9>   DW_AT_comp_dir    : (indirect string, offset: 0x9f): /home/…/projects/bitcoin/bitcoin/build/src
    3    <1f2597>   DW_AT_comp_dir    : (indirect string, offset: 0x9f): /home/…/projects/bitcoin/bitcoin/build/src
    4    <263930>   DW_AT_comp_dir    : (indirect string, offset: 0x9f): /home/…/projects/bitcoin/bitcoin/build/src
    5    <399a44>   DW_AT_comp_dir    : (indirect string, offset: 0x9f): /home/…/projects/bitcoin/bitcoin/build/src
    6$ strip src/bitcoind
    7$ objdump --dwarf=info src/bitcoind|grep DW_AT_comp_dir
    8$ strings -a src/bitcoind | grep /home/$USER|wc -l
    9134
    

    allow ccache to share objects between worktrees (provided the source and compile options are the same).

    This sound sensible on first glance but does make me wonder how ccache “knows” this? When can it assume objects are portable between build directories. Does it explicitly pay attention to this flag?

    Edit: also mind that if you’re doing depends builds (I didn’t in my examples above), things become even more complicated because the depends are also compiled from specific directories inside the specific work tree.

  4. laanwj commented at 8:54 am on November 9, 2020: member

    They may also be from FILE macro. Is there a flag for that? E.g.:

    FWIW there’s, starting with GCC 8, also -fmacro-prefix-map=old=new. Go wild :slightly_smiling_face:

  5. sipa commented at 8:58 am on November 9, 2020: member
    And also a -ffile-prefix-map=old=new, apparently.
  6. ajtowns commented at 9:58 am on November 9, 2020: member

    This sound sensible on first glance but does make me wonder how ccache “knows” this? When can it assume objects are portable between build directories. Does it explicitly pay attention to this flag?

    Yeah:

    0$ strings /usr/bin/ccache | grep .-map
    1-fdebug-prefix-map=
    2-ffile-prefix-map=
    3-fmacro-prefix-map=
    

    Not all of these come from debug information though! They may also be from __FILE__ macro.

    Ah, gcc and clang are both giving me relative paths for __FILE__ even without any special options.

    also mind that if you’re doing depends builds (I didn’t in my examples above), things become even more complicated ..

    This doesn’t pass the options through to even the in-tree submodules, so univalue and secp will continue getting the absolute path encoded; they don’t take much time/space to compile though, so I figure that’s okay.

    And also a -ffile-prefix-map=old=new, apparently.

    I think clang supports -fdebug-prefix-map but not macro or file so I think it makes sense to just do debug and macro independently.

  7. ajtowns force-pushed on Nov 9, 2020
  8. ajtowns force-pushed on Nov 9, 2020
  9. ajtowns commented at 10:13 am on November 9, 2020: member
    Added -fmacro-prefix-map and made them conditional on ccache being enabled.
  10. ajtowns renamed this:
    configure: Support -fdebug-prefix-map
    configure: Support -fdebug-prefix-map and -fmacro-prefix-map
    on Nov 9, 2020
  11. configure: Support -f{debug,macro}-prefix-map
    When bitcoin is checked out in two directories (eg via git worktree)
    object files between the two will differ due to the full path being
    included in the debug section. -fdebug-prefix-map is used to replace
    this with "." to avoid this unnecessary difference and allow ccache to
    share objects between worktrees (provided the source and compile options
    are the same).
    
    Also provide -fmacro-prefix-map if supported so that the working dir is
    not encoded in __FILE__ macros.
    7abac98d3e
  12. ajtowns force-pushed on Nov 9, 2020
  13. laanwj commented at 10:36 am on November 9, 2020: member

    Yeah: $ strings /usr/bin/ccache | grep .-map -fdebug-prefix-map= -ffile-prefix-map= -fmacro-prefix-map=

    Thanks, good to know.

    Ah, gcc and clang are both giving me relative paths for FILE even without any special options.

    It depends on what is passed to the compiler. I think for the macros they embed those paths literally. For the DWARF information, the absolute path is used (and there’s also header paths, not just compilation unit paths). In any case there seems variance between systems and compilers here. As always…

    Added -fmacro-prefix-map and made them conditional on ccache being enabled

    Concept ACK on making it depend on –enable-ccache.

  14. practicalswift commented at 12:02 pm on November 9, 2020: contributor
    Concept ACK: path name leaking should be avoided
  15. ajtowns commented at 6:21 am on January 10, 2021: member
    Anyone want to bump their acks from concept to reality?
  16. practicalswift commented at 3:55 pm on January 10, 2021: contributor
    cr ACK 7abac98d3e3c1bc8ad66cb5c05184b9c5cc674d5: patch looks correct
  17. fanquake commented at 7:42 am on March 24, 2021: member

    Concept ACK. Will test.

    I think clang supports -fdebug-prefix-map but not macro or file so I think it makes sense to just do debug and macro independently.

    There’s: -fdebug-prefix-map=OLD=NEW GCC (forever) & Clang 3.8+ -fmacro-prefix-map=OLD=NEW GCC 8+ Clang 10+ and also -ffile-prefix-map=OLD=NEW GCC 8+ & Clang 10+. which is just equal to -fdebug-prefix-map -fmacro-prefix-map; so setting both seems best for now. Could add a note that what were are doing is the equivalent to -ffile-prefix-map.

    Looks like there’s even a BUILD_PATH_PREFIX_MAP environment var under discussion.

  18. fanquake approved
  19. fanquake commented at 11:56 am on April 20, 2021: member

    ACK 7abac98d3e3c1bc8ad66cb5c05184b9c5cc674d5

    Tested this using git worktrees and ccache. Doing something like:

     0# master
     1DIR A
     2	./autogen.sh && ./configure --without-gui
     3	gmake -C src/bitcoind
     4	ccache --zero-stats
     5DIR B (git worktree)
     6	./autogen.sh && ./configure --without-gui
     7	gmake src/bitcoind -j8
     8	hit rate:
     9		cache miss                           217
    10		cache hit rate                      0.00 %
    11
    12# cleanup, remove worktree, nuke ccache
    13
    14[#20353](/bitcoin-bitcoin/20353/)
    15DIR A
    16	./autogen.sh && ./configure --without-gui
    17	gmake -C src/bitcoind
    18	ccache --zero-stats
    19DIR B (git worktree different again)
    20	./autogen.sh && ./configure --without-gui
    21	gmake src/bitcoind -j8
    22	hit rate:
    23		cache miss                             0
    24		cache hit rate                    100.00 %
    

    Also tested when building using depends, configured via: ./autogen.sh && CONFIG_SITE=bitcoin/depends/x86_64-apple-darwin19.6.0/share/config.site ./configure

  20. MarcoFalke commented at 12:02 pm on April 20, 2021: member
    Wouldn’t it be easier to set CCACHE_NOHASHDIR=1? Sure, the paths in the debug information might be wrong, but does it matter?
  21. ajtowns commented at 12:37 pm on April 20, 2021: member

    Wouldn’t it be easier to set CCACHE_NOHASHDIR=1? Sure, the paths in the debug information might be wrong, but does it matter?

    AIUI, if the __FILE__ macro doesn’t match (ie what macro-prefix-map fixes), the preprocessed input to ccache will be different, causing ccache misses. debug-prefix-map fixes the deug info paths, and then triggers ccache to automatically set CCACHE_NOHASHDIR=1. So these seems like the “proper” way to do it and it’s not a big patch, so setting CCACHE_NOHASHDIR=1 doesn’t seem that much easier?

  22. MarcoFalke commented at 7:02 pm on April 20, 2021: member
    Sure, just wanted to ask because the discussion didn’t mention CCACHE_NOHASHDIR at all.
  23. fanquake merged this on Apr 21, 2021
  24. fanquake closed this on Apr 21, 2021

  25. sidhujag referenced this in commit 96346b8092 on Apr 21, 2021
  26. PastaPastaPasta referenced this in commit 81126238a2 on Jun 27, 2021
  27. PastaPastaPasta referenced this in commit 6c3dc4286e on Jun 28, 2021
  28. PastaPastaPasta referenced this in commit 8f73cf78c9 on Jun 29, 2021
  29. PastaPastaPasta referenced this in commit 42a5ee69b1 on Jul 1, 2021
  30. PastaPastaPasta referenced this in commit 6da524570b on Jul 1, 2021
  31. PastaPastaPasta referenced this in commit 2206fad2a7 on Jul 15, 2021
  32. PastaPastaPasta referenced this in commit 94d62fef23 on Jul 15, 2021
  33. PastaPastaPasta referenced this in commit 9dc2ed4208 on Jul 16, 2021
  34. fanquake referenced this in commit 5b7210c874 on Oct 11, 2021
  35. sidhujag referenced this in commit 553b7e7dfa on Oct 11, 2021
  36. PastaPastaPasta referenced this in commit 87caa91a99 on Apr 3, 2022
  37. gades referenced this in commit f765763ac0 on May 9, 2022
  38. gades referenced this in commit 3c7c32baf4 on May 9, 2022
  39. DrahtBot locked this on Aug 18, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-30 00:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me