Source code mapping for debugger has changed since cmake #31204

developer-notes.md says:

1. Configure source file mapping.

For gdb create or append to .gdbinit file:

set substitute-path ./src /path/to/project/root/src

For lldb create or append to .lldbinit file:

settings set target.source-map ./src /path/to/project/root/src

But I found I needed to create a .lldbinit file with these lines:

settings append target.source-map build/src/test/ /Users/matthewzipkin/Desktop/work/bitcoin/
settings append target.source-map build/src/src/ /Users/matthewzipkin/Desktop/work/bitcoin/src/

Here's an example of lldb's confused guess of source location:

(lldb) image lookup -vn  ProcessMessages
1 match found in /Users/matthewzipkin/Desktop/work/bitcoin/build/src/bitcoind:
...
      LineEntry: [0x00000001003302f4-0x000000010033032c): build/src/src/net_processing.cpp:5373

Could just be fixed by updating developer-notes.md -- would we ever include a .lldbinit file in the repo itself? Or am I missing something else.

After many hours, I could make it work.

Add this to ~/.lldbinit to make it load a local .lldbinit file: settings set target.load-cwd-lldbinit true. This is a convenience so that I can keep everything related to Bitcoin Core in its repo.

In the Bitcoin Core repo folder, add a .lldbinit file with (mine is located at /Users/jose.edil/2-development/bitcoin/bitcoin-core-dev, adjust to your path):

settings set target.source-map /Users/jose.edil/2-development/bitcoin/bitcoin-core-dev/build/src/ /Users/jose.edil/2-development/bitcoin/bitcoin-core-dev/src/

Compiled with: $ cmake -B build -DBoost_INCLUDE_DIR=/opt/local/libexec/boost/1.87/include -DCMAKE_C_COMPILER=clang-mp-18 -DCMAKE_CXX_COMPILER=clang++-mp-18 -DCMAKE_BUILD_TYPE=Debug -DWITH_CCACHE=OFF

In particular, I suspect that -fdebug-prefix-map is not doing it's thing:

(lldb) set list target.source-map
  target.source-map -- Source path remappings apply substitutions to the paths of source files, typically
                       needed to debug from a different host than the one that built the target.  The
                       source-map property consists of an array of pairs, the first element is a path prefix,
                       and the second is its replacement.  The syntax is `prefix1 replacement1 prefix2
                       replacement2...`.  The pairs are checked in order, the first prefix that matches is
                       used, and that prefix is substituted with the replacement.  A common pattern is to use
                       source-map in conjunction with the clang -fdebug-prefix-map flag.  In the build, use
                       `-fdebug-prefix-map=/path/to/build_dir=.` to rewrite the host specific build directory
                       to `.`.  Then for debugging, use `settings set target.source-map . /path/to/local_dir`
                       to convert `.` to a valid local path.

Configure summary
=================
…
C++ compiler flags …................. -O0 -ftrapv -g3 -std=c++20 -fPIC -fdebug-prefix-map=/Users/jose.edil/2-development/bitcoin/bitcoin-core-dev/src=. -fmacro-prefix-map=/Users/jose.edil/2-development/bitcoin/bitcoin-core-dev/src=. …

I would expect it to use paths like ./<file> for debugging symbols, but it's not the case (note the LineEntry field below):

$ lldb build/src/bitcoind
(lldb) target create "build/src/bitcoind"
Current executable set to '/Users/jose.edil/2-development/bitcoin/bitcoin-core-dev/build/src/bitcoind' (arm64).

(lldb) image lookup -vn main
1 match found in /Users/jose.edil/2-development/bitcoin/bitcoin-core-dev/build/src/bitcoind:
        Address: bitcoind[0x00000001000050f0] (bitcoind.__TEXT.__text + 1680)
        Summary: bitcoind`main at bitcoind.cpp:256
         Module: file = "/Users/jose.edil/2-development/bitcoin/bitcoin-core-dev/build/src/bitcoind", arch = "arm64"
    CompileUnit: id = {0x00000000}, file = "/Users/jose.edil/2-development/bitcoin/bitcoin-core-dev/build/src/bitcoind.cpp", language = "c++14"
       Function: id = {0x400000000006a2ee}, name = "main", range = [0x00000001000050f0-0x0000000100005398)
       FuncType: id = {0x400000000006a2ee}, byte-size = 0, decl = bitcoind.cpp:255, compiler_type = "int (int, char **)"
         Blocks: id = {0x400000000006a2ee}, range = [0x1000050f0-0x100005398)
      LineEntry: [0x00000001000050f0-0x0000000100005128): /Users/jose.edil/2-development/bitcoin/bitcoin-core-dev/build/src/bitcoind.cpp:256
         Symbol: id = {0x00023777}, range = [0x00000001000050f0-0x0000000100005398), name="main"
       Variable: id = {0x400000000006a308}, name = "argc", type = "int", valid ranges = <block>, location = DW_OP_breg31 WSP+88, decl = bitcoind.cpp:255
       Variable: id = {0x400000000006a317}, name = "argv", type = "char **", valid ranges = <block>, location = DW_OP_breg31 WSP+80, decl = bitcoind.cpp:255
       Variable: id = {0x400000000006a326}, name = "node", type = "NodeContext", valid ranges = <block>, location = DW_OP_breg31 WSP+136, decl = bitcoind.cpp:262
       Variable: id = {0x400000000006a336}, name = "exit_status", type = "int", valid ranges = <block>, location = DW_OP_breg31 WSP+132, decl = bitcoind.cpp:263
       Variable: id = {0x400000000006a346}, name = "init", type = "unique_ptr<interfaces::Init, std::__1::default_delete<interfaces::Init> >", valid ranges = <block>, location = DW_OP_breg31 WSP+120, decl = bitcoind.cpp:264
       Variable: id = {0x400000000006a356}, name = "args", type = "ArgsManager &", valid ranges = <block>, location = DW_OP_breg31 WSP+56, decl = bitcoind.cpp:277

But I found I needed to create a .lldbinit file with these lines: @pinheadmz Has this fixed the mapping in your system permanently? I have not tried this yet and I observe that the source file detection fails because it seems to be generated in a new location everytime I build again like in the below errors.

Breakpoint at /Users/rkrux/projects/bitcoin/src/script/descriptor.cpp:1512 could not be resolved, but a valid location was found at /Users/rkrux/projects/bitcoin/build/src/script/descriptor.cpp:1512

Breakpoint at /Users/rkrux/projects/bitcoin/src/wallet/wallet.cpp:1326 could not be resolved, but a valid location was found at /Users/rkrux/projects/bitcoin/build/src/wallet/wallet/wallet.cpp:1326

I will try this solution.

I'm still tweaking the .lldbinit file actually -- sometimes I'll be debugging with lldb and end up in a new directory I hadn't linked yet like /fuzz or something and it'll break again. So I suspect there is a simple catch all solution I just haven't figured it out yet

It's not clear if this is something that needs to be fixed, (is a regression), or if it's a behaviour change that needs to be documented in some way, (in the developer notes), or something else? cc @hebasto

I find it okay to have to set a mapping for the root directory, but I currently need things like:

"/home/hodlinator/b2/build/src/test/util/test/util/": "/home/hodlinator/b2/src/test/util/"

Note how the subdirectory needs to be repeated for it to work: test/util/test/util/

Same thing happened for rkrux above:

...valid location was found at /Users/rkrux/projects/bitcoin/build/src/wallet/wallet/wallet.cpp

@hodlinator

Mind submitting a PR?

Any idea where to start looking for the duplicate subdirectory issue? Where do we get the virtual build/src/ paths to begin with?

This seems to work for me on Linux without any .lldbinit file:

will@ubuntu in …/src/core/bitcoin on  master [$] via △ v3.31.6 : 🐍 (core)
$ just clean build-depends
depends build running
copying packages: boost libevent qt expat libxcb xcb_proto libXau xproto freetype fontconfig libxkbcommon libxcb_util libxcb_util_render libxcb_util_keysyms libxcb_util_image libxcb_util_wm qrencode bdb sqlite systemtap zeromq
to: /home/will/src/core/bitcoin/depends/x86_64-pc-linux-gnu
To build Bitcoin Core with these packages, pass '--toolchain /home/will/src/core/bitcoin/depends/x86_64-pc-linux-gnu/toolchain.cmake' to the first CMake invocation.
depends build complete
configure running
configure complete
build running
build complete

will@ubuntu in …/src/core/bitcoin on  master [$] via △ v3.31.6 : 🐍 (core) took 2m13s
$ lldb --version
lldb version 19.1.7

will@ubuntu in …/src/core/bitcoin on  master [$] via △ v3.31.6 : 🐍 (core) took 27s
$ lldb
(lldb) target create build/bin/bitcoind
Current executable set to '/home/will/src/core/bitcoin/build/bin/bitcoind' (x86_64).
(lldb) image lookup -vn  ProcessMessages
1 match found in /home/will/src/core/bitcoin/build/bin/bitcoind:
        Address: bitcoind[0x00000000004c7d20] (bitcoind.PT_LOAD[1]..text + 1016240)
        Summary: bitcoind`(anonymous namespace)::PeerManagerImpl::ProcessMessages(CNode*, std::atomic<bool>&) at net_processing.cpp:4927
         Module: file = "/home/will/src/core/bitcoin/build/bin/bitcoind", arch = "x86_64"
    CompileUnit: id = {0x0000001d}, file = "/home/will/src/core/bitcoin/src/net_processing.cpp", language = "c++14"
       Function: id = {0x007eedbb}, name = "(anonymous namespace)::PeerManagerImpl::ProcessMessages(CNode*, std::atomic<bool>&)", mangled = "_ZN12_GLOBAL__N_115PeerManagerImpl15ProcessMessagesEP5CNodeRSt6atomicIbE", range = [0x00000000004c7d20-0x00000000004c8b17)
       FuncType: id = {0x007eedbb}, byte-size = 0, decl = net_processing.cpp:495, compiler_type = "_Bool (class CNode *, struct std::atomic<_Bool> &)"
         Blocks: id = {0x007eedbb}, range = [0x004c7d20-0x004c8b17)
      LineEntry: [0x00000000004c7d20-0x00000000004c7d4f): /home/will/src/core/bitcoin/src/net_processing.cpp:4927
         Symbol: id = {0x00001b57}, range = [0x00000000004c7d20-0x00000000004c8b17), name="(anonymous namespace)::PeerManagerImpl::ProcessMessages(CNode*, std::atomic<bool>&)", mangled="_ZN12_GLOBAL__N_115PeerManagerImpl15ProcessMessagesEP5CNodeRSt6atomicIbE"
       Variable: id = {0x007eedf7}, name = "peer", type = "PeerRef", valid ranges = <block>, location = DW_OP_fbreg +80, decl = net_processing.cpp:4931
       Variable: id = {0x007eee06}, name = "poll_result", type = "optional<std::pair<CNetMessage, bool> >", valid ranges = <block>, location = DW_OP_fbreg +288, decl = net_processing.cpp:4962

(lldb)

Perhaps #31161 affected this, or is it a macOS-specific issue? Is it still an issue on current master?

<details> <summary>configure log</summary>

Configuring secp256k1 subtree...


secp256k1 configure summary
===========================
Build artifacts:
  library type ........................ Static
Optional modules:
  ECDH ................................ OFF
  ECDSA pubkey recovery ............... ON
  extrakeys ........................... ON
  schnorrsig .......................... ON
  musig ............................... OFF
  ElligatorSwift ...................... ON
Parameters:
  ecmult window size .................. 15
  ecmult gen table size ............... 86 KiB
Optional features:
  assembly ............................ x86_64
  external callbacks .................. OFF
Optional binaries:
  benchmark ........................... OFF
  noverify_tests ...................... ON
  tests ............................... ON
  exhaustive tests .................... ON
  ctime_tests ......................... OFF
  examples ............................ OFF

Cross compiling ....................... FALSE
Valgrind .............................. OFF
Preprocessor defined macros ........... ENABLE_MODULE_ELLSWIFT=1 ENABLE_MODULE_SCHNORRSIG=1 ENABLE_MODULE_EXTRAKEYS=1 ENABLE_MODULE_RECOVERY=1 ECMULT_WINDOW_SIZE=15 COMB_BLOCKS=43 COMB_TEETH=6 USE_ASM_X86_64=1
C compiler ............................ Clang 19.1.7, /usr/lib/ccache/clang
CFLAGS ................................ -pipe -std=c11
Compile options ....................... -pedantic -Wall -Wcast-align -Wconditional-uninitialized -Wextra -Wnested-externs -Wno-long-long -Wno-overlength-strings -Wno-unused-function -Wreserved-identifier -Wshadow -Wstrict-prototypes -Wundef
Build type:
 - CMAKE_BUILD_TYPE ................... RelWithDebInfo
 - CFLAGS ............................. -O2 -O2 -g 
 - LDFLAGS for executables ............ 
 - LDFLAGS for shared libraries ....... 



Configure summary
=================
Executables:
  bitcoind ............................ ON
  bitcoin-node (multiprocess) ......... OFF
  bitcoin-qt (GUI) .................... ON
  bitcoin-gui (GUI, multiprocess) ..... OFF
  bitcoin-cli ......................... ON
  bitcoin-tx .......................... ON
  bitcoin-util ........................ ON
  bitcoin-wallet ...................... ON
  bitcoin-chainstate (experimental) ... OFF
  libbitcoinkernel (experimental) ..... OFF
Optional features:
  wallet support ...................... ON
   - legacy wallets (Berkeley DB) ..... ON
  external signer ..................... ON
  ZeroMQ .............................. ON
  USDT tracing ........................ ON
  QR code (GUI) ....................... ON
  DBus (GUI, Linux only) .............. ON
Tests:
  test_bitcoin ........................ ON
  test_bitcoin-qt ..................... ON
  bench_bitcoin ....................... OFF
  fuzz binary ......................... OFF

Cross compiling ....................... FALSE
C++ compiler .......................... Clang 19.1.7, /usr/lib/ccache/clang++
CMAKE_BUILD_TYPE ...................... RelWithDebInfo
Preprocessor defined macros ........... 
C++ compiler flags .................... -pipe -std=c++20 -O2 -O2 -g -std=c++20 -fPIC -fdebug-prefix-map=/home/will/src/core/bitcoin/src=. -fmacro-prefix-map=/home/will/src/core/bitcoin/src=. -Wall -Wextra -Wgnu -Wformat -Wformat-security -Wvla -Wshadow-field -Wthread-safety -Wloop-analysis -Wredundant-decls -Wunused-member-function -Wdate-time -Wconditional-uninitialized -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wdocumentation -Wself-assign -Wundef -Wno-unused-parameter -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection
Linker flags .......................... -pipe -std=c++20 -O2 -O2 -g -fuse-ld=mold -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie

NOTE: The summary above may not exactly match the final applied build flags
      if any additional CMAKE_* or environment variables have been modified.
      To see the exact flags applied, build with the --verbose option.

Attempt to harden executables ......... ON
Treat compiler warnings as errors ..... OFF
Use ccache for compiling .............. ON

</details>

I find it okay to have to set a mapping for the root directory, but I currently need things like:

"/home/hodlinator/b2/build/src/test/util/test/util/": "/home/hodlinator/b2/src/test/util/"

Note how the subdirectory needs to be repeated for it to work: test/util/test/util/

Same thing happened for rkrux above:

...valid location was found at /Users/rkrux/projects/bitcoin/build/src/wallet/wallet/wallet.cpp

I am also seeing non-existent folders in #31957:

  ./bld-c/src/crypto/./src/compat/byteswap.h
  ./bld-c/src/crypto/./src/crypto/sha256.cpp
  ./bld-c/src/crypto/./src/crypto/sha512.cpp
  ./bld-c/src/test/./src/test/crypto_tests.cpp
  ./bld-c/src/test/./src/test/util_tests.cpp
  ./bld-c/src/util/./src/random.cpp
  ./bld-c/src/util/./src/util/string.cpp

With the steps to reproduce there, it is a cmake regression. I am not sure if callgrind_annotate allows mapping, so it may be best to fix this inside the build system itself.

#31204 (comment):

This seems to work for me on Linux without any .lldbinit file:

I tested v28.1 on Ubuntu 24.04 with GCC 13.3:

$ ./configure --enable-ccache
$ make -j $(nproc) src/bitcoind
$ lldb
(lldb) version
lldb version 18.1.3
(lldb) target create src/bitcoind
Current executable set to '/home/hebasto/dev/bitcoin/src/bitcoind' (x86_64).
(lldb) settings show target.source-map
target.source-map (path-map) =
(lldb) image lookup -vn ProcessMessages
1 match found in /home/hebasto/dev/bitcoin/src/bitcoind:
        Address: bitcoind[0x00000000001e31e0] (bitcoind.PT_LOAD[1]..text + 1776496)
        Summary: bitcoind`(anonymous namespace)::PeerManagerImpl::ProcessMessages(CNode*, std::atomic<bool>&)
         Module: file = "/home/hebasto/dev/bitcoin/src/bitcoind", arch = "x86_64"
    CompileUnit: id = {0x00000008}, file = "src/net_processing.cpp", language = "c++14"
      LineEntry: [0x00000000001e31e0-0x00000000001e31f1): src/net_processing.cpp:5380:1
         Symbol: id = {0x000003ed}, range = [0x00000000001e31e0-0x00000000001e3cf0), name="(anonymous namespace)::PeerManagerImpl::ProcessMessages(CNode*, std::atomic<bool>&)", mangled="_ZN12_GLOBAL__N_115PeerManagerImpl15ProcessMessagesEP5CNodeRSt6atomicIbE"

(lldb) settings set target.source-map ./src /home/hebasto/dev/bitcoin/src
(lldb) settings show target.source-map
target.source-map (path-map) =
[0] "src" -> "/home/hebasto/dev/bitcoin/src"

(lldb) image lookup -vn ProcessMessages
1 match found in /home/hebasto/dev/bitcoin/src/bitcoind:
        Address: bitcoind[0x00000000001e31e0] (bitcoind.PT_LOAD[1]..text + 1776496)
        Summary: bitcoind`(anonymous namespace)::PeerManagerImpl::ProcessMessages(CNode*, std::atomic<bool>&)
         Module: file = "/home/hebasto/dev/bitcoin/src/bitcoind", arch = "x86_64"
    CompileUnit: id = {0x00000008}, file = "src/net_processing.cpp", language = "c++14"
      LineEntry: [0x00000000001e31e0-0x00000000001e31f1): src/net_processing.cpp:5380:1
         Symbol: id = {0x000003ed}, range = [0x00000000001e31e0-0x00000000001e3cf0), name="(anonymous namespace)::PeerManagerImpl::ProcessMessages(CNode*, std::atomic<bool>&)", mangled="_ZN12_GLOBAL__N_115PeerManagerImpl15ProcessMessagesEP5CNodeRSt6atomicIbE"

So the guideline for lldb does not work in my build environment, even for the pre-CMake branch.

Is this a blocker for 29.0? Seems fine to backport to 29.1, if it is ready later, no?

definitely not a blocker but I think I figured it out: https://github.com/bitcoin/bitcoin/pull/32205

For reference, inside GDB, the following worked for me:

(gdb) set substitute-path ./ /path/to/project/root/src

Not using the (outdated?) gdbinit advice:

set substitute-path ./src /path/to/project/root/src

As an alternative, i could use debugedit:

debugedit --base-dir ./ --dest-dir /path/to/project/root/src     bld-cmake/bin/test_kernel

@pinheadmz

Does the approach proposed in both #31957 (comment) and #31957 (comment) make any difference to this issue?

Hi, I proposed an alternative to #32205, instead of modifying the doc making each user configure the path, do it dynamically on cmake: https://github.com/bitcoin/bitcoin/pull/34081

Fixed in #34281. Tested on a fresh install of Ubuntu 26.04 LTS:

• Master:

# lldb
(lldb) version
lldb version 21.1.7
(lldb) target create ./bld-c/bin/bitcoind
Current executable set to '/b-c/bld-c/bin/bitcoind' (x86_64).
(lldb) settings show target.source-map
target.source-map (path-map) =
(lldb) breakpoint set --name AppInitSanityChecks
Breakpoint 1: where = bitcoind`AppInitSanityChecks(kernel::Context const&) at init.cpp:1161:1, address = 0x00000000001981c0
(lldb) run
error: Cannot launch '/b-c/bld-c/bin/bitcoind': personality set failed: Function not implemented
(lldb) settings set target.disable-aslr false
(lldb) run
Process 20302 launched: '/b-c/bld-c/bin/bitcoind' (x86_64)
Process 20302 stopped
* thread [#1](/bitcoin-bitcoin/1/), name = 'bitcoind', stop reason = breakpoint 1.1
    frame [#0](/bitcoin-bitcoin/0/): 0x0000563485dce1c0 bitcoind`AppInitSanityChecks(kernel=0x00005634923d71d0) at init.cpp:1161:1
(lldb) 

• My fix:

# lldb
(lldb) version
lldb version 21.1.7
(lldb) target create ./bld-c/bin/bitcoind
Current executable set to '/b-c-pull-34281/bld-c/bin/bitcoind' (x86_64).
(lldb) settings show target.source-map
target.source-map (path-map) =
(lldb) breakpoint set --name AppInitSanityChecks
Breakpoint 1: where = bitcoind`AppInitSanityChecks(kernel::Context const&) at init.cpp:1161:1, address = 0x00000000001981c0
(lldb) run
error: Cannot launch '/b-c-pull-34281/bld-c/bin/bitcoind': personality set failed: Function not implemented
(lldb) settings set target.disable-aslr false
(lldb) run
Process 20308 launched: '/b-c-pull-34281/bld-c/bin/bitcoind' (x86_64)
Process 20308 stopped
* thread [#1](/bitcoin-bitcoin/1/), name = 'bitcoind', stop reason = breakpoint 1.1
    frame [#0](/bitcoin-bitcoin/0/): 0x0000564415caf1c0 bitcoind`AppInitSanityChecks(kernel=0x000056443ed4d1d0) at init.cpp:1161:1
   1158	}
   1159	
   1160	bool AppInitSanityChecks(const kernel::Context& kernel)
-> 1161	{
   1162	    // ********************************************************* Step 4: sanity checks
   1163	    auto result{kernel::SanityChecks(kernel)};
   1164	    if (!result) {
(lldb)