Segmentation fault when compiling with libfuzzer and lto (x86_64) #25961

issue MarcoFalke openend this issue on August 30, 2022
  1. MarcoFalke commented at 7:08 pm on August 30, 2022: member

    Steps to reproduce:

    • Install fresh Ubuntu Jammy operating system
    • export DEBIAN_FRONTEND=noninteractive && apt update && apt install curl wget htop git vim ccache -y && git clone https://github.com/bitcoin/bitcoin.git bitcoin-core && cd bitcoin-core && apt install build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq libevent-dev libboost-dev libsqlite3-dev libdb++-dev clang llvm libc++-dev libc++abi-dev -y && ./autogen.sh && ./configure CC='clang -flto' CXX='clang++ -flto' --enable-fuzz --with-sanitizers=fuzzer && make -j$(nproc)

    Output:

    0  AR       minisketch/libminisketch.a
1  AR       libbitcoin_wallet.a
2  CXXLD    test/fuzz/fuzz
3clang: error: unable to execute command: Segmentation fault (core dumped)

    The same does not happen when compiling “normally” (dropping --with-sanitizers=fuzzer).

  2. MarcoFalke added the label Bug on Aug 30, 2022
  3. fanquake added the label Build system on Aug 30, 2022
  4. MarcoFalke commented at 7:25 pm on August 30, 2022: member

    Likely unrelated, but interestingly, with clang-12 it always fails and prints error adding symbols: archive has no index; run ranlib to add one. To reproduce:

    • Fresh install of Focal or Jammy
    • export DEBIAN_FRONTEND=noninteractive && apt update && apt install curl wget htop git vim ccache -y && git clone https://github.com/bitcoin/bitcoin.git bitcoin-core && cd bitcoin-core && apt install build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq libevent-dev libboost-dev libsqlite3-dev clang-12 llvm-12 -y && ./autogen.sh && ./configure CC='clang-12 -flto' CXX='clang++-12 -flto' --enable-fuzz && make -j$(nproc)
  5. MarcoFalke commented at 7:30 pm on August 30, 2022: member
    clang-13 seems to be working :man_shrugging:
  6. MarcoFalke commented at 8:12 am on August 31, 2022: member

    Going back to clang-14 and using different linkers:

    • -fuse-ld=ld: Segfault
    • -fuse-ld=gold: Segfault
    • -fuse-ld=mold (1.4): Segfault
    • -fuse-ld=lld: Segfault with error below

    lld error:

     0PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 1Stack dump:
 20.	Program arguments: /usr/bin/ld.lld -pie -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o test/fuzz/fuzz /lib/x86_64-linux-gnu/Scrt1.o /lib/x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/11/crtbeginS.o -L/usr/bin/../lib/gcc/x86_64-linux-gnu/11 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/llvm-14/bin/../lib -L/lib -L/usr/lib -plugin-opt=mcpu=x86-64 -plugin-opt=O2 --whole-archive /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a --no-whole-archive --whole-archive /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer_interceptors-x86_64.a --no-whole-archive -lstdc++ --whole-archive /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.ubsan_standalone-x86_64.a --no-whole-archive --dynamic-list=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.ubsan_standalone-x86_64.a.syms --whole-archive /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.ubsan_standalone_cxx-x86_64.a --no-whole-archive --dynamic-list=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.ubsan_standalone_cxx-x86_64.a.syms -z relro -z now -z separate-code wallet/test/fuzz/test_fuzz_fuzz-coinselection.o wallet/test/fuzz/test_fuzz_fuzz-notifications.o test/fuzz/fuzz-addition_overflow.o test/fuzz/fuzz-addrman.o test/fuzz/fuzz-asmap.o test/fuzz/fuzz-asmap_direct.o test/fuzz/fuzz-autofile.o test/fuzz/fuzz-banman.o test/fuzz/fuzz-base_encode_decode.o test/fuzz/fuzz-bech32.o test/fuzz/fuzz-bitdeque.o test/fuzz/fuzz-block.o test/fuzz/fuzz-block_header.o test/fuzz/fuzz-blockfilter.o test/fuzz/fuzz-bloom_filter.o test/fuzz/fuzz-buffered_file.o test/fuzz/fuzz-chain.o test/fuzz/fuzz-checkqueue.o test/fuzz/fuzz-coins_view.o test/fuzz/fuzz-connman.o test/fuzz/fuzz-crypto.o test/fuzz/fuzz-crypto_aes256.o test/fuzz/fuzz-crypto_aes256cbc.o test/fuzz/fuzz-crypto_chacha20.o test/fuzz/fuzz-crypto_chacha20_poly1305_aead.o test/fuzz/fuzz-crypto_common.o test/fuzz/fuzz-crypto_diff_fuzz_chacha20.o test/fuzz/fuzz-crypto_hkdf_hmac_sha256_l32.o test/fuzz/fuzz-crypto_poly1305.o test/fuzz/fuzz-cuckoocache.o test/fuzz/fuzz-decode_tx.o test/fuzz/fuzz-descriptor_parse.o test/fuzz/fuzz-deserialize.o test/fuzz/fuzz-eval_script.o test/fuzz/fuzz-fee_rate.o test/fuzz/fuzz-fees.o test/fuzz/fuzz-flatfile.o test/fuzz/fuzz-float.o test/fuzz/fuzz-golomb_rice.o test/fuzz/fuzz-hex.o test/fuzz/fuzz-http_request.o test/fuzz/fuzz-i2p.o test/fuzz/fuzz-integer.o test/fuzz/fuzz-key.o test/fuzz/fuzz-key_io.o test/fuzz/fuzz-kitchen_sink.o test/fuzz/fuzz-load_external_block_file.o test/fuzz/fuzz-locale.o test/fuzz/fuzz-merkleblock.o test/fuzz/fuzz-message.o test/fuzz/fuzz-miniscript.o test/fuzz/fuzz-minisketch.o test/fuzz/fuzz-muhash.o test/fuzz/fuzz-multiplication_overflow.o test/fuzz/fuzz-net.o test/fuzz/fuzz-net_permissions.o test/fuzz/fuzz-netaddress.o test/fuzz/fuzz-netbase_dns_lookup.o test/fuzz/fuzz-node_eviction.o test/fuzz/fuzz-p2p_transport_serialization.o test/fuzz/fuzz-parse_hd_keypath.o test/fuzz/fuzz-parse_iso8601.o test/fuzz/fuzz-parse_numbers.o test/fuzz/fuzz-parse_script.o test/fuzz/fuzz-parse_univalue.o test/fuzz/fuzz-policy_estimator.o test/fuzz/fuzz-policy_estimator_io.o test/fuzz/fuzz-pow.o test/fuzz/fuzz-prevector.o test/fuzz/fuzz-primitives_transaction.o test/fuzz/fuzz-process_message.o test/fuzz/fuzz-process_messages.o test/fuzz/fuzz-protocol.o test/fuzz/fuzz-psbt.o test/fuzz/fuzz-random.o test/fuzz/fuzz-rbf.o test/fuzz/fuzz-rolling_bloom_filter.o test/fuzz/fuzz-rpc.o test/fuzz/fuzz-script.o test/fuzz/fuzz-script_assets_test_minimizer.o test/fuzz/fuzz-script_bitcoin_consensus.o test/fuzz/fuzz-script_descriptor_cache.o test/fuzz/fuzz-script_flags.o test/fuzz/fuzz-script_format.o test/fuzz/fuzz-script_interpreter.o test/fuzz/fuzz-script_ops.o test/fuzz/fuzz-script_sigcache.o test/fuzz/fuzz-script_sign.o test/fuzz/fuzz-scriptnum_ops.o test/fuzz/fuzz-secp256k1_ec_seckey_import_export_der.o test/fuzz/fuzz-secp256k1_ecdsa_signature_parse_der_lax.o test/fuzz/fuzz-signature_checker.o test/fuzz/fuzz-signet.o test/fuzz/fuzz-socks5.o test/fuzz/fuzz-span.o test/fuzz/fuzz-spanparsing.o test/fuzz/fuzz-string.o test/fuzz/fuzz-strprintf.o test/fuzz/fuzz-system.o test/fuzz/fuzz-timedata.o test/fuzz/fuzz-torcontrol.o test/fuzz/fuzz-transaction.o test/fuzz/fuzz-tx_in.o test/fuzz/fuzz-tx_out.o test/fuzz/fuzz-tx_pool.o test/fuzz/fuzz-txorphan.o test/fuzz/fuzz-txrequest.o test/fuzz/fuzz-utxo_snapshot.o test/fuzz/fuzz-validation_load_mempool.o test/fuzz/fuzz-versionbits.o -lpthread libtest_util.a libtest_fuzz.a libbitcoin_node.a libbitcoin_wallet.a libbitcoin_common.a libbitcoin_util.a libbitcoin_consensus.a crypto/.libs/libbitcoin_crypto_base.a crypto/.libs/libbitcoin_crypto_sse41.a crypto/.libs/libbitcoin_crypto_avx2.a crypto/.libs/libbitcoin_crypto_x86_shani.a libbitcoin_cli.a ./.libs/libunivalue.a leveldb/.libs/libleveldb.a crc32c/.libs/libcrc32c.a crc32c/.libs/libcrc32c_sse42.a leveldb/.libs/libmemenv.a secp256k1/.libs/libsecp256k1.a minisketch/libminisketch.a minisketch/libminisketch_clmul.a -levent -levent_pthreads -levent -lsqlite3 -lstdc++ -lm --no-as-needed -lpthread -lrt -lm -ldl -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /usr/bin/../lib/gcc/x86_64-linux-gnu/11/crtendS.o /lib/x86_64-linux-gnu/crtn.o
 31.	Running pass 'Function Pass Manager' on module 'ld-temp.o'.
 42.	Running pass 'Live DEBUG_VALUE analysis' on function '@_Z22minisketch_fuzz_target4SpanIKhE'
 5 [#0](/bitcoin-bitcoin/0/) 0x00007fa9acecad01 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xe3fd01)
 6 [#1](/bitcoin-bitcoin/1/) 0x00007fa9acec8a3e llvm::sys::RunSignalHandlers() (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xe3da3e)
 7 [#2](/bitcoin-bitcoin/2/) 0x00007fa9acecb236 (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xe40236)
 8 [#3](/bitcoin-bitcoin/3/) 0x00007fa9abb72520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 9 [#4](/bitcoin-bitcoin/4/) 0x00007fa9ad4e75fc (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x145c5fc)
10 [#5](/bitcoin-bitcoin/5/) 0x00007fa9ad4dcc32 (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x1451c32)
11 [#6](/bitcoin-bitcoin/6/) 0x00007fa9ad4dca33 LiveDebugValues::InstrRefBasedLDV::depthFirstVLocAndEmit(unsigned int, llvm::DenseMap<llvm::LexicalScope const*, llvm::DILocation const*, llvm::DenseMapInfo<llvm::LexicalScope const*, void>, llvm::detail::DenseMapPair<llvm::LexicalScope const*, llvm::DILocation const*> > const&, llvm::DenseMap<llvm::LexicalScope const*, llvm::SmallSet<llvm::DebugVariable, 4u, std::less<llvm::DebugVariable> >, llvm::DenseMapInfo<llvm::LexicalScope const*, void>, llvm::detail::DenseMapPair<llvm::LexicalScope const*, llvm::SmallSet<llvm::DebugVariable, 4u, std::less<llvm::DebugVariable> > > > const&, llvm::DenseMap<llvm::LexicalScope const*, llvm::SmallPtrSet<llvm::MachineBasicBlock*, 4u>, llvm::DenseMapInfo<llvm::LexicalScope const*, void>, llvm::detail::DenseMapPair<llvm::LexicalScope const*, llvm::SmallPtrSet<llvm::MachineBasicBlock*, 4u> > >&, llvm::SmallVector<llvm::SmallVector<std::pair<llvm::DebugVariable, LiveDebugValues::DbgValue>, 8u>, 8u>&, LiveDebugValues::ValueIDNum**, LiveDebugValues::ValueIDNum**, llvm::SmallVectorImpl<LiveDebugValues::VLocTracker>&, llvm::MachineFunction&, llvm::DenseMap<llvm::DebugVariable, unsigned int, llvm::DenseMapInfo<llvm::DebugVariable, void>, llvm::detail::DenseMapPair<llvm::DebugVariable, unsigned int> >&, llvm::TargetPassConfig const&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x1451a33)
12 [#7](/bitcoin-bitcoin/7/) 0x00007fa9ad4df216 LiveDebugValues::InstrRefBasedLDV::ExtendRanges(llvm::MachineFunction&, llvm::MachineDominatorTree*, llvm::TargetPassConfig*, unsigned int, unsigned int) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x1454216)
13 [#8](/bitcoin-bitcoin/8/) 0x00007fa9ad2512be llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x11c62be)
14 [#9](/bitcoin-bitcoin/9/) 0x00007fa9ad0053c0 llvm::FPPassManager::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xf7a3c0)
15[#10](/bitcoin-bitcoin/10/) 0x00007fa9ad00c9b3 llvm::FPPassManager::runOnModule(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xf819b3)
16[#11](/bitcoin-bitcoin/11/) 0x00007fa9ad005f66 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xf7af66)
17[#12](/bitcoin-bitcoin/12/) 0x00007fa9ae3f0aff (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x2365aff)
18[#13](/bitcoin-bitcoin/13/) 0x00007fa9ae3efcdc llvm::lto::backend(llvm::lto::Config const&, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream> > > (unsigned int)>, unsigned int, llvm::Module&, llvm::ModuleSummaryIndex&) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x2364cdc)
19[#14](/bitcoin-bitcoin/14/) 0x00007fa9ae3e4889 llvm::lto::LTO::runRegularLTO(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream> > > (unsigned int)>) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x2359889)
20[#15](/bitcoin-bitcoin/15/) 0x00007fa9ae3e4083 llvm::lto::LTO::run(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream> > > (unsigned int)>, std::function<llvm::Expected<std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream> > > (unsigned int)> > (unsigned int, llvm::StringRef)>) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0x2359083)
21[#16](/bitcoin-bitcoin/16/) 0x00000000005ca433 lld::elf::BitcodeCompiler::compile() (/usr/bin/ld.lld+0x5ca433)
22[#17](/bitcoin-bitcoin/17/) 0x0000000000550206 void lld::elf::LinkerDriver::compileBitcodeFiles<llvm::object::ELFType<(llvm::support::endianness)1, true> >(bool) (/usr/bin/ld.lld+0x550206)
23[#18](/bitcoin-bitcoin/18/) 0x000000000054d05d lld::elf::LinkerDriver::link(llvm::opt::InputArgList&) (/usr/bin/ld.lld+0x54d05d)
24[#19](/bitcoin-bitcoin/19/) 0x0000000000541ac4 lld::elf::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) (/usr/bin/ld.lld+0x541ac4)
25[#20](/bitcoin-bitcoin/20/) 0x000000000053fd97 lld::elf::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) (/usr/bin/ld.lld+0x53fd97)
26[#21](/bitcoin-bitcoin/21/) 0x000000000048b235 (/usr/bin/ld.lld+0x48b235)
27[#22](/bitcoin-bitcoin/22/) 0x000000000048a998 main (/usr/bin/ld.lld+0x48a998)
28[#23](/bitcoin-bitcoin/23/) 0x00007fa9abb59d90 (/lib/x86_64-linux-gnu/libc.so.6+0x29d90)
29[#24](/bitcoin-bitcoin/24/) 0x00007fa9abb59e40 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e40)
30[#25](/bitcoin-bitcoin/25/) 0x000000000048a4d5 _start (/usr/bin/ld.lld+0x48a4d5)
31clang: error: unable to execute command: Segmentation fault (core dumped)
  7. fanquake commented at 8:12 am on August 31, 2022: member
    I think this might be a resource, or other local issue, as I can’t reproduce the failure using the steps outlined in the op. One suggestion I’d make would be to use ./configure CC=clang CXX=clang++ --enable-fuzz --with-sanitizers=fuzzer --enable-lto, to save the manual -flto additions, and to ensure you end up with -flto in your ldflags (although it is present). Can you provide any other more verbose output from a failure?
  8. MarcoFalke commented at 8:13 am on August 31, 2022: member

    I think this might be a resource, or other local issue

    I was using 16 GB of RAM. Do you have more than that?

  9. fanquake commented at 8:19 am on August 31, 2022: member

    I was using 16 GB or RAM. Do you have more than that?

    The machine I’m using has 32GB, but docker should be limited to only using 15GB. Will test further.

  10. MarcoFalke commented at 8:29 am on August 31, 2022: member

    Still happening with 32GB of RAM and 128GB of swap. Let me try a different CPU …

     0  CXXLD    test/fuzz/fuzz
 1clang: error: unable to execute command: Segmentation fault (core dumped)
 2clang: error: linker command failed due to signal (use -v to see invocation)
 3make[2]: *** [Makefile:7337: test/fuzz/fuzz] Error 254
 4make[2]: Leaving directory '/root/bitcoin-core/src'
 5make[1]: *** [Makefile:18926: all-recursive] Error 1
 6make[1]: Leaving directory '/root/bitcoin-core/src'
 7make: *** [Makefile:824: all-recursive] Error 1
 8# lscpu | grep 'AMD ' ; free -h ; df -h ./
 9Model name:                      AMD EPYC Processor
10               total        used        free      shared  buff/cache   available
11Mem:            30Gi       351Mi        26Gi       4.0Mi       3.7Gi        29Gi
12Swap:          127Gi          0B       127Gi
13Filesystem      Size  Used Avail Use% Mounted on
14/dev/sda1       338G  133G  191G  42% /
  11. fanquake commented at 9:08 am on August 31, 2022: member
    Have recreated the issue on x86_64 linux hardware, using LLVM 14, but still not on aarch64.
  12. MarcoFalke renamed this:
    Segmentation fault when compiling with libfuzzer and lto
    Segmentation fault when compiling with libfuzzer and lto (x86_64)
    on Aug 31, 2022
  13. MarcoFalke commented at 9:13 am on August 31, 2022: member
    Yes, this works on aarch64 for me as well.
  14. MarcoFalke commented at 10:05 am on August 31, 2022: member

    Trying a depends build seems to work fine as well:

    export CC=clang && export CXX=clang++ && export DEBIAN_FRONTEND=noninteractive && apt update && apt install curl wget htop git vim ccache -y && git clone https://github.com/bitcoin/bitcoin.git --depth=1 ./bitcoin-core && cd bitcoin-core && apt install clang llvm build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq make automake cmake curl g++-multilib libtool binutils-gold bsdmainutils pkg-config python3 patch bison lld mold -y && ( cd depends && make NO_QT=1 LTO=1 NO_WALLET=1 NO_ZMQ=1 NO_UPNP=1 NO_NATPMP=1 -j $(nproc) ) && ./autogen.sh && CONFIG_SITE="$PWD/depends/x86_64-pc-linux-gnu/share/config.site" ./configure --enable-fuzz --with-sanitizers=fuzzer && make -j $(nproc)

    Edit: The same command with --enable-debug added, again results in a segmentation fault

    Edit2: The same command with DEBUG=1 added to depends, results in:

    0  CXXLD    test/fuzz/fuzz
1/usr/bin/ld: error: LLVM gold plugin: <unknown>:0: Undefined temporary symbol .Ltmp298032
2
3clang: error: linker command failed with exit code 1 (use -v to see invocation)
  15. MarcoFalke commented at 2:14 pm on August 31, 2022: member

    clang-13 seems to be working man_shrugging

    clang-16 (16.0.0-++20220831042109+45c1ce321dce-1exp120220831042209.392) as well

  16. MarcoFalke commented at 3:19 pm on September 1, 2022: member
    Not sure what to do here. This is almost certainly a compiler bug, given that the behaviour is reproducible among some versions of clang, but not others.
  17. MarcoFalke closed this on Sep 1, 2022
  18. MarcoFalke added the label Upstream on Sep 1, 2022
  19. bitcoin locked this on Sep 1, 2023


MarcoFalke fanquake

Labels
Bug Build system Upstream

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me