wallet: remove BDB dependency from wallet migration benchmark #31241

1. 
   furszy commented at 4:42 pm on November 7, 2024: member

   Part of the legacy wallet removal working path #20160.

   Stops creating a bdb database in the wallet migration benchmark. Instead, the benchmark now creates the db in memory and re-uses it for the migration process.

2. 
   DrahtBot commented at 4:42 pm on November 7, 2024: contributor

   The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

   Code Coverage & Benchmarks

   For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31241.

   Reviews

   See the guideline for information on the review process. A summary of reviews will appear here.

   Conflicts

   Reviewers, this pull request conflicts with the following ones:

   • #31250 (wallet: Disable creating and loading legacy wallets by achow101)
   • #25722 (refactor: Use util::Result class for wallet loading by ryanofsky)

   If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

3. DrahtBot added the label Wallet on Nov 7, 2024
4. 
   fanquake commented at 5:21 pm on November 7, 2024: member

   https://cirrus-ci.com/task/4957802181951488?logs=ci#L1777:

    0[17:17:27.480]   Thread T4 'b-httpworker.1' (tid=27041, running) created by main thread at:
    1[17:17:27.480]     [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0xf7df5) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
    2[17:17:27.480]     [#1](/bitcoin-bitcoin/1/) std::__1::__libcpp_thread_create[abi:ne190103](unsigned long*, void* (*)(void*), void*) /usr/lib/llvm-19/bin/../include/c++/v1/__thread/support/pthread.h:182:10 (bitcoind+0x6e06fb) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
    3[17:17:27.480]     [#2](/bitcoin-bitcoin/2/) std::__1::thread::thread<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&, 0>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/__thread/thread.h:211:14 (bitcoind+0x6e06fb)
    4[17:17:27.480]     [#3](/bitcoin-bitcoin/3/) std::__1::thread* std::__1::construct_at[abi:ne190103]<std::__1::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&, std::__1::thread*>(std::__1::thread*, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/__memory/construct_at.h:41:46 (bitcoind+0x6e033a) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
    5[17:17:27.480]     [#4](/bitcoin-bitcoin/4/) std::__1::thread* std::__1::__construct_at[abi:ne190103]<std::__1::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&, std::__1::thread*>(std::__1::thread*, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/__memory/construct_at.h:49:10 (bitcoind+0x6e033a)
    6[17:17:27.480]     [#5](/bitcoin-bitcoin/5/) void std::__1::allocator_traits<std::__1::allocator<std::__1::thread>>::construct[abi:ne190103]<std::__1::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&, void, 0>(std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/__memory/allocator_traits.h:328:5 (bitcoind+0x6e033a)
    7[17:17:27.480]     [#6](/bitcoin-bitcoin/6/) std::__1::thread* std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread>>::__emplace_back_slow_path<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/vector:1538:3 (bitcoind+0x6e033a)
    8[17:17:27.480]     [#7](/bitcoin-bitcoin/7/) std::__1::thread& std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread>>::emplace_back<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/vector:1558:13 (bitcoind+0x6d6818) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
    9[17:17:27.480]     [#8](/bitcoin-bitcoin/8/) StartHTTPServer() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./httpserver.cpp:504:31 (bitcoind+0x6d6818)
   10[17:17:27.480]     [#9](/bitcoin-bitcoin/9/) AppInitServers(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:707:5 (bitcoind+0x1a5d89) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
   11[17:17:27.480]     [#10](/bitcoin-bitcoin/10/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1368:14 (bitcoind+0x19c2da) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
   12[17:17:27.480]     [#11](/bitcoin-bitcoin/11/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:231:43 (bitcoind+0x17e72f) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
   13[17:17:27.480]     [#12](/bitcoin-bitcoin/12/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:277:10 (bitcoind+0x17e72f)
   14[17:17:27.480] 
   15[17:17:27.480] SUMMARY: ThreadSanitizer: heap-use-after-free /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/wallet/./util/fs.h:77:57 in fs::u8path(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
   16[17:17:27.480] ================== 
   

5. DrahtBot added the label CI failed on Nov 7, 2024
6. wallet: remove BDB dependency from wallet migration benchmark

   Stops creating a bdb database in the wallet migration benchmark.
   Instead, the benchmark now creates the db in memory and re-uses
   it for the migration process.

   f7dbb300d7
7. furszy force-pushed on Nov 7, 2024
8. 
   furszy commented at 6:22 pm on November 7, 2024: member

   https://cirrus-ci.com/task/4957802181951488?logs=ci#L1777:

    0[17:17:27.480]   Thread T4 'b-httpworker.1' (tid=27041, running) created by main thread at:
    1[17:17:27.480]     [#0](/bitcoin-bitcoin/0/) pthread_create <null> (bitcoind+0xf7df5) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
    2[17:17:27.480]     [#1](/bitcoin-bitcoin/1/) std::__1::__libcpp_thread_create[abi:ne190103](unsigned long*, void* (*)(void*), void*) /usr/lib/llvm-19/bin/../include/c++/v1/__thread/support/pthread.h:182:10 (bitcoind+0x6e06fb) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
    3[17:17:27.480]     [#2](/bitcoin-bitcoin/2/) std::__1::thread::thread<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&, 0>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/__thread/thread.h:211:14 (bitcoind+0x6e06fb)
    4[17:17:27.480]     [#3](/bitcoin-bitcoin/3/) std::__1::thread* std::__1::construct_at[abi:ne190103]<std::__1::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&, std::__1::thread*>(std::__1::thread*, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/__memory/construct_at.h:41:46 (bitcoind+0x6e033a) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
    5[17:17:27.480]     [#4](/bitcoin-bitcoin/4/) std::__1::thread* std::__1::__construct_at[abi:ne190103]<std::__1::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&, std::__1::thread*>(std::__1::thread*, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/__memory/construct_at.h:49:10 (bitcoind+0x6e033a)
    6[17:17:27.480]     [#5](/bitcoin-bitcoin/5/) void std::__1::allocator_traits<std::__1::allocator<std::__1::thread>>::construct[abi:ne190103]<std::__1::thread, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&, void, 0>(std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/__memory/allocator_traits.h:328:5 (bitcoind+0x6e033a)
    7[17:17:27.480]     [#6](/bitcoin-bitcoin/6/) std::__1::thread* std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread>>::__emplace_back_slow_path<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/vector:1538:3 (bitcoind+0x6e033a)
    8[17:17:27.480]     [#7](/bitcoin-bitcoin/7/) std::__1::thread& std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread>>::emplace_back<void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int&>(void (&)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*&&, int&) /usr/lib/llvm-19/bin/../include/c++/v1/vector:1558:13 (bitcoind+0x6d6818) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
    9[17:17:27.480]     [#8](/bitcoin-bitcoin/8/) StartHTTPServer() /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./httpserver.cpp:504:31 (bitcoind+0x6d6818)
   10[17:17:27.480]     [#9](/bitcoin-bitcoin/9/) AppInitServers(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:707:5 (bitcoind+0x1a5d89) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
   11[17:17:27.480]     [#10](/bitcoin-bitcoin/10/) AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./init.cpp:1368:14 (bitcoind+0x19c2da) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
   12[17:17:27.480]     [#11](/bitcoin-bitcoin/11/) AppInit(node::NodeContext&) /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:231:43 (bitcoind+0x17e72f) (BuildId: 759dffa438a525872edb969c48b6c5c4c6341482)
   13[17:17:27.480]     [#12](/bitcoin-bitcoin/12/) main /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/./bitcoind.cpp:277:10 (bitcoind+0x17e72f)
   14[17:17:27.480] 
   15[17:17:27.480] SUMMARY: ThreadSanitizer: heap-use-after-free /ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/wallet/./util/fs.h:77:57 in fs::u8path(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
   16[17:17:27.480] ================== 
   

   fixed

9. DrahtBot removed the label CI failed on Nov 7, 2024
10. 
    achow101 commented at 2:26 am on November 13, 2024: member

    While this does remove the BDB requirement, it doesn’t remove the reliance on legacy wallet specific code that will be removed. Would it be possible to change this to create a wallet with a LegacyDataSPKM without utilizing any of the legacy wallet functions?

Contributors
furszy DrahtBot fanquake achow101

Labels
Wallet