This PR adds a fuzz target for ScriptPubKeyMan migration. It creates a LegacyDataSPKM which can have some keys/scripts/etc, and then migrate it to descriptor. I tried to keep it as compatible as possible with future legacy wallet removal.
fuzz: wallet: add target for spkm migration #29694
pull brunoerg wants to merge 4 commits into bitcoin:master from brunoerg:2024-03-fuzz-spkm-migration changing 7 files +174 −23-
brunoerg commented at 5:35 PM on March 21, 2024: contributor
-
DrahtBot commented at 5:35 PM on March 21, 2024: contributor
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--006a51241073e994b41acfe9ec718e94-->
Code Coverage & Benchmarks
For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/29694.
<!--021abf342d371248e50ceaed478a90ca-->
Reviews
See the guideline for information on the review process. A summary of reviews will appear here.
<!--174a7506f384e20aa4161008e828411d-->
Conflicts
Reviewers, this pull request conflicts with the following ones:
- #28710 (Remove the legacy wallet and BDB dependency by achow101)
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
<!--5faf32d7da4f0f540f40219e4f7537a3-->
- DrahtBot added the label Tests on Mar 21, 2024
- brunoerg force-pushed on Apr 19, 2024
-
in src/wallet/test/fuzz/scriptpubkeyman.cpp:228 in d0c3ca9269 outdated
223 | + keys.push_back(key); 224 | + } 225 | + 226 | + if (keys.empty()) return; 227 | + 228 | + auto& legacy_spkm{*wallet.GetOrCreateLegacyScriptPubKeyMan()};
furszy commented at 10:35 PM on June 16, 2024:The legacy spkm class will be deleted soon in #28710. You might need to add a mechanism to mock the bdb reader class just so you can feed the migration process with a hand-crafted
LegacyDataSPKM. This will let you avoid crafting the bdb file manually when bdb is not compiled anymore.
brunoerg commented at 8:36 PM on June 25, 2024:You might need to add a mechanism to mock the bdb reader class just so you can feed the migration process with a hand-crafted LegacyDataSPKM. This will let you avoid crafting the bdb file manually when bdb is not compiled anymore.
You're right, I'll move this PR to draft to work on it.
brunoerg commented at 11:33 AM on September 10, 2024:Just changed it to use
LegacyDataSPKM. Note that this target is about the spkm migration only, not the whole process.Mazzika1 approvedbrunoerg marked this as a draft on Jun 25, 2024brunoerg force-pushed on Sep 9, 2024DrahtBot added the label CI failed on Sep 10, 2024brunoerg marked this as ready for review on Sep 10, 2024DrahtBot removed the label CI failed on Sep 11, 2024achow101 requested review from achow101 on Oct 15, 2024brunoerg commented at 9:32 AM on October 23, 2024: contributorInputs for this are available at: https://github.com/brunoerg/qa-assets/pull/2
maflcko commented at 5:28 PM on October 23, 2024: memberin src/wallet/test/fuzz/scriptpubkeyman.cpp:248 in 2e7ac47bbc outdated
243 | + if (legacy_data.HaveKey(keys[key_index].GetPubKey().GetID())) return; 244 | + if (legacy_data.LoadKey(keys[key_index], keys[key_index].GetPubKey())) { 245 | + load_key_count++; 246 | + if (fuzzed_data_provider.ConsumeBool()) { 247 | + CHDChain hd_chain; 248 | + hd_chain.nVersion = CHDChain::VERSION_HD_CHAIN_SPLIT;
achow101 commented at 9:35 PM on November 4, 2024:I think it would also be good to cover hd chain before hd chain split.
brunoerg commented at 2:43 PM on November 5, 2024:Will do it.
in src/wallet/test/fuzz/scriptpubkeyman.cpp:289 in 2e7ac47bbc outdated
284 | + } 285 | + 286 | + std::optional<MigrationData> res{legacy_data.MigrateToDescriptor()}; 287 | + if (res.has_value()) { 288 | + assert(static_cast<int>(res->desc_spkms.size()) >= load_key_count); 289 | + if (!res->solvable_descs.empty()) assert(script_count > 0);
achow101 commented at 9:36 PM on November 4, 2024:Would it be possible for
script_countto really be watchonly script count and just have it increment only when a watchonly script is added?
brunoerg commented at 4:40 PM on November 5, 2024:I think so, it just needs to check if the script was built with a loaded key.
brunoerg commented at 2:09 PM on November 6, 2024:In fact, I tried to track watchonly and spendable scripts, but I don't think it's simple since we load keys, hd chain, diverse scripts, etc. I ended up writing similar or copying a lot of things fromwallet/scriptpubkey.cppwhich I don't think would be a valuable approach here.nevermind! will address it next push.
brunoerg commented at 8:47 PM on November 26, 2024:Done.
in src/wallet/test/fuzz/scriptpubkeyman.cpp:288 in 2e7ac47bbc outdated
283 | + ); 284 | + } 285 | + 286 | + std::optional<MigrationData> res{legacy_data.MigrateToDescriptor()}; 287 | + if (res.has_value()) { 288 | + assert(static_cast<int>(res->desc_spkms.size()) >= load_key_count);
achow101 commented at 9:37 PM on November 4, 2024:If watchonly script count can be tracked, then this check should be able to be an exact check as
desc_spkmsshould beload_key_countplus the number of spendable scripts.
brunoerg commented at 8:47 PM on November 26, 2024:Done.
in src/wallet/test/fuzz/scriptpubkeyman.cpp:265 in 2e7ac47bbc outdated
263 | + }, 264 | + [&] { 265 | + CScript script{ConsumeScript(fuzzed_data_provider)}; 266 | + const auto key_index{fuzzed_data_provider.ConsumeIntegralInRange<int>(0, keys.size() - 1)}; 267 | + LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 2) { 268 | + CallOneOf(
achow101 commented at 9:38 PM on November 4, 2024:It would be nice to have multisig scripts here as the handling of ismine for that is quite complicated and having fuzz testing of it would be great.
brunoerg commented at 4:17 PM on November 5, 2024:Sure, I will do it.
brunoerg force-pushed on Nov 11, 2024brunoerg commented at 4:57 PM on November 11, 2024: contributorForce-pushed addressing (multisig and hd chain cover):
brunoerg force-pushed on Nov 26, 2024brunoerg force-pushed on Nov 26, 2024DrahtBot added the label CI failed on Nov 26, 2024DrahtBot commented at 7:49 PM on November 26, 2024: contributor<!--85328a0da195eb286784d51f73fa0af9-->
🚧 At least one of the CI tasks failed. <sub>Debug: https://github.com/bitcoin/bitcoin/runs/33561912794</sub>
<details><summary>Hints</summary>
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
</details>
DrahtBot removed the label CI failed on Nov 26, 2024brunoerg commented at 8:48 PM on November 26, 2024: contributorForce-pushed addressing #29694 (review)
brunoerg commented at 8:48 PM on November 26, 2024: contributorbrunoerg marked this as a draft on Nov 27, 2024brunoerg force-pushed on Dec 2, 2024brunoerg force-pushed on Dec 2, 2024DrahtBot added the label CI failed on Dec 2, 2024DrahtBot commented at 12:27 PM on December 2, 2024: contributor<!--85328a0da195eb286784d51f73fa0af9-->
🚧 At least one of the CI tasks failed. <sub>Debug: https://github.com/bitcoin/bitcoin/runs/33784139981</sub>
<details><summary>Hints</summary>
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
</details>
brunoerg force-pushed on Dec 2, 2024brunoerg marked this as ready for review on Dec 2, 2024brunoerg commented at 3:42 PM on December 2, 2024: contributorForce-pushed changing the fuzz target to address the
ApplyMigrationDatafunction. Now, after migrating the spk to descriptor, it creates some transactions and apply the migrated data.Note that the CI failure is expected (see #31374):
[09:11:48.112] Run spkm_migration with args ['/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz', '-max_total_time=60']INFO: Running with entropic power schedule (0xFF, 100). [09:11:48.112] INFO: Seed: 2346481497 [09:11:48.112] INFO: Loaded 1 modules (629856 inline 8-bit counters): 629856 [0x55a29859f118, 0x55a298638d78), [09:11:48.112] INFO: Loaded 1 PC tables (629856 PCs): 629856 [0x55a298638d78,0x55a298fd5378), [09:11:48.112] INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes [09:11:48.112] wallet/wallet.cpp:4088 ApplyMigrationData: Assertion `!m_cached_spks.empty()' failed.in src/wallet/test/fuzz/scriptpubkeyman.cpp:331 in 514aff2f4b outdated
326 | + 327 | + WalletBatch batch{wallet.GetDatabase()}; 328 | + LOCK(wallet.cs_wallet); 329 | + CBlockLocator loc{chainstate.m_chain.GetLocator()}; 330 | + assert(batch.WriteBestBlock(loc)); 331 | + (void)wallet.ApplyMigrationData(batch, *res);
furszy commented at 6:55 PM on December 5, 2024:Cannot call
ApplyMigrationDatawithout executing the entireDoMigrationstep. That's just incorrect. The code skips the creation of the watch-only and solvable wallets from theMigrationDataresult, which causes a crash insideApplyMigrationDatabecause it is trying to access the watch-only wallet.
brunoerg commented at 9:23 PM on December 5, 2024:Yes, just realized it. I'm going to change it, but afaik
DoMigrationcan't be called directly here.
brunoerg commented at 8:54 PM on December 9, 2024:Done
brunoerg force-pushed on Dec 9, 2024brunoerg commented at 8:56 PM on December 9, 2024: contributorForce-pushed:
Now the fuzz target runs the entire migration process (
MigrateLegacyToDescriptor). Also, I added a new commit that allows to migrate in-memory wallets to speed up the target and avoid timeouts.brunoerg force-pushed on Dec 9, 2024brunoerg marked this as a draft on Dec 9, 2024brunoerg force-pushed on Dec 10, 2024brunoerg force-pushed on Dec 10, 2024DrahtBot removed the label CI failed on Dec 10, 2024brunoerg marked this as ready for review on Dec 10, 2024brunoerg force-pushed on Dec 11, 2024DrahtBot added the label CI failed on Dec 11, 2024brunoerg force-pushed on Dec 11, 2024brunoerg force-pushed on Dec 11, 2024DrahtBot removed the label CI failed on Dec 11, 2024brunoerg force-pushed on Jan 23, 2025brunoerg commented at 11:39 AM on January 23, 2025: contributorForce-pushed adding
SeedRandomStateForTestandSetMockTimefor better stability.DrahtBot added the label Needs rebase on Jan 25, 2025brunoerg force-pushed on Jan 27, 2025brunoerg commented at 7:02 PM on January 27, 2025: contributorRebased
DrahtBot removed the label Needs rebase on Jan 27, 2025DrahtBot added the label CI failed on Jan 27, 2025brunoerg force-pushed on Jan 27, 2025brunoerg force-pushed on Jan 27, 2025DrahtBot commented at 4:12 PM on February 3, 2025: contributorhttps://cirrus-ci.com/task/5046981191532544?logs=ci#L1799
[22:03:02.859] test 2025-01-27T22:03:01.984000Z TestFramework (ERROR): Assertion failed [22:03:02.859] Traceback (most recent call last): [22:03:02.859] File "/ci_container_base/test/functional/test_framework/test_framework.py", line 135, in main [22:03:02.859] self.run_test() [22:03:02.859] File "/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/test/functional/wallet_migration.py", line 1098, in run_test [22:03:02.859] self.test_failed_migration_cleanup() [22:03:02.859] File "/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/test/functional/wallet_migration.py", line 899, in test_failed_migration_cleanup [22:03:02.859] assert all(wallet not in self.master_node.listwallets() for wallet in ["failed", "failed_watchonly", "failed_solvables"]) [22:03:02.859] AssertionErrorbrunoerg marked this as a draft on Feb 3, 2025brunoerg force-pushed on Feb 5, 2025DrahtBot removed the label CI failed on Feb 5, 2025brunoerg marked this as ready for review on Feb 5, 2025brunoerg commented at 4:58 PM on February 5, 2025: contributorJust fixed CI, ready for review!
DrahtBot added the label Needs rebase on Apr 22, 2025in src/wallet/wallet.cpp:4580 in 0c0e79d8a4 outdated
4551 | @@ -4551,11 +4552,14 @@ util::Result<MigrationResult> MigrateLegacyToDescriptor(std::shared_ptr<CWallet> 4552 | success = reload_wallet(res.solvables_wallet); 4553 | } 4554 | } 4555 | - if (!success) { 4556 | + if (!success && !in_memory) {
pablomartin4btc commented at 8:22 AM on April 25, 2025:I think if you add the condition there, there are other stuff that's missing and I think they need to be done which are:
- the for after
// Unload the wallets; - the lines after
// Verify that there is no dangling wallet
brunoerg commented at 3:40 PM on May 2, 2025:Why? the for after
// Unload the walletsare basically playing withcreated_walletswhich is used only if the db is not in memory. Same for the lines after// Verify that there is no dangling walletwhich needsptr_wallet.
pablomartin4btc commented at 9:53 PM on May 5, 2025:the for after
// Unload the wallets...There's the removal of the wallet from the context (
if (!RemoveWallet(context, w, /*load_on_start=*/false)) {) usingcreated_wallets- that was done when the wallet was loaded (inLoadWalletInternal->AddWallet(context, wallet);).the lines after
// Verify that there is no dangling walletwhich needsptr_wallet.At the end of the condition is returning an error (
return util::Error{error};)Be aware that if you remove the commit wallet: skip backup restoration when using in-memory dbs altogether you need to add a conditional to avoid the restore section.
At some point I think we need to refactor the wallet and extract the wallet migration behaviour into a new/ few object/ s.
pablomartin4btc commented at 8:44 AM on April 25, 2025: memberYou are skipping the backup restore in commit: "wallet: skip backup restoration when using in-memory dbs"; shouldn't the backup itself be skipped too when
in_memory?wallet: add option to create in-memory SQLite DB in the migration ba8d151fe072a424e60awallet: sqlite: there is no need to have exclusive locking when mocking
For in-memory SQLite databases, exclusive locking is generally not necessary because in-memory dbs are private to the connection that created them.
6f0e025efawallet: skip backup restoration when using in-memory dbs
By using in-memory databases, we do not have dbs file to delete and restore.
fuzz: wallet: add target for spkm migration c2bbe1221cbrunoerg force-pushed on Apr 28, 2025DrahtBot removed the label Needs rebase on Apr 28, 2025DrahtBot commented at 2:33 PM on May 7, 2025: contributor<!--cf906140f33d8803c4a75a2196329ecb-->
🐙 This pull request conflicts with the target branch and needs rebase.
DrahtBot added the label Needs rebase on May 7, 2025brunoerg commented at 8:30 PM on May 8, 2025: contributorI'm reworking this target, will draft it.
brunoerg marked this as a draft on May 8, 2025brunoerg closed this on May 27, 2025
github-metadata-mirror
This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-21 18:13 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me