Multipath descriptors requires performing a deep copy, so a Clone function that does that is added to miniscript::Node instead of the current shallow copy.
Fixes #30864
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--006a51241073e994b41acfe9ec718e94-->
For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/30866.
<!--021abf342d371248e50ceaed478a90ca-->
See the guideline for information on the review process.
| Type | Reviewers |
|---|---|
| ACK | darosior, hodlinator, brunoerg |
| Stale ACK | sipa, davidgumberg |
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.
<!--174a7506f384e20aa4161008e828411d-->
No conflicts as of last run.
<!--85328a0da195eb286784d51f73fa0af9-->
🚧 At least one of the CI tasks failed. <sub>Debug: https://github.com/bitcoin/bitcoin/runs/29955732210</sub>
<details><summary>Hints</summary>
Make sure to run all tests locally, according to the documentation.
The failure may happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
</details>
https://github.com/bitcoin/bitcoin/pull/30866/checks?check_run_id=29955732214:
In file included from /ci_container_base/src/script/descriptor.cpp:10:
/ci_container_base/src/script/miniscript.h: In instantiation of ‘miniscript::Node<Key> miniscript::Node<Key>::Clone() const [with Key = unsigned int]’:
/ci_container_base/src/script/descriptor.cpp:1363:124: required from here
/ci_container_base/src/script/miniscript.h:535:33: error: moving a local object in a return statement prevents copy elision [-Werror=pessimizing-move]
535 | return std::move(ret);
| ^
/ci_container_base/src/script/miniscript.h:535:33: note: remove ‘std::move’ call
cc1plus: all warnings being treated as errors
1359 | @@ -1360,7 +1360,7 @@ class MiniscriptDescriptor final : public DescriptorImpl 1360 | for (const auto& arg : m_pubkey_args) { 1361 | providers.push_back(arg->Clone()); 1362 | } 1363 | - return std::make_unique<MiniscriptDescriptor>(std::move(providers), miniscript::MakeNodeRef<uint32_t>(*m_node)); 1364 | + return std::make_unique<MiniscriptDescriptor>(std::move(providers), miniscript::MakeNodeRef<uint32_t>(m_node->Clone()));
The PR description asserts:
Multipath descriptors requires performing a deep copy
Would be happy if you cared to add an elaboration on why that is.
It seemed to me like it should be safe to just have another shared_ptr point to the same const Node. Unless something on the outside could own a non-const reference into the node hierarchy and mutate it that way? In that case maybe it would be more robust for the MiniscriptDescriptor-ctor to be the one ensuring it holds a unique reference by doing the node->Clone() there instead (if .use_count() > 1).
Was able to avoid the crash in #30864 using only this on top of the parent commit of this PR:
return std::make_unique<MiniscriptDescriptor>(std::move(providers), m_node);
Probably causes other bugs that are obvious for those who understand more of the context. Change passes both unit and non-extended functional tests though.
It's not strictly necessary but I decided to do it this way to follow the pattern used for all other descriptors. The multipath expanded descriptors are treated as separate descriptors everywhere, and doing a deep copy retains that behavior which will allow for future changes that may modify specific descriptors.
How come the shallow copy in the version before the PR is causing a problem in this case though?
It seems the default-generated copy-ctor for Node used before was somehow ending up with corrupt/leaked data, but I've been unable to spot what it is. Can't see any slicing going on. Is something funky being done to the shared_ptrs?
Regardless, it might be worth adding:
Node(const Node&) = delete;
Node(Node&&) = delete;
Node& operator=(const Node&) = delete;
Node& operator=(Node&&) = delete;
Edit: Later versions of my change make the move-operators = default.
Not entirely sure either.
The shallow copy in the version before this PR is causing a problem because we mess with a Node's subs in the destructor to avoid a recursion stack overflow:
https://github.com/bitcoin/bitcoin/blob/2c90f8e08c4cf44d4c1ef3dde0e7f7991b8b9390/src/script/miniscript.h#L513-L524
In the scriptpubkeyman fuzz target using the seed from #30864 we would Parse the tr(%17/<2;3>,l:pk(%08)) multipath descriptor into a parsed_descs vector of 2 Descriptor pointers. We would use the first of those descriptors (tr(xpub..../2,l:pk(4bdf....))#jmgpm0u8) and move it into a WalletDescriptor. When exiting the CreateWalletDescriptor function the parsed_descs vector would be destructed, along with the second Descriptor pointer it contains. The destructor of the Node (l:pk(xpub...)) it contains would in turn be called, moving out all its subs that are also pointed to by the first Descriptor's Node. As soon as you try to perform an operation on it it will try to access moved structures.
Concept ACK.
I wrote a regression unit test for this: https://github.com/darosior/bitcoin/commit/d23ffe08ef42475dcd40a13857cd775ac98e0270. It's failing on master but also on this PR. Curiously i had to introduce a new test case with a descriptor close to the one from the fuzzer input as the test is not failing on the existing multipath miniscript descriptor.
533 | + ret.subs.push_back(MakeNodeRef<Key>(sub)); 534 | + } 535 | + return ret; 536 | + }; 537 | + return TreeEval<Node<Key>>(upfn); 538 | + }
More smartpointy less segfaulty:
diff --git a/src/script/descriptor.cpp b/src/script/descriptor.cpp
index 6053c853bb..b099fdfc3f 100644
--- a/src/script/descriptor.cpp
+++ b/src/script/descriptor.cpp
@@ -1360,7 +1360,7 @@ public:
for (const auto& arg : m_pubkey_args) {
providers.push_back(arg->Clone());
}
- return std::make_unique<MiniscriptDescriptor>(std::move(providers), miniscript::MakeNodeRef<uint32_t>(m_node->Clone()));
+ return std::make_unique<MiniscriptDescriptor>(std::move(providers), m_node->Clone());
}
};
diff --git a/src/script/miniscript.h b/src/script/miniscript.h
index a74e312e79..1472c9ac4f 100644
--- a/src/script/miniscript.h
+++ b/src/script/miniscript.h
@@ -523,18 +523,18 @@ struct Node {
}
}
- Node<Key> Clone() const
+ NodeRef<Key> Clone() const
{
// Use TreeEval() to avoid a stack-overflow due to recursion
- auto upfn = [](const Node& node, Span<Node> subs) {
- Node<Key> ret(node);
- ret.subs.clear();
+ auto upfn = [](const Node& node, Span<NodeRef<Key>> subs) {
+ auto ret(MakeNodeRef<Key>(node));
+ ret->subs.clear();
for (const auto& sub : subs) {
- ret.subs.push_back(MakeNodeRef<Key>(sub));
+ ret->subs.push_back(sub);
}
return ret;
};
- return TreeEval<Node<Key>>(upfn);
+ return TreeEval<NodeRef<Key>>(upfn);
}
private:
Thanks, taken.
I wrote a regression unit test for this: darosior@d23ffe0.
Added the test commit as well.
643 | @@ -630,7 +644,10 @@ struct Node { 644 | // If evaluation returns std::nullopt, abort immediately. 645 | if (!result) return {}; 646 | // Replace the last node.subs.size() elements of results with the new result. 647 | - results.erase(results.end() - node.subs.size(), results.end()); 648 | + // Use pop_back to truncate results to avoid MoveAssignable requirement of erase(). 649 | + for (size_t i = 0; i < node.subs.size(); ++i) { 650 | + results.pop_back(); 651 | + }
This is not necessary anymore.
Removed
ACK 6a00ea17c136e67a66f3558dd2d02a07860b0afe
I've experimented with a change on top of this PR that removes shared_ptr usage from src/script/ altogether in order to clarify ownership.
My change is in 80fca845b5f28677207a8fea4a173baaef23036f. It helped uncover another place where the node needs to be cloned (at the bottom of the outer loop):
diff --git a/src/script/descriptor.cpp b/src/script/descriptor.cpp
index b099fdfc3f..f0294df72a 100644
--- a/src/script/descriptor.cpp
+++ b/src/script/descriptor.cpp
@@ -2154,7 +2154,7 @@ std::vector<std::unique_ptr<DescriptorImpl>> ParseScript(uint32_t& key_exp_index
for (auto& pub : parser.m_keys) {
pubs.emplace_back(std::move(pub.at(i)));
}
- ret.emplace_back(std::make_unique<MiniscriptDescriptor>(std::move(pubs), node));
+ ret.emplace_back(std::make_unique<MiniscriptDescriptor>(std::move(pubs), node->Clone()));
}
return ret;
}
It would be preferable to add that one-line fix to this PR, but I'm also interested to hear if there is any chance of my shared_ptr-removal getting any review as a separate PR. I understand Wallet needs more review of existing PRs, so I'm open to leaving it as draft for a while.
Do you have a particular test case that triggered there?
Worked through several iterations of the shared_ptr-removal over a couple of days, at some point I had:
ret.emplace_back(std::make_unique<MiniscriptDescriptor>(std::move(pubs), std::move(*node)));
'cause I wasn't paying attention to the statement being inside a loop. The second descriptor created in the loop ended up with already-moved-from values, which resulted in an this test failing:
An OR_D node we were trying to convert into a string was failing because subs was empty (already moved away from).
utACK 6a00ea17c136e67a66f3558dd2d02a07860b0afe
Good catch, added that.
Looks like you didn't.
It helped uncover another place where the node needs to be cloned
Ugh. I should have caught that. Good catch @hodlinator.
I don't think anything in the Bitcoin Core codebase actually needs the std::shared_ptr way of storing miniscript subnodes, and std::unique_ptr would suffice. The std::shared_ptrs were inherited from the miniscript codebase, where the shared_ptrs matter for the policy compiler, but I could understand that Bitcoin Core doesn't want that burden. Using std::unique_ptr instead would not leave any chance for shallow duplication.
522 | @@ -523,6 +523,20 @@ struct Node { 523 | } 524 | } 525 | 526 | + NodeRef<Key> Clone() const 527 | + { 528 | + // Use TreeEval() to avoid a stack-overflow due to recursion 529 | + auto upfn = [](const Node& node, Span<NodeRef<Key>> children) { 530 | + NodeRef<Key> ret(MakeNodeRef<Key>(node));
Is there a reasonable way to do this without using the default copy constructor? Since miniscript::Node's destructor means using the implicit copy constructor is always dangerous without modifying the subs as is done here, wouldn't it be best to delete it (if possible) as another reviewer suggested? (https://github.com/bitcoin/bitcoin/pull/30866#discussion_r1788683926)
Yes, done and deleted the copy constructor.
Looks like you didn't.
Oops, pushed it.
I don't think anything in the Bitcoin Core codebase actually needs the
std::shared_ptrway of storing miniscript subnodes, andstd::unique_ptrwould suffice. Thestd::shared_ptrs were inherited from the miniscript codebase, where the shared_ptrs matter for the policy compiler, but I could understand that Bitcoin Core doesn't want that burden. Usingstd::unique_ptrinstead would not leave any chance for shallow duplication.
Indeed, the std:;shared_ptr wasn't really needed and I've changed NodeRef to be a std::unique_ptr. This appears to also catch the case found by @hodlinator earlier.
Any way to add a test that fails/crashes without the last push?
I believe changing to std::unique_ptr and removing the copy constructor covers that case.
535 | + Assert(node.data.empty() && node.subs.empty()); 536 | + ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.keys, node.k); 537 | + } else if (!node.data.empty()) { 538 | + Assert(node.keys.empty() && node.subs.empty()); 539 | + ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.data, node.k); 540 | + } else if (!node.subs.empty()) {
just an observation: in all branches other than this one children.empty() == true since node.subs.empty() == true, and more generally children.size() == node.subs.size()
Verified with the following diff:
diff --git a/src/script/miniscript.h b/src/script/miniscript.h
@@ -527,6 +527,7 @@ struct Node {
{
// Use TreeEval() to avoid a stack-overflow due to recursion
auto upfn = [](const Node& node, Span<NodeRef<Key>> children) {
+ Assert(node.subs.size() == children.size());
NodeRef<Key> ret;
crACK https://github.com/bitcoin/bitcoin/pull/30866/commits/815467f46f47df6ff52686b0144430c14a31f4a8
This resolves #30864.
$ echo "dHIoJTE3LzwyOzM+LGw6cGsoJTA4KSk=" | base64 --decode > scriptpubkeyman.crash
$ FUZZ=scriptpubkeyman ./build/src/test/fuzz/fuzz scriptpubkeyman.crash
scriptpubkeyman: succeeded against 1 files in 0s.
The default copy constructor of miniscript::Node can't be safe because of the custom iterative destructor miniscript::~Node and deleting it is an improvement.
1686 | @@ -1658,6 +1687,10 @@ struct Node { 1687 | : Node(internal::NoDupCheck{}, ctx.MsContext(), nt, std::move(sub), val) { DuplicateKeyCheck(ctx); } 1688 | template <typename Ctx> Node(const Ctx& ctx, Fragment nt, uint32_t val = 0) 1689 | : Node(internal::NoDupCheck{}, ctx.MsContext(), nt, val) { DuplicateKeyCheck(ctx); } 1690 | + 1691 | + // Delete copy constructor and assignment operator
nit: Could be more helpful?
// Delete copy constructor and assignment operator, use Clone() instead
Done
183 | @@ -184,11 +184,11 @@ inline consteval Type operator"" _mst(const char* c, size_t l) { 184 | using Opcode = std::pair<opcodetype, std::vector<unsigned char>>; 185 | 186 | template<typename Key> struct Node; 187 | -template<typename Key> using NodeRef = std::shared_ptr<const Node<Key>>; 188 | +template<typename Key> using NodeRef = std::unique_ptr<const Node<Key>>; 189 | 190 | //! Construct a miniscript node as a shared_ptr.
Still mentions shared_ptr.
Removed
ACK 815467f46f47df6ff52686b0144430c14a31f4a8
Amazed by how frictionless it was to switch from shared_ptr -> unique_ptr. unique_ptr still is an extra level of unnecessary indirection compared to my 80fca845b5f28677207a8fea4a173baaef23036f, but the latter is a much more disruptive change. Should probably fix remaining mention of shared_ptr, see inline comment.
Thanks for deleting the copy-ctor & assignment operator as I (imprecisely) suggested, making this kind of mistake no longer compile.
Good to see you could avoid adding a new Node-constructor just to be used in Clone().
Re-verified that fuzz failure is fixed by the first commit (and that it still works by the last one).
Passed modified unit tests.
trivial reACK https://github.com/bitcoin/bitcoin/commit/352391c2cf1a45231ae92ca92d2415b3786ab9ad
Recent push only changes comments.
<details>
<summary> git diff 815467f..352391c </summary>
$ git diff 815467f..352391c
diff --git a/src/script/miniscript.h b/src/script/miniscript.h
index 04e487f884..4f5c38bb7b 100644
--- a/src/script/miniscript.h
+++ b/src/script/miniscript.h
@@ -186,7 +186,7 @@ using Opcode = std::pair<opcodetype, std::vector<unsigned char>>;
template<typename Key> struct Node;
template<typename Key> using NodeRef = std::unique_ptr<const Node<Key>>;
-//! Construct a miniscript node as a shared_ptr.
+//! Construct a miniscript node as a unique_ptr.
template<typename Key, typename... Args>
NodeRef<Key> MakeNodeRef(Args&&... args) { return std::make_unique<const Node<Key>>(std::forward<Args>(args)...); }
@@ -1688,7 +1688,7 @@ public:
template <typename Ctx> Node(const Ctx& ctx, Fragment nt, uint32_t val = 0)
: Node(internal::NoDupCheck{}, ctx.MsContext(), nt, val) { DuplicateKeyCheck(ctx); }
- // Delete copy constructor and assignment operator
+ // Delete copy constructor and assignment operator, use Clone() instead
Node(const Node&) = delete;
Node& operator=(const Node&) = delete;
re-ACK 352391c2cf1a45231ae92ca92d2415b3786ab9ad
Confirmed through range-diff to only be comment changes following my previous feedback.
code review ACK 352391c2cf1a45231ae92ca92d2415b3786ab9ad
530 | + NodeRef<Key> ret; 531 | + // As all members of Node are const, except for subs, we need to construct the cloned node with all of these members. 532 | + // However, there is no constructor that takes all three of data, keys, and subs. 533 | + // But, they are mutually exclusive, so we can use the appropriate constructor depending on what is available. 534 | + if (!node.keys.empty()) { 535 | + Assert(node.data.empty() && node.subs.empty());
It seems much simpler and robust to just add the missing constructor than matching on available data and asserting?
diff --git a/src/script/miniscript.h b/src/script/miniscript.h
index 04e487f8845..07214bdf2a4 100644
--- a/src/script/miniscript.h
+++ b/src/script/miniscript.h
@@ -527,27 +527,11 @@ struct Node {
{
// Use TreeEval() to avoid a stack-overflow due to recursion
auto upfn = [](const Node& node, Span<NodeRef<Key>> children) {
- NodeRef<Key> ret;
- // As all members of Node are const, except for subs, we need to construct the cloned node with all of these members.
- // However, there is no constructor that takes all three of data, keys, and subs.
- // But, they are mutually exclusive, so we can use the appropriate constructor depending on what is available.
- if (!node.keys.empty()) {
- Assert(node.data.empty() && node.subs.empty());
- ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.keys, node.k);
- } else if (!node.data.empty()) {
- Assert(node.keys.empty() && node.subs.empty());
- ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.data, node.k);
- } else if (!node.subs.empty()) {
- Assert(node.data.empty() && node.keys.empty());
- std::vector<NodeRef<Key>> new_subs;
- for (auto child = children.begin(); child != children.end(); ++child) {
- new_subs.emplace_back(std::move(*child));
- }
- ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, std::move(new_subs), node.k);
- } else {
- ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.k);
+ std::vector<NodeRef<Key>> new_subs;
+ for (auto child = children.begin(); child != children.end(); ++child) {
+ new_subs.emplace_back(std::move(*child));
}
- return ret;
+ return MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, std::move(new_subs), node.keys, node.data, node.k);
};
return TreeEval<NodeRef<Key>>(upfn);
}
@@ -1661,6 +1645,8 @@ public:
bool operator==(const Node<Key>& arg) const { return Compare(*this, arg) == 0; }
// Constructors with various argument combinations, which bypass the duplicate key check.
+ Node(internal::NoDupCheck, MiniscriptContext script_ctx, Fragment nt, std::vector<NodeRef<Key>> sub, std::vector<Key> key, std::vector<unsigned char> arg, uint32_t val)
+ : fragment(nt), k(val), data(std::move(arg)), subs(std::move(sub)), m_script_ctx{script_ctx}, ops(CalcOps()), ss(CalcStackSize()), ws(CalcWitnessSize()), typ(CalcType()), scriptlen(CalcScriptLen()) {}
Node(internal::NoDupCheck, MiniscriptContext script_ctx, Fragment nt, std::vector<NodeRef<Key>> sub, std::vector<unsigned char> arg, uint32_t val = 0)
: fragment(nt), k(val), data(std::move(arg)), subs(std::move(sub)), m_script_ctx{script_ctx}, ops(CalcOps()), ss(CalcStackSize()), ws(CalcWitnessSize()), typ(CalcType()), scriptlen(CalcScriptLen()) {}
Node(internal::NoDupCheck, MiniscriptContext script_ctx, Fragment nt, std::vector<unsigned char> arg, uint32_t val = 0)
That simplifies Clone() a lot, only thing I would add is make the new constructor private if we want to discourage outside uses.
Done
530 | + std::vector<NodeRef<Key>> new_subs; 531 | + for (auto child = children.begin(); child != children.end(); ++child) { 532 | + new_subs.emplace_back(std::move(*child)); 533 | + } 534 | + // std::make_unique (and therefore MakeNodeRef) doesn't work on private constructors 535 | + return std::unique_ptr<const Node<Key>>(new Node<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, std::move(new_subs), node.keys, node.data, node.k));
nit: Could make less verbose:
return std::unique_ptr<Node>{new Node{internal::NoDupCheck{}, node.m_script_ctx, node.fragment, std::move(new_subs), node.keys, node.data, node.k}};
Good call on adding the make_unique comment.
Done
re-ACK 17e59eccec2a94417fe932f8bfcecee46c85d9a3
git range-diff master 352391c 17e59ec
Clone().reACK 17e59eccec2a94417fe932f8bfcecee46c85d9a3
352391c2cf1a45231ae92ca92d2415b3786ab9ad..17e59eccec2a94417fe932f8bfcecee46c85d9a3: nice simplification!
Multipath descriptors requires performing a deep copy, so a Clone
function that does that is added to miniscript::Node instead of the
current shallow copy.
Co-Authored-By: Antoine Poinsot <darosior@protonmail.com>
There's no need for it to be a shared_ptr.
re-ACK 66d21d0eb6517e04ebfb9fad4085e788de51b4dc
re-ACK 66d21d0eb6517e04ebfb9fad4085e788de51b4dc :rocket:
reACK 66d21d0eb6517e04ebfb9fad4085e788de51b4dc