Multipath descriptors requires performing a deep copy, so a Clone function that does that is added to miniscript::Node instead of the current shallow copy.
Fixes #30864
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/30866.
See the guideline for information on the review process.
Type | Reviewers |
---|---|
ACK | davidgumberg, hodlinator, brunoerg |
Stale ACK | darosior, sipa |
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/29955732210
Make sure to run all tests locally, according to the documentation.
The failure may happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
https://github.com/bitcoin/bitcoin/pull/30866/checks?check_run_id=29955732214:
0In file included from /ci_container_base/src/script/descriptor.cpp:10:
1/ci_container_base/src/script/miniscript.h: In instantiation of ‘miniscript::Node<Key> miniscript::Node<Key>::Clone() const [with Key = unsigned int]’:
2/ci_container_base/src/script/descriptor.cpp:1363:124: required from here
3/ci_container_base/src/script/miniscript.h:535:33: error: moving a local object in a return statement prevents copy elision [-Werror=pessimizing-move]
4 535 | return std::move(ret);
5 | ^
6/ci_container_base/src/script/miniscript.h:535:33: note: remove ‘std::move’ call
7cc1plus: all warnings being treated as errors
1359@@ -1360,7 +1360,7 @@ class MiniscriptDescriptor final : public DescriptorImpl
1360 for (const auto& arg : m_pubkey_args) {
1361 providers.push_back(arg->Clone());
1362 }
1363- return std::make_unique<MiniscriptDescriptor>(std::move(providers), miniscript::MakeNodeRef<uint32_t>(*m_node));
1364+ return std::make_unique<MiniscriptDescriptor>(std::move(providers), miniscript::MakeNodeRef<uint32_t>(m_node->Clone()));
The PR description asserts:
Multipath descriptors requires performing a deep copy
Would be happy if you cared to add an elaboration on why that is.
It seemed to me like it should be safe to just have another shared_ptr
point to the same const Node
. Unless something on the outside could own a non-const reference into the node hierarchy and mutate it that way? In that case maybe it would be more robust for the MiniscriptDescriptor
-ctor to be the one ensuring it holds a unique reference by doing the node->Clone()
there instead (if .use_count() > 1
).
Was able to avoid the crash in #30864 using only this on top of the parent commit of this PR:
0 return std::make_unique<MiniscriptDescriptor>(std::move(providers), m_node);
Probably causes other bugs that are obvious for those who understand more of the context. Change passes both unit and non-extended functional tests though.
How come the shallow copy in the version before the PR is causing a problem in this case though?
It seems the default-generated copy-ctor for Node
used before was somehow ending up with corrupt/leaked data, but I’ve been unable to spot what it is. Can’t see any slicing going on. Is something funky being done to the shared_ptr
s?
Regardless, it might be worth adding:
0 Node(const Node&) = delete;
1 Node(Node&&) = delete;
2 Node& operator=(const Node&) = delete;
3 Node& operator=(Node&&) = delete;
Edit: Later versions of my change make the move-operators = default
.
The shallow copy in the version before this PR is causing a problem because we mess with a Node
’s subs
in the destructor to avoid a recursion stack overflow:
https://github.com/bitcoin/bitcoin/blob/2c90f8e08c4cf44d4c1ef3dde0e7f7991b8b9390/src/script/miniscript.h#L513-L524
In the scriptpubkeyman fuzz target using the seed from #30864 we would Parse
the tr(%17/<2;3>,l:pk(%08))
multipath descriptor into a parsed_descs
vector of 2 Descriptor
pointers. We would use the first of those descriptors (tr(xpub..../2,l:pk(4bdf....))#jmgpm0u8
) and move it into a WalletDescriptor
. When exiting the CreateWalletDescriptor
function the parsed_descs
vector would be destructed, along with the second Descriptor
pointer it contains. The destructor of the Node
(l:pk(xpub...)
) it contains would in turn be called, moving out all its subs that are also pointed to by the first Descriptor
’s Node
. As soon as you try to perform an operation on it it will try to access moved structures.
533+ ret.subs.push_back(MakeNodeRef<Key>(sub));
534+ }
535+ return ret;
536+ };
537+ return TreeEval<Node<Key>>(upfn);
538+ }
More smartpointy less segfaulty:
0diff --git a/src/script/descriptor.cpp b/src/script/descriptor.cpp
1index 6053c853bb..b099fdfc3f 100644
2--- a/src/script/descriptor.cpp
3+++ b/src/script/descriptor.cpp
4@@ -1360,7 +1360,7 @@ public:
5 for (const auto& arg : m_pubkey_args) {
6 providers.push_back(arg->Clone());
7 }
8- return std::make_unique<MiniscriptDescriptor>(std::move(providers), miniscript::MakeNodeRef<uint32_t>(m_node->Clone()));
9+ return std::make_unique<MiniscriptDescriptor>(std::move(providers), m_node->Clone());
10 }
11 };
12
13diff --git a/src/script/miniscript.h b/src/script/miniscript.h
14index a74e312e79..1472c9ac4f 100644
15--- a/src/script/miniscript.h
16+++ b/src/script/miniscript.h
17@@ -523,18 +523,18 @@ struct Node {
18 }
19 }
20
21- Node<Key> Clone() const
22+ NodeRef<Key> Clone() const
23 {
24 // Use TreeEval() to avoid a stack-overflow due to recursion
25- auto upfn = [](const Node& node, Span<Node> subs) {
26- Node<Key> ret(node);
27- ret.subs.clear();
28+ auto upfn = [](const Node& node, Span<NodeRef<Key>> subs) {
29+ auto ret(MakeNodeRef<Key>(node));
30+ ret->subs.clear();
31 for (const auto& sub : subs) {
32- ret.subs.push_back(MakeNodeRef<Key>(sub));
33+ ret->subs.push_back(sub);
34 }
35 return ret;
36 };
37- return TreeEval<Node<Key>>(upfn);
38+ return TreeEval<NodeRef<Key>>(upfn);
39 }
40
41 private:
I wrote a regression unit test for this: darosior@d23ffe0.
Added the test commit as well.
643@@ -630,7 +644,10 @@ struct Node {
644 // If evaluation returns std::nullopt, abort immediately.
645 if (!result) return {};
646 // Replace the last node.subs.size() elements of results with the new result.
647- results.erase(results.end() - node.subs.size(), results.end());
648+ // Use pop_back to truncate results to avoid MoveAssignable requirement of erase().
649+ for (size_t i = 0; i < node.subs.size(); ++i) {
650+ results.pop_back();
651+ }
I’ve experimented with a change on top of this PR that removes shared_ptr
usage from src/script/ altogether in order to clarify ownership.
My change is in 80fca845b5f28677207a8fea4a173baaef23036f. It helped uncover another place where the node
needs to be cloned (at the bottom of the outer loop):
0diff --git a/src/script/descriptor.cpp b/src/script/descriptor.cpp
1index b099fdfc3f..f0294df72a 100644
2--- a/src/script/descriptor.cpp
3+++ b/src/script/descriptor.cpp
4@@ -2154,7 +2154,7 @@ std::vector<std::unique_ptr<DescriptorImpl>> ParseScript(uint32_t& key_exp_index
5 for (auto& pub : parser.m_keys) {
6 pubs.emplace_back(std::move(pub.at(i)));
7 }
8- ret.emplace_back(std::make_unique<MiniscriptDescriptor>(std::move(pubs), node));
9+ ret.emplace_back(std::make_unique<MiniscriptDescriptor>(std::move(pubs), node->Clone()));
10 }
11 return ret;
12 }
It would be preferable to add that one-line fix to this PR, but I’m also interested to hear if there is any chance of my shared_ptr
-removal getting any review as a separate PR. I understand Wallet needs more review of existing PRs, so I’m open to leaving it as draft for a while.
Do you have a particular test case that triggered there?
Worked through several iterations of the shared_ptr
-removal over a couple of days, at some point I had:
0ret.emplace_back(std::make_unique<MiniscriptDescriptor>(std::move(pubs), std::move(*node)));
‘cause I wasn’t paying attention to the statement being inside a loop. The second descriptor created in the loop ended up with already-moved-from values, which resulted in an this test failing:
An OR_D
node we were trying to convert into a string was failing because subs
was empty (already moved away from).
Good catch, added that.
Looks like you didn’t.
It helped uncover another place where the node needs to be cloned
Ugh. I should have caught that. Good catch @hodlinator.
std::shared_ptr
way of storing miniscript subnodes, and std::unique_ptr
would suffice. The std::shared_ptr
s were inherited from the miniscript codebase, where the shared_ptrs matter for the policy compiler, but I could understand that Bitcoin Core doesn’t want that burden. Using std::unique_ptr
instead would not leave any chance for shallow duplication.
522@@ -523,6 +523,20 @@ struct Node {
523 }
524 }
525
526+ NodeRef<Key> Clone() const
527+ {
528+ // Use TreeEval() to avoid a stack-overflow due to recursion
529+ auto upfn = [](const Node& node, Span<NodeRef<Key>> children) {
530+ NodeRef<Key> ret(MakeNodeRef<Key>(node));
miniscript::Node
’s destructor means using the implicit copy constructor is always dangerous without modifying the subs as is done here, wouldn’t it be best to delete it (if possible) as another reviewer suggested? (https://github.com/bitcoin/bitcoin/pull/30866#discussion_r1788683926)
Looks like you didn’t.
Oops, pushed it.
Multipath descriptors requires performing a deep copy, so a Clone
function that does that is added to miniscript::Node instead of the
current shallow copy.
Co-Authored-By: Antoine Poinsot <darosior@protonmail.com>
I don’t think anything in the Bitcoin Core codebase actually needs the
std::shared_ptr
way of storing miniscript subnodes, andstd::unique_ptr
would suffice. Thestd::shared_ptr
s were inherited from the miniscript codebase, where the shared_ptrs matter for the policy compiler, but I could understand that Bitcoin Core doesn’t want that burden. Usingstd::unique_ptr
instead would not leave any chance for shallow duplication.
Indeed, the std:;shared_ptr
wasn’t really needed and I’ve changed NodeRef
to be a std::unique_ptr
. This appears to also catch the case found by @hodlinator earlier.
Any way to add a test that fails/crashes without the last push?
I believe changing to std::unique_ptr
and removing the copy constructor covers that case.
535+ Assert(node.data.empty() && node.subs.empty());
536+ ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.keys, node.k);
537+ } else if (!node.data.empty()) {
538+ Assert(node.keys.empty() && node.subs.empty());
539+ ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.data, node.k);
540+ } else if (!node.subs.empty()) {
just an observation: in all branches other than this one children.empty() == true
since node.subs.empty() == true
, and more generally children.size() == node.subs.size()
Verified with the following diff:
0diff --git a/src/script/miniscript.h b/src/script/miniscript.h
1@@ -527,6 +527,7 @@ struct Node {
2 {
3 // Use TreeEval() to avoid a stack-overflow due to recursion
4 auto upfn = [](const Node& node, Span<NodeRef<Key>> children) {
5+ Assert(node.subs.size() == children.size());
6 NodeRef<Key> ret;
crACK https://github.com/bitcoin/bitcoin/pull/30866/commits/815467f46f47df6ff52686b0144430c14a31f4a8
This resolves #30864.
0$ echo "dHIoJTE3LzwyOzM+LGw6cGsoJTA4KSk=" | base64 --decode > scriptpubkeyman.crash
1$ FUZZ=scriptpubkeyman ./build/src/test/fuzz/fuzz scriptpubkeyman.crash
2scriptpubkeyman: succeeded against 1 files in 0s.
The default copy constructor of miniscript::Node
can’t be safe because of the custom iterative destructor miniscript::~Node
and deleting it is an improvement.
1686@@ -1658,6 +1687,10 @@ struct Node {
1687 : Node(internal::NoDupCheck{}, ctx.MsContext(), nt, std::move(sub), val) { DuplicateKeyCheck(ctx); }
1688 template <typename Ctx> Node(const Ctx& ctx, Fragment nt, uint32_t val = 0)
1689 : Node(internal::NoDupCheck{}, ctx.MsContext(), nt, val) { DuplicateKeyCheck(ctx); }
1690+
1691+ // Delete copy constructor and assignment operator
nit: Could be more helpful?
0 // Delete copy constructor and assignment operator, use Clone() instead
183@@ -184,11 +184,11 @@ inline consteval Type operator"" _mst(const char* c, size_t l) {
184 using Opcode = std::pair<opcodetype, std::vector<unsigned char>>;
185
186 template<typename Key> struct Node;
187-template<typename Key> using NodeRef = std::shared_ptr<const Node<Key>>;
188+template<typename Key> using NodeRef = std::unique_ptr<const Node<Key>>;
189
190 //! Construct a miniscript node as a shared_ptr.
shared_ptr
.
ACK 815467f46f47df6ff52686b0144430c14a31f4a8
Amazed by how frictionless it was to switch from shared_ptr
-> unique_ptr
. unique_ptr
still is an extra level of unnecessary indirection compared to my 80fca845b5f28677207a8fea4a173baaef23036f, but the latter is a much more disruptive change. Should probably fix remaining mention of shared_ptr
, see inline comment.
Thanks for deleting the copy-ctor & assignment operator as I (imprecisely) suggested, making this kind of mistake no longer compile.
Good to see you could avoid adding a new Node
-constructor just to be used in Clone()
.
Re-verified that fuzz failure is fixed by the first commit (and that it still works by the last one).
Passed modified unit tests.
There's no need for it to be a shared_ptr.
trivial reACK https://github.com/bitcoin/bitcoin/commit/352391c2cf1a45231ae92ca92d2415b3786ab9ad
Recent push only changes comments.
0$ git diff 815467f..352391c
1diff --git a/src/script/miniscript.h b/src/script/miniscript.h
2index 04e487f884..4f5c38bb7b 100644
3--- a/src/script/miniscript.h
4+++ b/src/script/miniscript.h
5@@ -186,7 +186,7 @@ using Opcode = std::pair<opcodetype, std::vector<unsigned char>>;
6 template<typename Key> struct Node;
7 template<typename Key> using NodeRef = std::unique_ptr<const Node<Key>>;
8
9-//! Construct a miniscript node as a shared_ptr.
10+//! Construct a miniscript node as a unique_ptr.
11 template<typename Key, typename... Args>
12 NodeRef<Key> MakeNodeRef(Args&&... args) { return std::make_unique<const Node<Key>>(std::forward<Args>(args)...); }
13
14@@ -1688,7 +1688,7 @@ public:
15 template <typename Ctx> Node(const Ctx& ctx, Fragment nt, uint32_t val = 0)
16 : Node(internal::NoDupCheck{}, ctx.MsContext(), nt, val) { DuplicateKeyCheck(ctx); }
17
18- // Delete copy constructor and assignment operator
19+ // Delete copy constructor and assignment operator, use Clone() instead
20 Node(const Node&) = delete;
21 Node& operator=(const Node&) = delete;
re-ACK 352391c2cf1a45231ae92ca92d2415b3786ab9ad
Confirmed through range-diff to only be comment changes following my previous feedback.
530+ NodeRef<Key> ret;
531+ // As all members of Node are const, except for subs, we need to construct the cloned node with all of these members.
532+ // However, there is no constructor that takes all three of data, keys, and subs.
533+ // But, they are mutually exclusive, so we can use the appropriate constructor depending on what is available.
534+ if (!node.keys.empty()) {
535+ Assert(node.data.empty() && node.subs.empty());
It seems much simpler and robust to just add the missing constructor than matching on available data and asserting?
0diff --git a/src/script/miniscript.h b/src/script/miniscript.h
1index 04e487f8845..07214bdf2a4 100644
2--- a/src/script/miniscript.h
3+++ b/src/script/miniscript.h
4@@ -527,27 +527,11 @@ struct Node {
5 {
6 // Use TreeEval() to avoid a stack-overflow due to recursion
7 auto upfn = [](const Node& node, Span<NodeRef<Key>> children) {
8- NodeRef<Key> ret;
9- // As all members of Node are const, except for subs, we need to construct the cloned node with all of these members.
10- // However, there is no constructor that takes all three of data, keys, and subs.
11- // But, they are mutually exclusive, so we can use the appropriate constructor depending on what is available.
12- if (!node.keys.empty()) {
13- Assert(node.data.empty() && node.subs.empty());
14- ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.keys, node.k);
15- } else if (!node.data.empty()) {
16- Assert(node.keys.empty() && node.subs.empty());
17- ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.data, node.k);
18- } else if (!node.subs.empty()) {
19- Assert(node.data.empty() && node.keys.empty());
20- std::vector<NodeRef<Key>> new_subs;
21- for (auto child = children.begin(); child != children.end(); ++child) {
22- new_subs.emplace_back(std::move(*child));
23- }
24- ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, std::move(new_subs), node.k);
25- } else {
26- ret = MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, node.k);
27+ std::vector<NodeRef<Key>> new_subs;
28+ for (auto child = children.begin(); child != children.end(); ++child) {
29+ new_subs.emplace_back(std::move(*child));
30 }
31- return ret;
32+ return MakeNodeRef<Key>(internal::NoDupCheck{}, node.m_script_ctx, node.fragment, std::move(new_subs), node.keys, node.data, node.k);
33 };
34 return TreeEval<NodeRef<Key>>(upfn);
35 }
36@@ -1661,6 +1645,8 @@ public:
37 bool operator==(const Node<Key>& arg) const { return Compare(*this, arg) == 0; }
38
39 // Constructors with various argument combinations, which bypass the duplicate key check.
40+ Node(internal::NoDupCheck, MiniscriptContext script_ctx, Fragment nt, std::vector<NodeRef<Key>> sub, std::vector<Key> key, std::vector<unsigned char> arg, uint32_t val)
41+ : fragment(nt), k(val), data(std::move(arg)), subs(std::move(sub)), m_script_ctx{script_ctx}, ops(CalcOps()), ss(CalcStackSize()), ws(CalcWitnessSize()), typ(CalcType()), scriptlen(CalcScriptLen()) {}
42 Node(internal::NoDupCheck, MiniscriptContext script_ctx, Fragment nt, std::vector<NodeRef<Key>> sub, std::vector<unsigned char> arg, uint32_t val = 0)
43 : fragment(nt), k(val), data(std::move(arg)), subs(std::move(sub)), m_script_ctx{script_ctx}, ops(CalcOps()), ss(CalcStackSize()), ws(CalcWitnessSize()), typ(CalcType()), scriptlen(CalcScriptLen()) {}
44 Node(internal::NoDupCheck, MiniscriptContext script_ctx, Fragment nt, std::vector<unsigned char> arg, uint32_t val = 0)
Clone()
a lot, only thing I would add is make the new constructor private
if we want to discourage outside uses.