prevector uses memmove to move around data, that means it can only be used with types that are trivially copyable. That implies that the types are trivially destructible, thus the checks for is_trivially_destructible are not needed.
prevector: enforce is_trivially_copyable_v #24962
pull martinus wants to merge 1 commits into bitcoin:master from martinus:2022-04_prevector_fixes changing 1 files +3 −12-
martinus commented at 6:08 PM on April 24, 2022: contributor
-
11e7908484
prevector: only allow trivially copyable types
The prevector implementation currently can't be used with types that are not trivially copyable, due to the use of memmove. Trivially copyable implies that it is trivially destructible, see https://eel.is/c++draft/class.prop#1.3 That means that the checks for std::is_trivially_destructible are not necessary, and in fact where used it wouldn't be enough. E.g. in `erase(iterator, iterator)` the elements in range first-last are destructed, but it does not destruct elements left after `memmove`. This commit removes the checks for `std::is_trivially_destructible` and instead adds a `static_assert(std::is_trivially_copyable_v<T>);` to make sure `prevector` is only used with supported types.
-
DrahtBot commented at 12:55 AM on April 25, 2022: member
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--174a7506f384e20aa4161008e828411d-->
Conflicts
No conflicts as of last run.
-
in src/prevector.h:418 in 1ed610d389 outdated
424 | - _size -= last - p; 425 | - } 426 | - memmove(&(*first), &(*last), endp - ((char*)(&(*last)))); 427 | + char* endp = reinterpret_cast<char*>(end().ptr); 428 | + _size -= last - p; 429 | + memmove(first.ptr, last.ptr, endp - reinterpret_cast<char*>(last.ptr));
MarcoFalke commented at 7:20 AM on April 25, 2022:std::memmove(first.ptr, last.ptr, endp - reinterpret_cast<char*>(last.ptr));Maybe switch to
std::memmove, which comes with the documentation:https://en.cppreference.com/w/cpp/string/byte/memmove
If the objects are potentially-overlapping or not TriviallyCopyable, the behavior of memmove is not specified and may be undefined.
martinus commented at 3:13 PM on April 25, 2022:I'll change all occurrences of memmove in that file, plus memcpy as well, right? Then I'll include
cstringinstead ofstring.h
MarcoFalke commented at 3:28 PM on April 25, 2022:Oh, if the changes are more expensive, then maybe do it in another commit or pull request.
jonatack commented at 4:05 PM on April 29, 2022:Does the UB section of the documentation in https://en.cppreference.com/w/cpp/string/byte/memmove (C++ memmove) apply to https://en.cppreference.com/w/c/string/byte/memmove (C memmove)?
jonatack commented at 4:07 PM on April 29, 2022:In the C version:
The behavior is undefined if access occurs beyond the end of the dest array.
The behavior is undefined if either dest or src is an invalid or null pointer. <-- also specified in the C++ memmove doc
The behavior is undefined if the size of the character array pointed to by dest < count <= destsz; in other words, an erroneous value of destsz does not expose the impending buffer overflow.
In the C++ version:
If the objects are potentially-overlapping or not TriviallyCopyable, the behavior of memmove is not specified and may be undefined. <-- not stated in the C version
MarcoFalke commented at 4:11 PM on April 29, 2022:I am not familiar with C, but maybe every class is trivially copyable in C?
sipa commented at 4:12 PM on April 29, 2022:C doesn't have classes.
jonatack commented at 4:27 PM on April 29, 2022:Right (interesting historic context here). So IIUC the premise in the pull description only applies to the C++ memmove and it may make sense to change to it at the same time (or not do this change at all).
martinus commented at 7:50 AM on April 30, 2022:the premise in the pull description only applies to the C++ memmove and it may make sense to change to it at the same time (or not do this change at all).
I still think it is necessary to do the change, even when we stay with
memmove. The current implementation just does not work with complex objects. E.g. here is a small example:BOOST_AUTO_TEST_CASE(Erase) { prevector<8, std::string> pv; pv.push_back("a"); pv.push_back("b"); pv.erase(pv.begin()); pv.push_back("X"); std::cout << pv.front() << std::endl; }One would expect that this prints "b", as "a" was deleted and "b" should now be the first element. But l get this output:
Running 1 test case... X free(): invalid pointer unknown location(0): fatal error: in "prevector_tests/Erase": signal: SIGABRT (application abort requested)So suddenly the first element is not "b" but "X". This is due to to small string optimization, the strings point to member data, and this is simply copied. Complex objects have to be moved for this to work.
here is another motivating example with
std::swap:BOOST_AUTO_TEST_CASE(NonTrivial) { prevector<8, std::string> a; prevector<8, std::string> b; a.push_back("a"); b.push_back("b"); std::cout << "a=" << a.front() << ", b=" << b.front() << std::endl; std::swap(a, b); std::cout << "a=" << a.front() << ", b=" << b.front() << std::endl; }Running that test produces this output:
Running 1 test case... a=a, b=b a=a, b=b free(): invalid pointer unknown location(0): fatal error: in "prevector_tests/NonTrivial": signal: SIGABRT (application abort requested)So swap doesn't seem to do anything for these objects, and destructing will cause a SIGABRT.
I think we should either restrict prevector to only work with supported types, or to correctly implement it for nontrivial types. Fixing this is quite a bit more work though, and would need a lot more testing.
theStack commented at 12:39 PM on April 25, 2022: memberLight Concept ACK (I'm not deeply familiar with templates, type traits, etc. but from a higher level perspective, removing unused code and fixing UB seems like a good idea!)
jonatack commented at 11:28 AM on April 28, 2022: memberACK 1ed610d3892917240589ca55720340838154742d
MarcoFalke commented at 7:36 AM on April 29, 2022: memberI haven't reviewed the second commit and I wonder why no sanitizer complains about them. I presume the reason is that this is no stdlib iterator? See also: #21817 (review) and https://github.com/bitcoin/bitcoin/pull/21869
martinus commented at 11:08 AM on April 29, 2022: contributorI wonder why no sanitizer complains about them
I was wondering about that too, and thought this might be because the optimizer is just optimizing that away. After some googling though it seems this is actually allowed and no UB:
- In C there is an explicit rule that allowes to do
&*whatever, even when whatever is a nullptr: https://twitter.com/shafikyaghmour/status/1185235611407417344 - But in C++ there is no such rule that allows this: https://twitter.com/shafikyaghmour/status/1185235735743352833
- That issue was then addressed a few years ago: http://www.open-std.org/jtc1/sc22/wg21/docs/cwg_active.html#232
- and now this definition should say that it's actually allowed: https://eel.is/c++draft/expr.unary.op#1
TL;DR: I was most likely wrong and it is fine to do
&*end()MarcoFalke commented at 11:51 AM on April 29, 2022: memberOk, maybe drop the second commit or move it to a follow-up pull?
martinus force-pushed on Apr 29, 2022martinus renamed this:prevector: enforce `is_trivially_copyable_v`, don't dereference iterators
prevector: enforce `is_trivially_copyable_v`
on Apr 29, 2022martinus renamed this:prevector: enforce `is_trivially_copyable_v`
prevector: enforce is_trivially_copyable_v
on Apr 29, 2022jonatack commented at 4:08 PM on April 29, 2022: memberre-ACK modulo question in #24962 (review)
w0xlt approvedw0xlt commented at 5:19 PM on April 29, 2022: contributorin src/prevector.h:38 in 11e7908484
34 | @@ -35,6 +35,8 @@ 35 | */ 36 | template<unsigned int N, typename T, typename Size = uint32_t, typename Diff = int32_t> 37 | class prevector { 38 | + static_assert(std::is_trivially_copyable_v<T>);
MarcoFalke commented at 8:06 AM on April 30, 2022:Out of interest, I am wondering to what extend trivially copyable is related to trivially destructible?
martinus commented at 9:37 AM on April 30, 2022:Per the standard a class is only trivially copyable when it also has a trivial destructor: https://eel.is/c++draft/class#prop-1.3
MarcoFalke approvedMarcoFalke commented at 11:08 AM on April 30, 2022: memberreview ACK 11e79084845a78e2421ea3abafe0de5a54ca2bde 🏯
I checked that the code is unused at run-time. Also, in light of silent merge conflicts, the added static_assert should ensure that any code that tried to use it fails to compile. According to https://eel.is/c++draft/class#prop-1.3 is_trivially_copyable_v is false if is_trivially_destructible_v is false.
I have not verified that this is actually UB, but the provided example seems to imply so. Regardless, removing code because it is unused seems good enough.
<details><summary>Show signature</summary>
Signature:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 review ACK 11e79084845a78e2421ea3abafe0de5a54ca2bde 🏯 I checked that the code is unused at run-time. Also, in light of silent merge conflicts, the added static_assert should ensure that any code that tried to use it fails to compile. According to https://eel.is/c++draft/class#prop-1.3 is_trivially_copyable_v is false if is_trivially_destructible_v is false. I have not verified that this is actually UB, but the provided example seems to imply so. Regardless, removing code because it is unused seems good enough. -----BEGIN PGP SIGNATURE----- iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p pUhd7gv/X+b9gxeWmyrk43wUYLfe9bXgwHPM3JRI068e7fVFeml8flhkQIYaY4eP Km+G8XGwEf0TaTY2Y1ExmSxsBE3EKF9sYRLLga/FOeBamKpQwBHefNOQyPEdP9wr KQSYv7mB27U6xt8tFv+p1i9Hr4LWQVEEWX2dYCl8qio+kObOQQFI4xhManXnp06d +8Vp+7IbS2H9jLIPK+1byiozLWaT8pr15PRRgQL1XuvlBwEdoBmF/usMP/tV4MaC Ns8CXIXB5guiA96HDSsK0dsDzQgSqrX8ZrqSP0007lkW4kOW/TqsxMQhtTbEppOO BadS8y0QIBcrOl2WGICJlfSqm3BKb7/YCbR+jdAaS5LHT5YxM6/k4a0KjPt1ijpM VibEmNkimCin3cpVbxAouptTdP1KyNoeTCrDU1yGM3o/tMAy0axjPhos9Iv2avvn xtBGzMFe6XzRQR25mvXBsEd/v7Dbe/zBJPLRFGFr7NhR+2XBSq5B7ncaj9PwK5uy r4uqloDb =xRUi -----END PGP SIGNATURE-----</details>
MarcoFalke deleted a comment on May 1, 2022fanquake requested review from ajtowns on May 16, 2022ajtowns approvedajtowns commented at 2:19 PM on May 16, 2022: memberACK 11e79084845a78e2421ea3abafe0de5a54ca2bde -- code review only
MarcoFalke merged this on May 16, 2022MarcoFalke closed this on May 16, 2022martinus deleted the branch on May 16, 2022luke-jr referenced this in commit 1c8106b473 on May 21, 2022sidhujag referenced this in commit 504723f366 on May 28, 2022DrahtBot locked this on May 19, 2023
This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-22 09:14 UTC
More mirrored repositories can be found on mirror.b10c.me