miniscript: make operator""_mst consteval #28657

pull sipa wants to merge 1 commits into bitcoin:master from sipa:202310_miniscript_consteval changing 3 files +38 −31
  1. sipa commented at 2:19 pm on October 16, 2023: member

    It seems modern compilers don’t realize that all invocations of operator""_mst can be evaluated at compile time, despite the constexpr keyword.

    Since C++20, we can force them to evaluate at compile time using consteval, turning all the miniscript type constants into actual compile-time constants.

    This should give a nice but not very important speedup for miniscript logic, but it’s also a way to start testing C++20 features.

  2. DrahtBot commented at 2:19 pm on October 16, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK hebasto, theuni
    Concept ACK fanquake

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

  3. DrahtBot added the label Descriptors on Oct 16, 2023
  4. sipa commented at 2:21 pm on October 16, 2023: member
    Ping @maflcko.
  5. in src/script/miniscript.h:29 in 3d3ad8a1ee outdated
    25@@ -26,6 +26,12 @@
    26 #include <util/string.h>
    27 #include <util/vector.h>
    28 
    29+#if __cplusplus >= 201811L
    


    maflcko commented at 2:27 pm on October 16, 2023:
    Maybe wait until next Monday and then remove the if-else macro?

    sipa commented at 2:32 pm on October 16, 2023:
    Wow, you’re saying the future is next week?

    maflcko commented at 2:44 pm on October 16, 2023:
    Heh, maybe too optimistic, given that it is still waiting on macOS (https://github.com/bitcoin/bitcoin/pull/28622) and also Windows (see the apparent CI failure here).

    fanquake commented at 2:46 pm on October 16, 2023:

    and also Windows (see the apparent CI failure here).

    For reference:

    0 D:\a\bitcoin\bitcoin\src\script\miniscript.h(160,13): error C2248: 'miniscript::Type::Type': cannot access private member declared in class 'miniscript::Type' [D:\a\bitcoin\bitcoin\build_msvc\libbitcoin_common\libbitcoin_common.vcxproj]
    1D:\a\bitcoin\bitcoin\src\script\miniscript.h(133,5): message : see declaration of 'miniscript::Type::Type' [D:\a\bitcoin\bitcoin\build_msvc\libbitcoin_common\libbitcoin_common.vcxproj]
    2D:\a\bitcoin\bitcoin\src\script\miniscript.h(128,7): message : see declaration of 'miniscript::Type' [D:\a\bitcoin\bitcoin\build_msvc\libbitcoin_common\libbitcoin_common.vcxproj]
    3D:\a\bitcoin\bitcoin\src\script\miniscript.cpp(17,27): message : while evaluating consteval function 'miniscript::operator ""_mst' [D:\a\bitcoin\bitcoin\build_msvc\libbitcoin_common\libbitcoin_common.vcxproj]
    4D:\a\bitcoin\bitcoin\src\script\miniscript.cpp(234,33): error C7595: 'miniscript::operator ""_mst': call to immediate function is not a constant expression [D:\a\bitcoin\bitcoin\build_msvc\libbitcoin_common\libbitcoin_common.vcxproj]
    5D:\a\bitcoin\bitcoin\src\script\miniscript.cpp(234,33): fatal  error C1003: error count exceeds 100; stopping compilation [D:\a\bitcoin\bitcoin\build_msvc\libbitcoin_common\libbitcoin_common.vcxproj]
    

    fanquake commented at 2:47 pm on October 16, 2023:
    Note that the failing Windows CI is also already using MSVCs C++20 mode.

    maflcko commented at 3:04 pm on October 16, 2023:

    Now it is

    0  miniscript.cpp
    1D:\a\bitcoin\bitcoin\src\script\miniscript.cpp(234,33): error C7595: 'miniscript::operator ""_mst': call to immediate function is not a constant expression [D:\a\bitcoin\bitcoin\build_msvc\libbitcoin_common\libbitcoin_common.vcxproj]
    2D:\a\bitcoin\bitcoin\src\script\miniscript.cpp(234,45): error C7595: 'miniscript::operator ""_mst': call to immediate function is not a constant expression [D:\a\bitcoin\bitcoin\build_msvc\libbitcoin_common\libbitcoin_common.vcxproj]
    

    sipa commented at 3:06 pm on October 16, 2023:

    Go home MSVC, you’re drunk.

    Somehow it thinks that calls to ""_mst are immediate, unless they occur inside ternary operators!?


    maflcko commented at 3:11 pm on October 16, 2023:
    Yeah, not sure if we are using the latest msvc, and if this is a bug that is fixed in a newer release. cc @hebasto @sipsorcery

    hebasto commented at 3:21 pm on October 16, 2023:

    Yeah, not sure if we are using the latest msvc, and if this is a bug that is fixed in a newer release. cc @hebasto @sipsorcery

    0MSBuild version 17.7.2+d6990bcfa for .NET Framework
    

    It is the latest one.



    sipa commented at 3:31 pm on October 16, 2023:
    I don’t think so; MSVC is only complaining about consteval literals inside ( ? : ) ternaries. I’ve pushed another attempt to avoid it by moving the subexpressions out of the ternaries.

    sipa commented at 4:40 pm on October 16, 2023:
    Ok that appears to have worked.
  6. sipa force-pushed on Oct 16, 2023
  7. sipa force-pushed on Oct 16, 2023
  8. sipa force-pushed on Oct 16, 2023
  9. sipa force-pushed on Oct 16, 2023
  10. sipa force-pushed on Oct 16, 2023
  11. sipa renamed this:
    miniscript: make operator_mst consteval in C++20 mode
    miniscript: make operator""_mst consteval in C++20 mode
    on Oct 16, 2023
  12. fanquake added this to the milestone 27.0 on Oct 17, 2023
  13. fanquake commented at 1:32 pm on October 17, 2023: member
    Concept ACK. It’s annoying that we have to refactor this code just to accomodate MSVC. I agree that we should just wait until branch-off, and use conteval directly. Put this on 27.x.
  14. DrahtBot added the label CI failed on Oct 22, 2023
  15. DrahtBot removed the label CI failed on Oct 27, 2023
  16. sipa force-pushed on Dec 11, 2023
  17. sipa commented at 3:08 pm on December 11, 2023: member
    Rebased.
  18. sipa renamed this:
    miniscript: make operator""_mst consteval in C++20 mode
    miniscript: make operator""_mst consteval
    on Dec 11, 2023
  19. fanquake requested review from darosior on Dec 11, 2023
  20. DrahtBot added the label CI failed on Dec 11, 2023
  21. darosior commented at 2:17 pm on December 31, 2023: member

    CI is failing with

     0/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x23
     1libbitcoin_common.a(libbitcoin_common_a-descriptor.o): in function `(anonymous namespace)::ParseScript(unsigned int&, Span<char const>&, (anonymous namespace)::ParseScriptContext, FlatSigningProvider&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)':
     2descriptor.cpp:(.text+0x390f): undefined reference to `miniscript::operator"" _mst(char const*, unsigned int)'
     3/usr/bin/ld: descriptor.cpp:(.text+0x396c): undefined reference to `miniscript::operator"" _mst(char const*, unsigned int)'
     4/usr/bin/ld: descriptor.cpp:(.text+0x39ed): undefined reference to `miniscript::operator"" _mst(char const*, unsigned int)'
     5/usr/bin/ld: libbitcoin_common.a(libbitcoin_common_a-descriptor.o): in function `std::shared_ptr<miniscript::Node<unsigned int> const> miniscript::internal::Parse<unsigned int, (anonymous namespace)::KeyParser>(Span<char const>, (anonymous namespace)::KeyParser const&)':
     6descriptor.cpp:(.text+0x1df07): undefined reference to `miniscript::operator"" _mst(char const*, unsigned int)'
     7/usr/bin/ld: libbitcoin_common.a(libbitcoin_common_a-descriptor.o): in function `(anonymous namespace)::MiniscriptDescriptor::MaxSatisfactionElems() const':
     8descriptor.cpp:(.text+0x2d316): undefined reference to `miniscript::operator"" _mst(char const*, unsigned int)'
     9/usr/bin/ld: libbitcoin_common.a(libbitcoin_common_a-descriptor.o):descriptor.cpp:(.text+0x2d333): more undefined references to `miniscript::operator"" _mst(char const*, unsigned int)' follow
    10clang: error: linker command failed with exit code 1 (use -v to see invocation)
    
  22. hebasto commented at 1:31 pm on January 1, 2024: member
    Concept ACK.
  23. hebasto commented at 5:19 pm on January 1, 2024: member
    Could the CI issues be due to Clang partially supporting consteval before clang-17?
  24. maflcko commented at 11:08 am on January 2, 2024: member
    Locally, it works, starting with clang++-15.
  25. maflcko commented at 1:14 pm on January 2, 2024: member
    Confirmed that this cuts a third of the runtime off of the fuzz input from #28832 (comment)
  26. fanquake commented at 1:16 pm on January 2, 2024: member
    Maybe we should also retry the straight constexpr -> consteval change, without having to do all the refactoring for MSVC? Microsoft might have fixed their compiler.
  27. sipa commented at 4:24 pm on January 2, 2024: member
  28. maflcko commented at 3:22 pm on January 12, 2024: member
    Maybe mark as draft while this is waiting on #29165? (29165 won’t happen unless someone bumps google’s oss-fuzz’s clang)
  29. sipa marked this as a draft on Jan 12, 2024
  30. maflcko removed this from the milestone 27.0 on Jan 29, 2024
  31. maflcko added this to the milestone 28.0 on Jan 29, 2024
  32. DrahtBot removed this from the milestone 28.0 on Apr 25, 2024
  33. maflcko commented at 2:39 pm on April 29, 2024: member
    Could rebase, as CI on master should now be passing with clang-15 in.
  34. maflcko commented at 9:32 am on May 3, 2024: member
  35. hebasto commented at 11:59 am on May 3, 2024: member

    Could rebase, as CI on master should now be passing with clang-15 in.

    As we now build the fuzz.exe on MSVC, this PR seems require a little adjustment while rebasing:

     0--- a/src/test/fuzz/miniscript.cpp
     1+++ b/src/test/fuzz/miniscript.cpp
     2@@ -391,6 +391,7 @@ std::optional<NodeInfo> ConsumeNodeStable(MsCtx script_ctx, FuzzedDataProvider&
     3     bool allow_K = (type_needed == ""_mst) || (type_needed << "K"_mst);
     4     bool allow_V = (type_needed == ""_mst) || (type_needed << "V"_mst);
     5     bool allow_W = (type_needed == ""_mst) || (type_needed << "W"_mst);
     6+    static constexpr auto B{"B"_mst}, K{"K"_mst}, V{"V"_mst}, W{"W"_mst};
     7 
     8     switch (provider.ConsumeIntegral<uint8_t>()) {
     9         case 0:
    10@@ -440,22 +441,22 @@ std::optional<NodeInfo> ConsumeNodeStable(MsCtx script_ctx, FuzzedDataProvider&
    11         }
    12         case 11:
    13             if (!(allow_B || allow_K || allow_V)) return {};
    14-            return {{{"B"_mst, type_needed, type_needed}, Fragment::ANDOR}};
    15+            return {{{B, type_needed, type_needed}, Fragment::ANDOR}};
    16         case 12:
    17             if (!(allow_B || allow_K || allow_V)) return {};
    18-            return {{{"V"_mst, type_needed}, Fragment::AND_V}};
    19+            return {{{V, type_needed}, Fragment::AND_V}};
    20         case 13:
    21             if (!allow_B) return {};
    22-            return {{{"B"_mst, "W"_mst}, Fragment::AND_B}};
    23+            return {{{B, W}, Fragment::AND_B}};
    24         case 15:
    25             if (!allow_B) return {};
    26-            return {{{"B"_mst, "W"_mst}, Fragment::OR_B}};
    27+            return {{{B, W}, Fragment::OR_B}};
    28         case 16:
    29             if (!allow_V) return {};
    30-            return {{{"B"_mst, "V"_mst}, Fragment::OR_C}};
    31+            return {{{B, V}, Fragment::OR_C}};
    32         case 17:
    33             if (!allow_B) return {};
    34-            return {{{"B"_mst, "B"_mst}, Fragment::OR_D}};
    35+            return {{{B, B}, Fragment::OR_D}};
    36         case 18:
    37             if (!(allow_B || allow_K || allow_V)) return {};
    38             return {{{type_needed, type_needed}, Fragment::OR_I}};
    39@@ -472,25 +473,25 @@ std::optional<NodeInfo> ConsumeNodeStable(MsCtx script_ctx, FuzzedDataProvider&
    40         }
    41         case 20:
    42             if (!allow_W) return {};
    43-            return {{{"B"_mst}, Fragment::WRAP_A}};
    44+            return {{{B}, Fragment::WRAP_A}};
    45         case 21:
    46             if (!allow_W) return {};
    47-            return {{{"B"_mst}, Fragment::WRAP_S}};
    48+            return {{{B}, Fragment::WRAP_S}};
    49         case 22:
    50             if (!allow_B) return {};
    51-            return {{{"K"_mst}, Fragment::WRAP_C}};
    52+            return {{{K}, Fragment::WRAP_C}};
    53         case 23:
    54             if (!allow_B) return {};
    55-            return {{{"V"_mst}, Fragment::WRAP_D}};
    56+            return {{{V}, Fragment::WRAP_D}};
    57         case 24:
    58             if (!allow_V) return {};
    59-            return {{{"B"_mst}, Fragment::WRAP_V}};
    60+            return {{{B}, Fragment::WRAP_V}};
    61         case 25:
    62             if (!allow_B) return {};
    63-            return {{{"B"_mst}, Fragment::WRAP_J}};
    64+            return {{{B}, Fragment::WRAP_J}};
    65         case 26:
    66             if (!allow_B) return {};
    67-            return {{{"B"_mst}, Fragment::WRAP_N}};
    68+            return {{{B}, Fragment::WRAP_N}};
    69         case 27: {
    70             if (!allow_B || !IsTapscript(script_ctx)) return {};
    71             const auto k = provider.ConsumeIntegral<uint16_t>();
    
  36. sipa force-pushed on May 3, 2024
  37. sipa commented at 12:05 pm on May 3, 2024: member
    @hebasto Thanks, rebased and applied.
  38. sipa marked this as ready for review on May 3, 2024
  39. hebasto commented at 12:18 pm on May 3, 2024: member

    @hebasto Thanks, rebased and applied.

    Sorry. It doesn’t work – #30031.

    UPD. More adjustment needed.

  40. hebasto approved
  41. hebasto commented at 12:55 pm on May 3, 2024: member

    ACK 73619b819c37e98bd9a100bea14a3366aebfabb8.

    I apologise for messing up.

    The #30031, which is built on top of this PR, succeeds. It includes additional code changes in src/test/fuzz/miniscript.cpp (with some inconsistency in local variable naming). @sipa feel free to pull changes from #30031 into this PR.

  42. DrahtBot requested review from fanquake on May 3, 2024
  43. DrahtBot removed the label CI failed on May 3, 2024
  44. theuni commented at 1:54 pm on May 3, 2024: member
    utACK with or without @hebasto’s additional changes.
  45. miniscript: make operator_mst consteval
    It seems modern compilers don't realize that all invocations of operator""_mst
    can be evaluated at compile time, despite the constexpr keyword.
    
    Since C++20, we can force them to evaluate at compile time, turning all the
    miniscript type constants into actual compile-time constants.
    
    It appears that MSVC does not support consteval operator"" when used inside
    certain expressions. For the few places where this happens, define a
    constant outside the operator call.
    
    Co-Authored-By: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
    63317103c9
  46. sipa force-pushed on May 3, 2024
  47. sipa commented at 3:39 pm on May 3, 2024: member
    Incorporated the changes from #30031.
  48. hebasto approved
  49. hebasto commented at 3:44 pm on May 3, 2024: member
    re-ACK 63317103c9f2b0635558da814567bb79c17ae851.
  50. DrahtBot requested review from theuni on May 3, 2024
  51. theuni approved
  52. theuni commented at 3:51 pm on May 3, 2024: member
    utACK 63317103c9f2b0635558da814567bb79c17ae851
  53. fanquake merged this on May 4, 2024
  54. fanquake closed this on May 4, 2024


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me