tinyformat: refactor: increase compile-time checks and don’t throw for tfm::format_error #30928

pull stickies-v wants to merge 4 commits into bitcoin:master from stickies-v:2024-09/try-format changing 15 files +138 −130
  1. stickies-v commented at 2:45 pm on September 19, 2024: contributor

    Avoid unexpected run-time crashes from string formatting by enforcing compile-time format string checks for most* tfm::format and strprintf usage, and by returning an error string instead of throwing for a run-time tfm::format_error so callsites don’t need to implement this error handling.

    This PR should introduce no behaviour change. The main changes are:

    • remove the const std::string& tfm::format overload since it’s not necessary. Usage of this overload is removed by one of:
      • replacing string concatenation in the format string with just an extra parameter
      • using the bilingual_str overload
      • using the tfm::format_raw functions (only for tinyformat implementation or tests)
    • rename the throwing tfm::format functions to tfm::format_raw, so call existing callsites by default use the safer util::ConstevalFormatString overloads. Callsites that for some reason don’t pass the compile-time checks (such as in bitcoin-cli.cpp) can use tfm::format_raw.
    • make tfm::format_error internal to tinyformat and just return error strings instead, so callsites don’t need to implement error handling to prevent crashing.

    *The bilingual_str format(const bilingual_str& fmt, const Args&... args) is not yet compile-time checked, which means the lint-format-strings.py test can’t be removed yet.

    Follow-up on #30889 (review), an alternative to #15926.

  2. DrahtBot commented at 2:45 pm on September 19, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    Stale ACK l0rinc, maflcko

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #30930 (netinfo: add peer services column and outbound-only peers list by jonatack)
    • #30929 (log: Enforce trailing newline at compile time by maflcko)
    • #29936 (fuzz: wallet: add target for CreateTransaction by brunoerg)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Utils/log/libs on Sep 19, 2024
  4. stickies-v force-pushed on Sep 19, 2024
  5. DrahtBot added the label CI failed on Sep 19, 2024
  6. DrahtBot commented at 2:55 pm on September 19, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/30382700932

    Make sure to run all tests locally, according to the documentation.

    The failure may happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  7. in src/util/string.h:258 in bb9b55f51c outdated
    255+        log_msg = tfm::format(fmt.fmt, args...);
    256+    } catch (tinyformat::format_error& fmterr) {
    257+        /* Original format string will have newline so don't add one here */
    258+        log_msg = "Error \"" + std::string{fmterr.what()} + "\" while formatting log message: \"" + fmt.fmt + "\"";
    259+    }
    260+    return log_msg;
    


    l0rinc commented at 3:02 pm on September 19, 2024:

    Since we have simple, atomic instructions in both branches after which we immediately return, we might as well return directly:

    0    try {
    1        return tfm::format(fmt.fmt, args...);
    2    } catch (tinyformat::format_error& fmterr) {
    3        /* Original format string will have newline so don't add one here */
    4        return "Error \""s + fmterr.what() + "\" while formatting log message: \"" + fmt.fmt + "\"";
    5    }
    

    Note: I’ve used using namespace std::string_literals to simplify "Error \""s + fmterr.what() concat here.


    stickies-v commented at 3:33 pm on September 19, 2024:
    Taken (minus string literals), thanks!
  8. in src/netbase.cpp:562 in ed4c60e38b outdated
    563-    if (manual_connection) {
    564-        LogPrintf("%s\n", error_message);
    565-    } else {
    566-        LogDebug(BCLog::NET, "%s\n", error_message);
    567-    }
    568+    auto log_level{manual_connection ? BCLog::Level::Info : BCLog::Level::Debug};
    


    maflcko commented at 3:06 pm on September 19, 2024:
    Not sure about the first commit. One is unconditionally logging and the other not. That is a pretty stark difference and would be better to leave on separate lines. In any case, this isn’t a refactor, because you drop the trailing \n, and it isn’t needed for the changes here, so it may be best to drop the first commit.

    stickies-v commented at 3:33 pm on September 19, 2024:
    Thanks for pointing out the behaviour change, I’ve dropped the commit.
  9. in src/netbase.cpp:563 in ed4c60e38b outdated
    564-        LogPrintf("%s\n", error_message);
    565-    } else {
    566-        LogDebug(BCLog::NET, "%s\n", error_message);
    567-    }
    568+    auto log_level{manual_connection ? BCLog::Level::Info : BCLog::Level::Debug};
    569+    LogPrintLevel(BCLog::NET, log_level, fmt, args...);
    


    l0rinc commented at 3:07 pm on September 19, 2024:
    Is the LogFlags::NONE -> BCLog::NET category in case of manual_connection a behavior change?

    stickies-v commented at 3:34 pm on September 19, 2024:
    It is indeed, thanks for pointing it out. Resolved by dropping the first commit and applying a minimal diff to netbase.cpp.
  10. in src/test/util_string_tests.cpp:91 in bb9b55f51c outdated
    83@@ -83,4 +84,12 @@ BOOST_AUTO_TEST_CASE(ConstevalFormatString_NumSpec)
    84     FailFmtWithError<1>("%1$", err_term);
    85 }
    86 
    87+BOOST_AUTO_TEST_CASE(tinyformat_try_format)
    88+{
    89+    // ensure invalid format strings don't throw at run-time
    


    l0rinc commented at 3:08 pm on September 19, 2024:
    nit: comment seems redundant here

    stickies-v commented at 3:37 pm on September 19, 2024:

    I don’t agree it’s redundant:

    • it’s not obvious from the tests that we’re not just checking error messages, but that we’re ensuring no errors are thrown, which is the difference with tfm::format()
    • additional tests to try_format may be added in the future that test other aspects, so this acts as a little delineator

    l0rinc commented at 5:54 pm on September 19, 2024:
    thanks for clarifying
  11. stickies-v force-pushed on Sep 19, 2024
  12. in src/util/string.h:249 in 6dc6e4583c outdated
    245+ * to handle the error.
    246+ */
    247 namespace tinyformat {
    248 template <typename... Args>
    249-std::string format(util::ConstevalFormatString<sizeof...(Args)> fmt, const Args&... args)
    250+std::string try_format(util::ConstevalFormatString<sizeof...(Args)> fmt, const Args&... args)
    


    maflcko commented at 3:37 pm on September 19, 2024:

    Not sure about this. tfm::format is the only remaining function preventing the removal of ./test/lint/lint-format-strings.py. I wanted to do this in a follow-up, but if this function is renamed, it will be harder, because all call sites will have to be renamed.

    My preference would be to leave the name as-is and instead just delete the unchecked one, or rename that one to format_maybe_throw (or similar).

    If you don’t mind, it would be good to delete ./test/lint/lint-format-strings.py in this pull request. Otherwise there will need to be another follow-up, or a possibly conflicting pull.


    stickies-v commented at 1:03 pm on September 20, 2024:

    I renamed it to try_format to be clear about the different behaviour compared to tfm::format, but agreed that avoiding merge conflicts is useful to coordinate on.

    My preference would be to leave the name as-is and instead just delete the unchecked one

    Could you elaborate on this a bit more? We have these 4 other format overloads, I don’t think you want to just remove them all? At least 2) and 4) would imply touching quite a bit of code to refactor, and it seems not all call sites of format are currently constant expressions?

    1. std::string format(const std::string &fmt, const Args&... args)
    2. std::string format(const char* fmt, const Args&... args)
    3. void format(std::ostream& out, const char* fmt, const Args&... args)
    4. bilingual_str format(const bilingual_str& fmt, const Args&... args)

    maflcko commented at 1:20 pm on September 20, 2024:

    IIUC the compiler will lookup format(const char*, ...) for format("%s, 1), so format(const char*, ...) would have to be deleted for the compiler to pick format(ConstevalFormatString, ...). (My recommendation for this pull would be to do at least this)

    IIUC the remaining format taking a std::string can be left as-is for now, as they are less common anyway than the ones taking a compile-time string literal. (Possibly something can be done for them as well, but it seems fine to leave them for a follow-up)

    If there is a need for the throwing format, it seems less churn to add a throwing_format than to switch everything else to try_format.

    But maybe I am missing something obvious?


    stickies-v commented at 1:49 pm on September 20, 2024:

    so format(const char*, ...) would have to be deleted for the compiler to pick format(ConstevalFormatString, ...)

    I think just deleting format(const char*, ...) leads to ambiguous overloading?

     0../../../../src/checkqueue.h:136:36: error: call to 'format' is ambiguous
     1                util::ThreadRename(strprintf("scriptch.%i", n));
     2                                   ^~~~~~~~~
     3../../../../src/tinyformat.h:1151:19: note: expanded from macro 'strprintf'
     4#define strprintf tfm::format
     5                  ^~~~~~~~~~~
     6../../../../src/tinyformat.h:1140:13: note: candidate function [with Args = <int>]
     7std::string format(const std::string &fmt, const Args&... args)
     8            ^
     9../../../../src/util/string.h:249:13: note: candidate function [with Args = <int>]
    10std::string format(util::ConstevalFormatString<sizeof...(Args)> fmt, const Args&... args)
    

    IIUC the remaining format taking a std::string can be left as-is for now

    That (and keeping the bilingual_str overload) would be my preference too, but you also asked to remove ./test/lint/lint-format-strings.py in this pull which I think we shouldn’t do before all tfm::format overloads are compile-time checked? So that’s why I’m a bit confused.


    maflcko commented at 1:57 pm on September 20, 2024:

    I think just deleting format(const char*, ...) leads to ambiguous overloading?

    Thanks for actually checking. I guess this means that the std::string overload would have to be renamed to avoid the overload problem. Hopefully only the fuzz tests and a few lines in real code use the string overload?

    you also asked to remove ./test/lint/lint-format-strings.py in this pull which I think we shouldn’t do before all tfm::format overloads are compile-time checked? So that’s why I’m a bit confused.

    Jup, something would have to be done to the std::string overloads. It should be possible and I can take a look at this later, however it hopefully shouldn’t be a blocker for this pull request either way, if the (temporary?) rename isn’t too involved. I just have the feeling that it is easier to rename the one taking std::string, than the one taking the consteval literal.


    maflcko commented at 4:47 pm on September 24, 2024:

    Hmm, so I took a look here and the following diff seems to compile (almost):

     0diff --git a/src/tinyformat.h b/src/tinyformat.h
     1index f536306375..079b0ccbff 100644
     2--- a/src/tinyformat.h
     3+++ b/src/tinyformat.h
     4@@ -142,6 +142,7 @@ namespace tfm = tinyformat;
     5 //------------------------------------------------------------------------------
     6 // Implementation details.
     7 #include <algorithm>
     8+#include <concepts> // Added for Bitcoin Core
     9 #include <iostream>
    10 #include <sstream>
    11 #include <stdexcept> // Added for Bitcoin Core
    12@@ -1056,7 +1057,7 @@ inline void vformat(std::ostream& out, const char* fmt, FormatListRef list)
    13 
    14 /// Format list of arguments to the stream according to given format string.
    15 template<typename... Args>
    16-void format(std::ostream& out, const char* fmt, const Args&... args)
    17+void format_raw(std::ostream& out, const char* fmt, const Args&... args) // Renamed for Bitcoin Core
    18 {
    19     vformat(out, fmt, makeFormatList(args...));
    20 }
    21@@ -1064,24 +1065,24 @@ void format(std::ostream& out, const char* fmt, const Args&... args)
    22 /// Format list of arguments according to the given format string and return
    23 /// the result as a string.
    24 template<typename... Args>
    25-std::string format(const char* fmt, const Args&... args)
    26+std::string format_raw(const char* fmt, const Args&... args) // Renamed for Bitcoin Core
    27 {
    28     std::ostringstream oss;
    29-    format(oss, fmt, args...);
    30+    format_raw(oss, fmt, args...);
    31     return oss.str();
    32 }
    33 
    34 /// Format list of arguments to std::cout, according to the given format string
    35 template<typename... Args>
    36-void printf(const char* fmt, const Args&... args)
    37+void printf_raw(const char* fmt, const Args&... args) // Renamed for Bitcoin Core
    38 {
    39-    format(std::cout, fmt, args...);
    40+    format_raw(std::cout, fmt, args...);
    41 }
    42 
    43 template<typename... Args>
    44-void printfln(const char* fmt, const Args&... args)
    45+void printfln_raw(const char* fmt, const Args&... args) // Renamed for Bitcoin Core
    46 {
    47-    format(std::cout, fmt, args...);
    48+    format_raw(std::cout, fmt, args...);
    49     std::cout << '\n';
    50 }
    51 
    52@@ -1146,11 +1147,12 @@ TINYFORMAT_FOREACH_ARGNUM(TINYFORMAT_MAKE_FORMAT_FUNCS)
    53 #endif
    54 
    55 // Added for Bitcoin Core
    56-template<typename... Args>
    57-std::string format(const std::string &fmt, const Args&... args)
    58+template <typename String, typename... Args>
    59+    requires std::same_as<String, std::string>
    60+std::string format(const String& fmt, const Args&... args)
    61 {
    62     std::ostringstream oss;
    63-    format(oss, fmt.c_str(), args...);
    64+    format_raw(oss, fmt.c_str(), args...);
    65     return oss.str();
    66 }
    67 
    68@@ -1160,4 +1162,6 @@ std::string format(const std::string &fmt, const Args&... args)
    69 /** Format arguments and return the string or write to given std::ostream (see tinyformat::format doc for details) */
    70 #define strprintf tfm::format
    71 
    72+#include <util/string.h> // Added for Bitcoin Core
    73+
    74 #endif // TINYFORMAT_H_INCLUDED
    75diff --git a/src/util/string.h b/src/util/string.h
    76index c5183d6c80..5a9b55602f 100644
    77--- a/src/util/string.h
    78+++ b/src/util/string.h
    79@@ -239,7 +239,12 @@ namespace tinyformat {
    80 template <typename... Args>
    81 std::string format(util::ConstevalFormatString<sizeof...(Args)> fmt, const Args&... args)
    82 {
    83-    return format(fmt.fmt, args...);
    84+    return tfm::format_raw(fmt.fmt, args...);
    85+}
    86+template <typename... Args>
    87+void format(std::ostream& out, util::ConstevalFormatString<sizeof...(Args)> fmt, const Args&... args)
    88+{
    89+    tfm::format_raw(out, fmt.fmt, args...);
    90 }
    91 } // namespace tinyformat
    92 
    

    The problem is that %*s are actually used in bitcoin-cli. However, in an unspecified form (%*s can not be used after using positional specifiers). I am not sure if it is worth it to change the format strings, or to adjust the consteval parser. So an easy fix would be to just call the unchecked c-string functions in bitcoin-cli, which I think is fine.


    stickies-v commented at 1:51 pm on September 25, 2024:

    Thanks for looking into this further! I was also exploring the concepts route, but hadn’t yet resolved all concerns.

    1. Re naming: having format overloads with significantly differently behaviour (throwing vs non-throwing) didn’t quite sit right with me, so I think your format_raw (or format_throw) suggestion is preferable, and I agree it makes more sense to rename the throwing variant instead of the non-throwing variant (although the downside is that it makes reviewing this PR less straightforward).
    2. forcing the util::ConstevalFormatString overload for all string literals either requires that: a) all callsites include util/string.h b) tinyformat.h includes util/string.h (as in your suggestion, although we should probably iwyu export it too), but this introduces a circular dependency c) the util::ConstevalFormatString overloads are moved to tinyformat.h, so the tinyformat.h dependency in util/string.h can be removed.

    I think c) is the preferable approach. It adds more core-specific LoC to tinyformat but I think it is the cleanest overall?

    So an easy fix would be to just call the unchecked c-string functions in bitcoin-cli, which I think is fine.

    I agree, although I think I’ll clean up the incorrect bitcoin-cli format strings here too since that seems like an easy win.


    maflcko commented at 2:13 pm on September 25, 2024:

    I think c) is the preferable approach. It adds more core-specific LoC to tinyformat but I think it is the cleanest overall?

    Yeah, probably. Maybe a large header // === Everything below was added for Bitcoin Core === in the end and then appending the Bitcoin Core stuff would be cleaner than inlining every function into the existing header with a comment by itself.


    stickies-v commented at 12:13 pm on September 26, 2024:
    Latest force-push incorporates what we’ve discussed here, and goes one step further by removing the const std::string& overload entirely. I inspected the callsites and none of them seemed necessary to have the overload (see updated PR description), so this felt like the better way forward.
  13. DrahtBot removed the label CI failed on Sep 19, 2024
  14. l0rinc approved
  15. l0rinc commented at 5:55 pm on September 19, 2024: contributor
    utACK 6dc6e4583c02f0d21dffeb6b2c81358b49a1f340
  16. move-only: ConstevalFormatString tfm::format to tinyformat.h
    In future commits, the non-ConstevalFormatString tfm::format
    overloads will be renamed or removed. To avoid all callsites
    having to include util/string.h (or to have a circular dependency
    between tinyformat.h and util/string.h), just move the
    ConstevalFormatString overloads to tinyformat.h
    1f2d0ffd56
  17. tinyformat: remove std::string overload
    All (non-test) format strings are known at compile-time, so we
    don't need the std::string overload. Remove it so users use the
    safer util::ConstevalFormatString where possible, or the
    bilingual_str overload where applicable.
    9b05c64dd0
  18. stickies-v force-pushed on Sep 26, 2024
  19. stickies-v force-pushed on Sep 26, 2024
  20. DrahtBot commented at 12:06 pm on September 26, 2024: contributor

    🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/30701048188

    Make sure to run all tests locally, according to the documentation.

    The failure may happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

  21. DrahtBot added the label CI failed on Sep 26, 2024
  22. stickies-v renamed this:
    util: refactor: add and use run-time safe tinyformat::try_format
    tinyformat: refactor: increase compile-time checks and don't throw for tfm::format_error
    on Sep 26, 2024
  23. stickies-v commented at 12:16 pm on September 26, 2024: contributor

    Latest force-push increases the scope of this PR, see updated PR description for an overview. In a nutshell:

    • all non-bilingual_str format/strprintf usage is now compile-time checked. Format errors are returned as strings instead of thrown as error, but throwing behaviour can be maintained through tfm::format_raw.
    • the const std::string& format overload is removed because it’s unnecessary

    Thanks a lot @maflcko and @l0rinc for your review!

  24. in src/tinyformat.h:1174 in ed57f6ee02 outdated
    1171 {
    1172-    return format_raw(out, fmt.fmt, args...);
    1173+    try {
    1174+        tfm::format_raw(out, fmt.fmt, args...);
    1175+    } catch (tinyformat::format_error& fmterr) {
    1176+        out << "Error \"" + std::string{fmterr.what()} + "\" while formatting log message: \"" + fmt.fmt + "\"";
    


    maflcko commented at 12:21 pm on September 26, 2024:

    nit in ed57f6ee02d5a6ada23abaea6e2176cd4f173e4f: Would be nice to do it in tinyformat itself, or not at all. ConstevalFormatString is already checked, so extremely unlikely to hit. However, translated strings (which are apparently unreviewed, see #30897) are left unchecked and may throw.

    It would be good to validate and review the translated strings, or remove translations completely for now, but this is probably a separate issue.

    Here, it would be good to apply the try-catch to them.


    stickies-v commented at 10:51 am on September 27, 2024:

    I assumed the next step would be to validate the bilingual_str overload at compile-time too, but I’ve been looking at implementing that and with the run-time translation dependency (QCoreApplication::translate) it doesn’t really seem feasible.

    I hadn’t considered that translations can introduce format errors too, which makes the case for bilingual_str format errors not throwing ever stronger indeed. I was worried that blanket catching all formatting errors at the tfm level could be considered controversial so went for a throwing format_raw and non-throwing format, but perhaps just never having throwing behaviour makes for a more consistent and safer interface, so I’ll move the try/catch to tfm::vformat now.

  25. maflcko approved
  26. maflcko commented at 12:22 pm on September 26, 2024: member

    lgtm

    Thanks for taking the feedback

  27. DrahtBot removed the label CI failed on Sep 26, 2024
  28. stickies-v force-pushed on Sep 27, 2024
  29. stickies-v commented at 11:18 am on September 27, 2024: contributor
    Force-pushed to address @maflcko’s comment about generally not letting tinyformat throw format errors, instead of just for the util::ConstevalFormatString overloads. Most notably, this changes behaviour for the bilingual_str format overload to also not throw.
  30. maflcko approved
  31. maflcko commented at 11:28 am on September 27, 2024: member
    lgtm
  32. in src/wallet/test/fuzz/notifications.cpp:67 in 5d69f2d33b outdated
    63@@ -64,7 +64,7 @@ void ImportDescriptors(CWallet& wallet, const std::string& seed_insecure)
    64 
    65     for (const std::string& desc_fmt : DESCS) {
    66         for (bool internal : {true, false}) {
    67-            const auto descriptor{(strprintf)(desc_fmt.c_str(), "[5aa9973a/66h/4h/2h]" + seed_insecure, int{internal})};
    68+            const auto descriptor{(tfm::format_raw)(desc_fmt.c_str(), "[5aa9973a/66h/4h/2h]" + seed_insecure, int{internal})};
    


    maflcko commented at 1:01 pm on September 27, 2024:
    nit in 5d69f2d33b64a025d0b93b81558f2f51bd54b104: I think the () are no longer needed to silence the linter and can be removed: {tfm::format_raw(desc_fmt.c_str(), ...}

    stickies-v commented at 2:49 pm on September 27, 2024:
    Ah, I wasn’t sure why that was. Fuzzer compiles and linter seems happy so updated, thanks!
  33. maflcko approved
  34. maflcko commented at 1:07 pm on September 27, 2024: member

    review ACK bb12305737e3c22325930b5bdb04028f1a6f2ef6 💅

    Signature:

    0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    2trusted comment: review ACK bb12305737e3c22325930b5bdb04028f1a6f2ef6 💅
    3Fge/JMP/cEBpCNfrVWYhr7mIRY4zXGiZRkBzWtbVouQ4q6WPnKlif9AdGqIb4Jrmkp/sIcvaECTG9QyCy1/mBA==
    
  35. DrahtBot requested review from l0rinc on Sep 27, 2024
  36. tinyformat: force compile-time checks for format string literals
    Callsites such as bitcoin-cli that (incorrectly) fail compile-time
    validation can still use tfm::format_raw.
    352904c6a6
  37. tinyformat: don't throw format string errors
    In almost all cases, invalid format strings should not lead to crashing
    behaviour, so suppress run-time format errors by returning the error as
    a string instead of throwing a tinyformat::format_error.
    
    Note: most format string literals are partially validated at compile-time
    through util::ConstevalFormatString. Notably, bilingual_str format strings
    do not have this compile-time validation.
    2d4f82d590
  38. stickies-v force-pushed on Sep 27, 2024
  39. in doc/developer-notes.md:955 in 2d4f82d590
    949@@ -950,7 +950,9 @@ Strings and formatting
    950 
    951   - Do not use it when passing strings to `tfm::format`, `strprintf`, `LogInfo`, `LogDebug`, etc.
    952 
    953-    - *Rationale*: This is redundant. Tinyformat handles strings.
    954+    - *Rationale*: Tinyformat handles string parameters. Format strings
    955+      should be known at compile-time, so using a string literal or
    956+      `constexpr const char*` allows for compile-time validation.
    


    l0rinc commented at 2:55 pm on September 27, 2024:

    nit: we could avoid mentioning compile-time twice:

    0    - *Rationale*: Tinyformat handles string parameters.
    1      Using a string literal or `constexpr const char*` ensures
    2      format strings are validated at compile time.
    
  40. in src/test/fuzz/strprintf.cpp:49 in 2d4f82d590
    47@@ -48,35 +48,32 @@ FUZZ_TARGET(str_printf)
    48     //
    49     // Upstream bug report: https://github.com/c42f/tinyformat/issues/70
    


    l0rinc commented at 2:58 pm on September 27, 2024:
    is this bug still possible now? If it is, consider updating the strprintf example in the comment
  41. in src/clientversion.cpp:90 in 9b05c64dd0 outdated
    86@@ -87,15 +87,17 @@ std::string LicenseInfo()
    87 {
    88     const std::string URL_SOURCE_CODE = "<https://github.com/bitcoin/bitcoin>";
    89 
    90-    return CopyrightHolders(strprintf(_("Copyright (C) %i-%i").translated, 2009, COPYRIGHT_YEAR) + " ") + "\n" +
    91+    return CopyrightHolders(strprintf(_("Copyright (C) %i-%i"), 2009, COPYRIGHT_YEAR).translated + " ") + "\n" +
    


    l0rinc commented at 3:05 pm on September 27, 2024:

    I’m not sure I understand how this change is supposed to work.

    "Copyright (C) %i-%i" can be translated by checking the exact string from https://github.com/bitcoin/bitcoin/blob/master/src/qt/bitcoinstrings.cpp#L280, but how is "Copyright (C) 2009-2024" supposed to have a translated counterpart?


    stickies-v commented at 5:27 pm on September 27, 2024:
    "Copyright (C) %i-%i" remains the translateable string both before and after this change. The only difference is that we use the bilingual_str format overload instead of the const std::string& one.

    l0rinc commented at 3:39 pm on September 28, 2024:
    Got it, thanks for clarifying!
  42. in src/test/txrequest_tests.cpp:181 in 9b05c64dd0 outdated
    177@@ -178,9 +178,9 @@ class Scenario
    178             size_t real_total = runner.txrequest.Count(peer);
    179             size_t real_candidates = runner.txrequest.CountCandidates(peer);
    180             size_t real_inflight = runner.txrequest.CountInFlight(peer);
    181-            BOOST_CHECK_MESSAGE(real_total == total, strprintf("[" + comment + "] total %i (%i expected)", real_total, total));
    


    l0rinc commented at 3:06 pm on September 27, 2024:
    lol
  43. in src/tinyformat.h:148 in 1f2d0ffd56 outdated
    144@@ -145,6 +145,7 @@ namespace tfm = tinyformat;
    145 #include <iostream>
    146 #include <sstream>
    147 #include <stdexcept> // Added for Bitcoin Core
    148+#include <util/string.h> // Added for Bitcoin Core
    


    l0rinc commented at 3:10 pm on September 27, 2024:
    👍, should have been there all along
  44. in src/test/txrequest_tests.cpp:184 in 9b05c64dd0 outdated
    182-            BOOST_CHECK_MESSAGE(real_inflight == inflight, strprintf("[" + comment + "] inflight %i (%i expected)", real_inflight, inflight));
    183-            BOOST_CHECK_MESSAGE(real_candidates == candidates, strprintf("[" + comment + "] candidates %i (%i expected)", real_candidates, candidates));
    184+            BOOST_CHECK_MESSAGE(real_total == total, strprintf("[%s] total %i (%i expected)", comment, real_total, total));
    185+            BOOST_CHECK_MESSAGE(real_inflight == inflight, strprintf("[%s] inflight %i (%i expected)", comment, real_inflight, inflight));
    186+            BOOST_CHECK_MESSAGE(real_candidates == candidates, strprintf("[%s] candidates %i (%i expected)", comment, real_candidates, candidates));
    187             BOOST_CHECK_MESSAGE(ret == expected, "[" + comment + "] mismatching requestables");
    


    l0rinc commented at 3:14 pm on September 27, 2024:

    for consistency, consider using strprintf here as well:

    0            BOOST_CHECK_MESSAGE(ret == expected, strprintf("[%s] mismatching requestables", comment");
    

    Would be cool if we could use BOOST_CHECK_EQUAL_COLLECTIONS instead, but may not be trivial to do, haven’t investigated in detail.

  45. in src/test/util_string_tests.cpp:89 in 2d4f82d590
    85@@ -83,4 +86,15 @@ BOOST_AUTO_TEST_CASE(ConstevalFormatString_NumSpec)
    86     FailFmtWithError<1>("%1$", err_term);
    87 }
    88 
    89+BOOST_AUTO_TEST_CASE(tinyformat_ConstevalFormatString)
    


    l0rinc commented at 4:05 pm on September 27, 2024:
    Unrelated: just realized that we should eventually add a test for "%1", since while ('0' <= *it && *it <= '9') { can go beyond the string and passes accidentally - but seems to fail in tinyformat, see: https://godbolt.org/z/nbTKnrqch. Added it in the patch, feel free to use it here, not worth adding a new PR for it
  46. in src/bitcoin-cli.cpp:543 in 2d4f82d590
    539@@ -540,15 +540,15 @@ class NetinfoRequestHandler : public BaseRequestHandler
    540         // Report detailed peer connections list sorted by direction and minimum ping time.
    541         if (DetailsRequested() && !m_peers.empty()) {
    542             std::sort(m_peers.begin(), m_peers.end());
    543-            result += strprintf("<->   type   net  v  mping   ping send recv  txn  blk  hb %*s%*s%*s ",
    544-                                m_max_addr_processed_length, "addrp",
    545-                                m_max_addr_rate_limited_length, "addrl",
    546-                                m_max_age_length, "age");
    547+            result += tfm::format_raw("<->   type   net  v  mping   ping send recv  txn  blk  hb %*s%*s%*s ",
    


    l0rinc commented at 4:29 pm on September 27, 2024:

    we could avoid formatting these by adding something like:

    0if (*it == '*' || ((*it == '_' || *it == '-') && *std::next(it) == '*')) {
    1    return; // We can't validate format string with %* or %_
    2}
    

    to Detail_CheckNumFormatSpecifiers , which would enable us using tfm::format here - maybe later we’ll implement full validation, but this way at least everything before the * format is validated properly.


    maflcko commented at 2:25 pm on September 28, 2024:
    I am not sure about “partial validation”. Either the string is fully validated, or not at all. Prepending a valid string to an invalid string doesn’t make it valid.

    l0rinc commented at 3:06 pm on September 28, 2024:

    Weren’t we already doing partial validation? Doing a full one for these examples wouldn’t be that difficult - do you suggest we do that instead?

    e.g. something like:

    0if (*it == '*' || ((*it == '_' || *it == '-') && *std::next(it) == '*')) {
    1    ++count_normal; // width
    2    ++count_normal; // value
    3    if (*it != '*') ++it;
    4    ++it;
    5    continue;
    6}
    

    l0rinc commented at 4:08 pm on September 28, 2024:

    Alternatively, if that’s not desired, what if we replaced the dynamic widths with explicit PadLeft/PadRight calls, e.g.

    0result += tfm::format("<->   type   net  v  mping   ping send recv  txn  blk  hb %s%s%s ",
    1                      PadRight("addrp", m_max_addr_processed_length),
    2                      PadRight("addrl", m_max_addr_rate_limited_length),
    3                      PadRight("age", m_max_age_length));
    

    and

    0result += tfm::format("%s %s%s\n",
    1                      PadLeft("id", m_max_id_length),
    2                      IsAddressSelected() ? PadRight("address", m_max_addr_length) : "",
    3                      IsVersionSelected() ? "version" : "");
    

    etc. I can provide either of these in a follow-up, if the reviewers think it’s a good direction.

     0[[nodiscard]] inline std::string PadLeft(const std::string_view& str, size_t width, const char pad = ' ')
     1{
     2    if (str.size() >= width) return std::string(str);
     3    return std::string(width - str.size(), pad) + std::string(str);
     4}
     5
     6[[nodiscard]] inline std::string PadRight(const std::string_view& str, size_t width, const char pad = ' ')
     7{
     8    if (str.size() >= width) return std::string(str);
     9    return std::string(str) + std::string(width - str.size(), pad);
    10}
    
  47. in src/test/util_string_tests.cpp:91 in 2d4f82d590
    85@@ -83,4 +86,15 @@ BOOST_AUTO_TEST_CASE(ConstevalFormatString_NumSpec)
    86     FailFmtWithError<1>("%1$", err_term);
    87 }
    88 
    89+BOOST_AUTO_TEST_CASE(tinyformat_ConstevalFormatString)
    90+{
    91+    // ensure invalid format strings don't throw at run-time
    


    l0rinc commented at 4:29 pm on September 27, 2024:

    nit:

    0    // Ensure invalid format strings don't throw at run-time
    
  48. in src/test/util_string_tests.cpp:88 in 2d4f82d590
    85@@ -83,4 +86,15 @@ BOOST_AUTO_TEST_CASE(ConstevalFormatString_NumSpec)
    86     FailFmtWithError<1>("%1$", err_term);
    87 }
    88 
    


    l0rinc commented at 4:30 pm on September 27, 2024:

    We could add the examples from bitcoin-cli here:

    0BOOST_AUTO_TEST_CASE(ConstevalFormatString_SpecificUsageTests)
    1{
    2    PassFmt<6>("<->   type   net  v  mping   ping send recv  txn  blk  hb %*s%*s%*s ");
    3    PassFmt<5>("%*s %-*s%s\n");
    4    PassFmt<24>("%3s %6s %5s %2s%7s%7s%5s%5s%5s%5s  %2s %*s%*s%*s%*i %*s %-*s%s\n");
    5    PassFmt<2>("                        ms     ms  sec  sec  min  min                %*s\n\n");
    6    PassFmt<4>("\n%-*s    port %6i    score %6i");
    7    PassFmt<3>("%*s %s\n");
    8}
    
  49. l0rinc changes_requested
  50. l0rinc commented at 4:37 pm on September 27, 2024: contributor

    I have concerns about the translations, using raw format instead of adjusting the validator, and I think we could sneak in a fix for unterminated positionals.

      0Index: src/util/string.h
      1<+>UTF-8
      2===================================================================
      3diff --git a/src/util/string.h b/src/util/string.h
      4--- a/src/util/string.h	(revision 2d4f82d590845f9b2ea11bbcfe9377ef14df8775)
      5+++ b/src/util/string.h	(date 1727560531914)
      6@@ -51,12 +51,19 @@
      7                 ++it;
      8                 continue;
      9             }
     10+            if (*it == '*' || ((*it == '_' || *it == '-') && *std::next(it) == '*')) {
     11+                ++count_normal; // width
     12+                ++count_normal; // value
     13+                if (*it != '*') ++it;
     14+                ++it;
     15+                continue;
     16+            }
     17 
     18             unsigned maybe_num{0};
     19             while ('0' <= *it && *it <= '9') {
     20                 maybe_num *= 10;
     21                 maybe_num += *it - '0';
     22-                ++it;
     23+                if (++it >= str.end()) throw "Format specifier incorrectly terminated by end of string";
     24             };
     25 
     26             if (*it == '$') {
     27Index: src/test/util_string_tests.cpp
     28<+>UTF-8
     29===================================================================
     30diff --git a/src/test/util_string_tests.cpp b/src/test/util_string_tests.cpp
     31--- a/src/test/util_string_tests.cpp	(revision 2d4f82d590845f9b2ea11bbcfe9377ef14df8775)
     32+++ b/src/test/util_string_tests.cpp	(date 1727560389361)
     33@@ -61,7 +61,6 @@
     34     // The `*` specifier behavior is unsupported and can lead to runtime
     35     // errors when used in a ConstevalFormatString. Please refer to the
     36     // note in the ConstevalFormatString docs.
     37-    PassFmt<1>("%*c");
     38     PassFmt<2>("%2$*3$d");
     39     PassFmt<1>("%.*f");
     40 
     41@@ -83,18 +82,27 @@
     42 
     43     auto err_term{"Format specifier incorrectly terminated by end of string"};
     44     FailFmtWithError<1>("%", err_term);
     45+    FailFmtWithError<1>("%1", err_term);
     46     FailFmtWithError<1>("%1$", err_term);
     47 }
     48 
     49+BOOST_AUTO_TEST_CASE(ConstevalFormatString_SpecificUsageTests)
     50+{
     51+    PassFmt<6>("<->   type   net  v  mping   ping send recv  txn  blk  hb %*s%*s%*s ");
     52+    PassFmt<5>("%*s %-*s%s\n");
     53+    PassFmt<24>( "%3s %6s %5s %2s%7s%7s%5s%5s%5s%5s  %2s %*s%*s%*s%*i %*s %-*s%s\n");
     54+    PassFmt<2>("                        ms     ms  sec  sec  min  min                %*s\n\n");
     55+    PassFmt<4>("\n%-*s    port %6i    score %6i");
     56+    PassFmt<3>("%*s %s\n");
     57+}
     58+
     59 BOOST_AUTO_TEST_CASE(tinyformat_ConstevalFormatString)
     60 {
     61-    // ensure invalid format strings don't throw at run-time
     62-    BOOST_CHECK_EQUAL(tfm::format("%*c", "dummy"), R"(Error "tinyformat: Cannot convert from argument type to integer for use as variable width or precision" while formatting log message: "%*c")");
     63+    // Ensure invalid format strings don't throw at run-time
     64     BOOST_CHECK_EQUAL(tfm::format("%2$*3$d", "dummy", "value"), R"(Error "tinyformat: Positional argument out of range" while formatting log message: "%2$*3$d")");
     65     std::ostringstream oss;
     66     tfm::format(oss, "%.*f", 5);
     67     BOOST_CHECK_EQUAL(oss.str(), R"(Error "tinyformat: Too many conversion specifiers in format string" while formatting log message: "%.*f")");
     68 }
     69 
     70-
     71 BOOST_AUTO_TEST_SUITE_END()
     72Index: src/bitcoin-cli.cpp
     73<+>UTF-8
     74===================================================================
     75diff --git a/src/bitcoin-cli.cpp b/src/bitcoin-cli.cpp
     76--- a/src/bitcoin-cli.cpp	(revision 2d4f82d590845f9b2ea11bbcfe9377ef14df8775)
     77+++ b/src/bitcoin-cli.cpp	(date 1727560250888)
     78@@ -540,42 +540,46 @@
     79         // Report detailed peer connections list sorted by direction and minimum ping time.
     80         if (DetailsRequested() && !m_peers.empty()) {
     81             std::sort(m_peers.begin(), m_peers.end());
     82-            result += tfm::format_raw("<->   type   net  v  mping   ping send recv  txn  blk  hb %*s%*s%*s ",
     83-                                      m_max_addr_processed_length, "addrp",
     84-                                      m_max_addr_rate_limited_length, "addrl",
     85-                                      m_max_age_length, "age");
     86+            result += tfm::format("<->   type   net  v  mping   ping send recv  txn  blk  hb %*s%*s%*s ",
     87+                                  m_max_addr_processed_length, "addrp",
     88+                                  m_max_addr_rate_limited_length, "addrl",
     89+                                  m_max_age_length, "age");
     90             if (m_is_asmap_on) result += " asmap ";
     91-            result += tfm::format_raw("%*s %-*s%s\n", m_max_id_length, "id", IsAddressSelected() ? m_max_addr_length : 0, IsAddressSelected() ? "address" : "", IsVersionSelected() ? "version" : "");
     92+            result += tfm::format("%*s %-*s%s\n",
     93+                                  m_max_id_length, "id",
     94+                                  IsAddressSelected() ? m_max_addr_length : 0,
     95+                                  IsAddressSelected() ? "address" : "",
     96+                                  IsVersionSelected() ? "version" : "");
     97             for (const Peer& peer : m_peers) {
     98                 std::string version{ToString(peer.version) + peer.sub_version};
     99-                result += tfm::format_raw(
    100-                    "%3s %6s %5s %2s%7s%7s%5s%5s%5s%5s  %2s %*s%*s%*s%*i %*s %-*s%s\n",
    101-                    peer.is_outbound ? "out" : "in",
    102-                    ConnectionTypeForNetinfo(peer.conn_type),
    103-                    peer.network,
    104-                    (peer.transport_protocol_type.size() == 2 && peer.transport_protocol_type[0] == 'v') ? peer.transport_protocol_type[1] : ' ',
    105-                    PingTimeToString(peer.min_ping),
    106-                    PingTimeToString(peer.ping),
    107-                    peer.last_send ? ToString(time_now - peer.last_send) : "",
    108-                    peer.last_recv ? ToString(time_now - peer.last_recv) : "",
    109-                    peer.last_trxn ? ToString((time_now - peer.last_trxn) / 60) : peer.is_tx_relay ? "" : "*",
    110-                    peer.last_blck ? ToString((time_now - peer.last_blck) / 60) : "",
    111-                    strprintf("%s%s", peer.is_bip152_hb_to ? "." : " ", peer.is_bip152_hb_from ? "*" : " "),
    112-                    m_max_addr_processed_length, // variable spacing
    113-                    peer.addr_processed ? ToString(peer.addr_processed) : peer.is_addr_relay_enabled ? "" : ".",
    114-                    m_max_addr_rate_limited_length, // variable spacing
    115-                    peer.addr_rate_limited ? ToString(peer.addr_rate_limited) : "",
    116-                    m_max_age_length, // variable spacing
    117-                    peer.age,
    118-                    m_is_asmap_on ? 7 : 0, // variable spacing
    119-                    m_is_asmap_on && peer.mapped_as ? ToString(peer.mapped_as) : "",
    120-                    m_max_id_length, // variable spacing
    121-                    peer.id,
    122-                    IsAddressSelected() ? m_max_addr_length : 0, // variable spacing
    123-                    IsAddressSelected() ? peer.addr : "",
    124-                    IsVersionSelected() && version != "0" ? version : "");
    125+                result += tfm::format("%3s %6s %5s %2s%7s%7s%5s%5s%5s%5s  %2s %*s%*s%*s%*i %*s %-*s%s\n",
    126+                                      peer.is_outbound ? "out" : "in",
    127+                                      ConnectionTypeForNetinfo(peer.conn_type),
    128+                                      peer.network,
    129+                                      (peer.transport_protocol_type.size() == 2 && peer.transport_protocol_type[0] == 'v') ? peer.transport_protocol_type[1] : ' ',
    130+                                      PingTimeToString(peer.min_ping),
    131+                                      PingTimeToString(peer.ping),
    132+                                      peer.last_send ? ToString(time_now - peer.last_send) : "",
    133+                                      peer.last_recv ? ToString(time_now - peer.last_recv) : "",
    134+                                      peer.last_trxn ? ToString((time_now - peer.last_trxn) / 60) : peer.is_tx_relay ? "" : "*",
    135+                                      peer.last_blck ? ToString((time_now - peer.last_blck) / 60) : "",
    136+                                      strprintf("%s%s", peer.is_bip152_hb_to ? "." : " ", peer.is_bip152_hb_from ? "*" : " "),
    137+                                      m_max_addr_processed_length, // variable spacing
    138+                                      peer.addr_processed ? ToString(peer.addr_processed) : peer.is_addr_relay_enabled ? "" : ".",
    139+                                      m_max_addr_rate_limited_length, // variable spacing
    140+                                      peer.addr_rate_limited ? ToString(peer.addr_rate_limited) : "",
    141+                                      m_max_age_length, // variable spacing
    142+                                      peer.age,
    143+                                      m_is_asmap_on ? 7 : 0, // variable spacing
    144+                                      m_is_asmap_on && peer.mapped_as ? ToString(peer.mapped_as) : "",
    145+                                      m_max_id_length, // variable spacing
    146+                                      peer.id,
    147+                                      IsAddressSelected() ? m_max_addr_length : 0, // variable spacing
    148+                                      IsAddressSelected() ? peer.addr : "",
    149+                                      IsVersionSelected() && version != "0" ? version : "");
    150             }
    151-            result += tfm::format_raw("                        ms     ms  sec  sec  min  min                %*s\n\n", m_max_age_length, "min");
    152+            result += tfm::format("                        ms     ms  sec  sec  min  min                %*s\n\n",
    153+                                  m_max_age_length, "min");
    154         }
    155 
    156         // Report peer connection totals by type.
    157@@ -624,7 +628,10 @@
    158                 max_addr_size = std::max(addr["address"].get_str().length() + 1, max_addr_size);
    159             }
    160             for (const UniValue& addr : local_addrs) {
    161-                result += tfm::format_raw("\n%-*s    port %6i    score %6i", max_addr_size, addr["address"].get_str(), addr["port"].getInt<int>(), addr["score"].getInt<int>());
    162+                result += tfm::format("\n%-*s    port %6i    score %6i",
    163+                                      max_addr_size, addr["address"].get_str(),
    164+                                      addr["port"].getInt<int>(),
    165+                                          addr["score"].getInt<int>());
    166             }
    167         }
    168 
    169@@ -1116,10 +1123,9 @@
    170         }
    171 
    172         for (const std::string& wallet : result["balances"].getKeys()) {
    173-            result_string += tfm::format_raw("%*s %s\n",
    174-                                             max_balance_length,
    175-                                             result["balances"][wallet].getValStr(),
    176-                                             wallet.empty() ? "\"\"" : wallet);
    177+            result_string += tfm::format("%*s %s\n",
    178+                                         max_balance_length, result["balances"][wallet].getValStr(),
    179+                                         wallet.empty() ? "\"\"" : wallet);
    180         }
    181         result_string += "\n";
    182     }
    183Index: src/test/txrequest_tests.cpp
    184<+>UTF-8
    185===================================================================
    186diff --git a/src/test/txrequest_tests.cpp b/src/test/txrequest_tests.cpp
    187--- a/src/test/txrequest_tests.cpp	(revision 2d4f82d590845f9b2ea11bbcfe9377ef14df8775)
    188+++ b/src/test/txrequest_tests.cpp	(date 1727560250869)
    189@@ -181,7 +181,7 @@
    190             BOOST_CHECK_MESSAGE(real_total == total, strprintf("[%s] total %i (%i expected)", comment, real_total, total));
    191             BOOST_CHECK_MESSAGE(real_inflight == inflight, strprintf("[%s] inflight %i (%i expected)", comment, real_inflight, inflight));
    192             BOOST_CHECK_MESSAGE(real_candidates == candidates, strprintf("[%s] candidates %i (%i expected)", comment, real_candidates, candidates));
    193-            BOOST_CHECK_MESSAGE(ret == expected, "[" + comment + "] mismatching requestables");
    194+            BOOST_CHECK_MESSAGE(ret == expected, strprintf("[%s] mismatching requestables", comment));
    195         });
    196     }
    

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-09-29 01:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me