contrib: Fixup verify-binaries OS platform parsing #30147

pull BenWestgate wants to merge 1 commits into bitcoin:master from BenWestgate:30145-parse-os-version-correctly changing 3 files +32 −19
  1. BenWestgate commented at 5:18 pm on May 21, 2024: contributor

    Closes #30145.

    This PR solves two major issues in the parse_version_string function of verify-binaries:

    1. -aarch64 binaries cannot be specifically downloaded. The -platform string gets interpreted as a release candidate that doesn’t exist due to containing sub-string “rc”.
    2. Specifying a platform with a “-” in the name causes the parser to ignore both “-platform” AND “-rcN” and download the potentially wrong (non-rc) version for every platform. This also prevented specifying just one platform binary the user wished to download.

    It also updates the accompanying test.py to cover problem two and adds two examples that were formerly broken to README.md to show what is now possible. Including the most useful case of downloading only 1 specific platform’s binary.

    This improves the Bitcoin verify-binaries tools user experience by not:

    1. Failing to download for inexplicable reasons,
    2. Downloading more files than what the user told it to, or in the worst case
    3. Downloading only the wrong files.
    • A test was added to cover the command verify-binaries/verify.py pub 22.0-x86_64-linux-gnu.tar.gz which checks that bitcoin-22.0-x86_64-linux-gnu.tar.gz downloads successfully AND ONLY bitcoin-22.0-x86_64-linux-gnu.tar.gz downloads.
    • The steps to reproduce each bug are in the referenced issue #30145. Explanation of the potential issue as well as reasoning for the way the bug was fixed are in my commit descriptions.
    • This delivers the promised feature of “only download the binaries for a certain platform”, by allowing strings with ‘-’ to be accepted, allowing for single file downloads for any specific platform which was not always possible before.
    • Removes 6 lines of code from the offending parse_version_string function, while fixing the bugs/errors, and extending the functionality to be practical for users with slow connections.
    • Makes the error message more helpful when no file matches the provided platform string, now prints “Did you mean: closest-match” to help correct typos.

    Thanks for reading my PR. I look forward to getting this helpful tool in its best shape yet.

    Log of this branch passing the new test.py:

    0python3 test.py
    1✓ 'Nonexistent version should fail' passed
    2✓ 'Malformed version should fail' passed
    3✓ '--min-good-sigs 20 should fail' passed
    4- testing verification (22.0-x86_64-linux-gnu.tar.gz)
    5✓ '22.0-x86_64-linux-gnu.tar.gz should succeed' passed
    6- testing verification (22.0)
    7✓ '22.0 should succeed' passed
    

    Log of master failing the new test.py:

     0python3 test.py
     1✓ 'Nonexistent version should fail' passed
     2✓ 'Malformed version should fail' passed
     3✓ '--min-good-sigs 20 should fail' passed
     4- testing verification (22.0-x86_64-linux-gnu.tar.gz)
     5✓ '22.0-x86_64-linux-gnu.tar.gz should succeed' passed
     6Traceback (most recent call last):
     7  File "/home/ben/Documents/GitHub/bitcoin/contrib/verify-binaries/test.py", line 74, in <module>
     8    main()
     9  File "/home/ben/Documents/GitHub/bitcoin/contrib/verify-binaries/test.py", line 27, in main
    10    assert len(v) == 1
    11           ^^^^^^^^^^^
    12AssertionError
    
  2. DrahtBot commented at 5:18 pm on May 21, 2024: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK stickies-v, willcl-ark

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    No conflicts as of last run.

  3. DrahtBot added the label Consensus on May 21, 2024
  4. DrahtBot added the label CI failed on May 21, 2024
  5. DrahtBot commented at 5:47 pm on May 21, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/25239186886

  6. BenWestgate force-pushed on May 21, 2024
  7. BenWestgate commented at 5:37 am on May 22, 2024: contributor
    @DrahtBot: I am fairly sure this PR is NOT a consensus change. It’s a bugfix to a download verification tool script. Replace that label with a more accurate one.
  8. maflcko commented at 8:08 am on May 22, 2024: member
    @BenWestgate I believe this is a typo in the documentation, fixed in #30150. (You can change the title yourself to contrib: and the bot will pick it up at some point.)
  9. DrahtBot removed the label Consensus on May 22, 2024
  10. fanquake renamed this:
    script: Fix errors in verify-binaries/verify.py OS platform parsing, update test.py & docs
    contrib: Fixup verify-binaries OS platform parsing
    on May 22, 2024
  11. maflcko commented at 8:20 am on May 22, 2024: member
    Merge commits in pull requests are discouraged in this repo, please squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits
  12. willcl-ark commented at 9:18 am on May 22, 2024: member
    There are also similar changes in #28418 but I didn’t check this PR yet to see if they are the same.
  13. BenWestgate commented at 5:23 pm on May 22, 2024: contributor

    There are also similar changes in #28418 but I didn’t check this PR yet to see if they are the same.

    Update: Thank you. I gave your PR a code review and test. I don’t think its approach fully achieves what it set out to do: always download 1 single file if enough info is provided. I borrowed your strings though.

    This one can download any 1 file where #28418 can’t. Given the more complex approach it may be easier to continue with this PR than fix it. This PR updates test.py to cover the new functionality and a much smaller diff: only 3 logic adds to verify-binaries/verify.py.

    Also my bad for almost duplicating someone else’s PR. I searched for “verify.py” but didn’t find any issues or pull requests.

  14. DrahtBot removed the label CI failed on May 23, 2024
  15. DrahtBot added the label Scripts and tools on May 23, 2024
  16. BenWestgate force-pushed on May 23, 2024
  17. BenWestgate closed this on May 23, 2024

  18. BenWestgate force-pushed on May 23, 2024
  19. BenWestgate reopened this on May 23, 2024

  20. BenWestgate force-pushed on May 23, 2024
  21. DrahtBot added the label CI failed on May 23, 2024
  22. DrahtBot commented at 3:45 am on May 23, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/25310724896

  23. BenWestgate force-pushed on May 23, 2024
  24. BenWestgate force-pushed on May 23, 2024
  25. BenWestgate force-pushed on May 23, 2024
  26. BenWestgate commented at 4:33 am on May 23, 2024: contributor

    Merge commits in pull requests are discouraged in this repo, please squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits

    Done.

  27. BenWestgate marked this as ready for review on May 23, 2024
  28. DrahtBot removed the label CI failed on May 23, 2024
  29. willcl-ark commented at 9:50 am on May 24, 2024: member

    Yep I agree this is cleaner and works nicely in my initial manual testing :)

    I will do a code review early next week, thanks!

  30. in contrib/verify-binaries/verify.py:512 in 8135632c33 outdated
    508@@ -514,7 +509,7 @@ def cleanup():
    509     # Extract hashes and filenames
    510     hashes_to_verify = parse_sums_file(SUMS_FILENAME, [os_filter])
    511     if not hashes_to_verify:
    512-        log.error("no files matched the platform specified")
    513+        log.error("no files matched the platform specified. check available versions on bitcoincore.org/bin")
    


    willcl-ark commented at 10:30 am on May 29, 2024:
    Not sure if we want this to specify bitcoincore.org, bitcoin.org does host some binaries too (and recently even added a few which aren’t totally out of date!), and we do try both hosts. On the other hand, this advice is actually more useful, accurate and likely to be correct, so 🤷🏼‍♂️

    BenWestgate commented at 8:24 am on June 9, 2024:

    @willcl-ark: I agree, but this advice is even more useful, accurate and likely to be correct: https://github.com/bitcoin/bitcoin/blob/5cc2ae4ff287c6fef10b2e634e35353588f9ea3d/contrib/verify-binaries/verify.py#L511-L515 This error message is smarter, it suggests the [-platform] from SHA256SUMS which is closest to what the user typed.

    If you like the concept, give https://github.com/bitcoin/bitcoin/commit/5cc2ae4ff287c6fef10b2e634e35353588f9ea3d a test, and I will squash it into this PR to resolve this conversion.


    willcl-ark commented at 12:06 pm on June 21, 2024:

    Well, I’ve enjoyed learning that python has a difflib builtin library. I thought I’d used all the string libraries by now, but somehow never come across that one.

    If you did wwant to incorporate that change here, then I’d be happy to reACK this. I did a few tests and it did for example nicely match 27.1-x64-linux to suggest Did you mean: x86_64-linux-gnu.tar.gz, along with a few other typos I tried, which does seem neat.


    BenWestgate commented at 4:19 pm on June 23, 2024:
    Incorporated.
  31. willcl-ark commented at 10:32 am on May 29, 2024: member

    I think the version parser here is is pretty good shape generally, or at least improved from its current state.

    I am not convinced yet that it is handling rc correctly yet though, see results of this additional test i added:

     0₿ ./contrib/verify-binaries/test2.py
     1Checking 27.0
     2Found RC version: version_base=27.0, version_rc= os_filter=
     3Checking 27.0-x86_64
     4Found RC version: version_base=27.0, version_rc= os_filter=x86_64
     5Checking 27.0-linux
     6Found RC version: version_base=27.0, version_rc= os_filter=linux
     7Checking 27.0-win64.zip
     8Found RC version: version_base=27.0, version_rc= os_filter=win64.zip
     9Checking bitcoin-27.0.tar.gz
    10Found RC version: version_base=bitcoin, version_rc= os_filter=27.0.tar.gz
    11Checking bitcoin-27.1-rc1.tar.gz
    12Found RC version: version_base=bitcoin, version_rc= os_filter=27.1-rc1.tar.gz
    13Missing rc
    14Checking 23.0-rc5-linux-gnu
    15Found RC version: version_base=23.0, version_rc=rc5 os_filter=linux-gnu
    16Checking bitcoin-27.0-rc1-x86_64-linux-gnu
    17Found RC version: version_base=bitcoin, version_rc= os_filter=27.0-rc1-x86_64-linux-gnu
    18Missing rc
    19Checking bitcoin-27.0-rc1-x86_64-linux-gnu.tar.gz
    20Found RC version: version_base=bitcoin, version_rc= os_filter=27.0-rc1-x86_64-linux-gnu.tar.gz
    21Missing rc
    

    Passing an architecture and platform, or an extension sees it miss-parse version_rc.

    Without a live release candidate build being hosted anywhere, I am not sure what exactly bitcoin-27.0-rc1-x86_64-linux-gnu would download with this os_filter. I could probably try and work it out, but didn’t bother here yet.

  32. willcl-ark commented at 10:34 am on May 29, 2024: member

    Here is the file I was using to check the new parsing

     0#!/usr/bin/env python3
     1
     2import tempfile
     3from verify import parse_version_string, parse_sums_file
     4
     5SHASUMS = """cb35e250ae9d0328aa90e7aad0b877ed692597420a1092e8ab1a5dd756209722  bitcoin-27.0-aarch64-linux-gnu.tar.gz
     69d4c28e7620d03bf346ebea388f222e4d6d2b996d7eb32fab72707b8320d5249  bitcoin-27.0-arm-linux-gnueabihf.tar.gz
     77f060f2cd07746ff9d09b000b4195fee88dfca8444ab7a73f0c76aff4225227c  bitcoin-27.0-arm64-apple-darwin.zip
     81d9d9b837297a73fc7a3b1cfed376644e3fa25c4e1672fbc143d5946cb52431d  bitcoin-27.0-arm64-apple-darwin.tar.gz
     99c1ee651d3b157baccc3388be28b8cf3bfcefcd2493b943725ad6040ca6b146b  bitcoin-27.0.tar.gz
    106ceaedb59ca33b751387b15f2c8da7f2f7cd2739c6464fc6cbef440852869b92  bitcoin-27.0-powerpc64-linux-gnu.tar.gz
    113c00f81a7c67b4cf3e382fae7eaa2c7facea2dfdf39f4c281512237c06b71960  bitcoin-27.0-powerpc64le-linux-gnu.tar.gz
    12371e53b21c3ba29a90e69c30b7213d75c165d084bde50ae6d73ee0e1ef179e68  bitcoin-27.0-riscv64-linux-gnu.tar.gz
    138c94d3a7e34b59effdcf283263d5e84f2b009e601076282e9697ab4244bef3e8  bitcoin-27.0-x86_64-apple-darwin.zip
    14e1efd8c4605b2aabc876da93b6eee2bedd868ce7d1f02b0220c1001f903b3e2c  bitcoin-27.0-x86_64-apple-darwin.tar.gz
    152a6974c5486f528793c79d42694b5987401e4a43c97f62b1383abf35bcee44a8  bitcoin-27.0-x86_64-linux-gnu.tar.gz
    16a2aa3db390a768383e8556878250a44f3eb3b7a6e91e94e47fa35c06b6e8d09f  bitcoin-27.0-win64-setup.exe
    17ca75babeaa3fb75f5a166f544adaa93fd7c1f06cf20d4e2c8c2a8b010f4c7603  bitcoin-27.0-win64.zip
    18"""
    19
    20expected_results = [
    21    (
    22        "27.0",
    23        [
    24            "bitcoin-27.0-aarch64-linux-gnu.tar.gz",
    25            "bitcoin-27.0-arm-linux-gnueabihf.tar.gz",
    26            "bitcoin-27.0-arm64-apple-darwin.zip",
    27            "bitcoin-27.0-arm64-apple-darwin.tar.gz",
    28            "bitcoin-27.0.tar.gz",
    29            "bitcoin-27.0-powerpc64-linux-gnu.tar.gz",
    30            "bitcoin-27.0-powerpc64le-linux-gnu.tar.gz",
    31            "bitcoin-27.0-riscv64-linux-gnu.tar.gz",
    32            "bitcoin-27.0-x86_64-apple-darwin.zip",
    33            "bitcoin-27.0-x86_64-apple-darwin.tar.gz",
    34            "bitcoin-27.0-x86_64-linux-gnu.tar.gz",
    35            "bitcoin-27.0-win64-setup.exe",
    36            "bitcoin-27.0-win64.zip",
    37        ],
    38    ),
    39    (
    40        "27.0-x86_64",
    41        [
    42            "bitcoin-27.0-x86_64-apple-darwin.zip",
    43            "bitcoin-27.0-x86_64-apple-darwin.tar.gz",
    44            "bitcoin-27.0-x86_64-linux-gnu.tar.gz",
    45        ],
    46    ),
    47    (
    48        "27.0-linux",
    49        [
    50            "bitcoin-27.0-aarch64-linux-gnu.tar.gz",
    51            "bitcoin-27.0-arm-linux-gnueabihf.tar.gz",
    52            "bitcoin-27.0-powerpc64-linux-gnu.tar.gz",
    53            "bitcoin-27.0-powerpc64le-linux-gnu.tar.gz",
    54            "bitcoin-27.0-riscv64-linux-gnu.tar.gz",
    55            "bitcoin-27.0-x86_64-linux-gnu.tar.gz",
    56        ],
    57    ),
    58    ("27.0-win64.zip", ["bitcoin-27.0-win64.zip"]),
    59    ("bitcoin-27.0.tar.gz", ["bitcoin-27.0.tar.gz"]),
    60    ("bitcoin-27.1-rc1.tar.gz", ["bitcoin-27.0.tar.gz"]),
    61    ("23.0-rc5-linux-gnu", []),
    62    ("bitcoin-27.0-rc1-x86_64-linux-gnu", []),
    63    ("bitcoin-27.0-rc1-x86_64-linux-gnu.tar.gz", []),
    64]
    65
    66
    67def test_version_parser():
    68    with tempfile.NamedTemporaryFile(mode="w+t", delete=False) as tmp_sums_file:
    69        tmp_sums_file.write(SHASUMS)
    70        tmp_sums_file_path = tmp_sums_file.name
    71
    72    for term, matches in expected_results:
    73        print(f"Checking {term}")
    74        version_base, version_rc, os_filter = parse_version_string(term)
    75        print(f"Found RC version: {version_base=:}, {version_rc=:} {os_filter=:}")
    76        if not version_rc and ("rc" in term):
    77            print("Missing rc")
    78        hashes_to_verify = parse_sums_file(tmp_sums_file_path, [os_filter])
    79        results = [name for hash, name in hashes_to_verify]
    80        try:
    81            assert sorted(matches) == sorted(results)
    82        except:
    83            continue
    84
    85
    86if __name__ == "__main__":
    87    test_version_parser()
    
  33. BenWestgate commented at 7:37 pm on May 29, 2024: contributor

    Here’s the issue:

    Checking bitcoin-27.0.tar.gz Found RC version: version_base=bitcoin, version_rc= os_filter=27.0.tar.gz

    It’s not designed to work when the version string begins with bitcoin- What else would we be using bitcoin/bitcoin contrib/verify-binaries for?

    27.0-.tar.gz would be the correct way to search for all binaries ending in this extension with this parser.

    Whatever comes before the 1st dash must be the version number in . or .. format.

    Even without starting with bitcoin- 27.0.tar.gz should be rejected as it’s not a valid version number and the first field before ‘-‘ is always the version number.

    Do you want it to not get confused when the string starts with bitcoin before the version number?

    Not sure if we want this to specify bitcoincore.org,bitcoin.orgdoes host some binaries too (and recently even added a few which aren’t totally out of date!), and we do try both hosts. On the other hand, this advice is actually more useful, accurate and likely to be correct, so 🤷🏼‍♂️

    For this, I thought the script should actually print the available versions that it did find rather than tell the user to navigate to a website.

    That is a little out of the scope of this PR but I’d be happy to update that warning to spit out all the file names (or search strings that would exact match a file name, i.e. with “bitcoin-“ removed.) or better: the closest matches to the platform they searched for by edit distance.

    That way something like 27.0-win65.zop would suggest “27.0-win64.zip” first in the list of “did you mean?” Instead of printing every platform.

    It looks like it’s parsing -rcN correctly when it’s the 2nd field after version number.

    On Wed, May 29, 2024 at 05:32, Will Clark @.***(mailto:On Wed, May 29, 2024 at 05:32, Will Clark «a href=)> wrote:

    @willcl-ark commented on this pull request.

    I think the version parser here is is pretty good shape generally, or at least improved from its current state.

    I am not convinced yet that it is handling rc correctly yet though, see results of this additional test i added:

    ₿ ./contrib/verify-binaries/test2.py Checking 27.0 Found RC version: version_base=27.0, version_rc= os_filter= Checking 27.0-x86_64 Found RC version: version_base=27.0, version_rc= os_filter=x86_64 Checking 27.0-linux Found RC version: version_base=27.0, version_rc= os_filter=linux Checking 27.0-win64.zip Found RC version: version_base=27.0, version_rc= os_filter=win64.zip Checking bitcoin-27.0.tar.gz Found RC version: version_base=bitcoin, version_rc= os_filter=27.0.tar.gz Checking bitcoin-27.1-rc1.tar.gz Found RC version: version_base=bitcoin, version_rc= os_filter=27.1-rc1.tar.gz Missing rc Checking 23.0-rc5-linux-gnu Found RC version: version_base=23.0, version_rc=rc5 os_filter=linux-gnu Checking bitcoin-27.0-rc1-x86_64-linux-gnu Found RC version: version_base=bitcoin, version_rc= os_filter=27.0-rc1-x86_64-linux-gnu Missing rc Checking bitcoin-27.0-rc1-x86_64-linux-gnu.tar.gz Found RC version: version_base=bitcoin, version_rc= os_filter=27.0-rc1-x86_64-linux-gnu.tar.gz Missing rc

    Passing an architecture and platform, or an extension sees it miss-parse version_rc.

    Without a live release candidate build being hosted anywhere, I am not sure what exactly bitcoin-27.0-rc1-x86_64-linux-gnu would download with this os_filter. I could probably try and work it out, but didn’t bother here yet.


    In contrib/verify-binaries/verify.py:

    @@ -514,7 +509,7 @@ def cleanup(): # Extract hashes and filenames hashes_to_verify = parse_sums_file(SUMS_FILENAME, [os_filter]) if not hashes_to_verify:

    •    log.error("no files matched the platform specified")
      
    •    log.error("no files matched the platform specified. check available versions on bitcoincore.org/bin")
      

    Not sure if we want this to specify bitcoincore.org, bitcoin.org does host some binaries too (and recently even added a few which aren’t totally out of date!), and we do try both hosts. On the other hand, this advice is actually more useful, accurate and likely to be correct, so 🤷🏼‍♂️

    — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: @.***>

  34. willcl-ark approved
  35. willcl-ark commented at 8:50 am on May 30, 2024: member

    tACK 8135632c33f4bad8434403b5807c48d7dba3b3d7

    This works much better than the current verifier, and being able to specify an exact version to download is more convenient in general, and especially in places such as building docker images where downloading multiple unneeded versions is purely wasteful.

    Still one open comment regarding specifying bitcoincore.org, but happy to reACK if that gets changed, and it’s not a blocker for me.

  36. BenWestgate commented at 8:39 am on June 9, 2024: contributor

    tACK 8135632

    Still one open comment regarding specifying bitcoincore.org, but happy to reACK if that gets changed, and it’s not a blocker for me.

    I wrote a better error message that gives a “Did you mean: <closest_match>” when the platform typed isn’t in the SHA256SUMS. Avoiding the domain issue and being significantly more helpful. Added it to this PR.

  37. BenWestgate force-pushed on Jun 23, 2024
  38. BenWestgate force-pushed on Jun 23, 2024
  39. BenWestgate requested review from willcl-ark on Jun 23, 2024
  40. willcl-ark approved
  41. willcl-ark commented at 9:09 am on June 24, 2024: member

    ACK a2fc9ddee9cbaeffb3c460973bab3c2dd734db55

    Compared changes since last review using git range-diff 8135632...a2fc9ddee9cbaeffb3c460973bab3c2dd734db55, which added the difflib helper and updated the commit message.

  42. fanquake requested review from stickies-v on Jun 24, 2024
  43. in contrib/verify-binaries/verify.py:107 in a2fc9ddee9 outdated
    110-            version_rc = parts[1]
    111-        else:
    112-            version_os = parts[1]
    113-    elif len(parts) == 3:  # "<version>-rcN-platform"
    114+    version_os = "-".join(parts[1:]) # "<version>" or "<version>-platform"
    115+    if "-rc" in "-".join(parts[:2]): # "<version>-rcN[-platform]"
    


    stickies-v commented at 3:48 pm on June 24, 2024:

    nit: I think using startswith is a bit more intuitive here.

    The entire logic can be simplified too, I think:

     0diff --git a/contrib/verify-binaries/verify.py b/contrib/verify-binaries/verify.py
     1index b7f64d4336..d09ae0dbfe 100755
     2--- a/contrib/verify-binaries/verify.py
     3+++ b/contrib/verify-binaries/verify.py
     4@@ -100,13 +100,10 @@ VERSION_FORMAT = "<major>.<minor>[.<patch>][-rc[0-9]][-platform]"
     5 VERSION_EXAMPLE = "22.0 or 23.1-rc1-darwin.dmg or 27.0-x86_64-linux-gnu"
     6 
     7 def parse_version_string(version_str):
     8-    parts = version_str.split('-')
     9-    version_base = parts[0]
    10+    version_base, _, version_os = version_str.partition('-')
    11     version_rc = ""
    12-    version_os = "-".join(parts[1:]) # "<version>" or "<version>-platform"
    13-    if "-rc" in "-".join(parts[:2]): # "<version>-rcN[-platform]"
    14-        version_rc = parts[1]
    15-        version_os = "-".join(parts[2:])
    16+    if version_os.startswith("rc"):
    17+        version_rc, _, version_os = version_os.partition('-')
    18 
    19     return version_base, version_rc, version_os
    20 
    

    BenWestgate commented at 11:44 am on June 25, 2024:
    That does look nicer, implementing your suggestion. Thanks for teaching me .partition() and .startswith().

    stickies-v commented at 11:47 am on June 25, 2024:
    Great. I didn’t mean to take out your example docstrings btw so feel free to add those/similar ones in, it’s helpful.

    BenWestgate commented at 11:49 am on June 25, 2024:
    0def parse_version_string(version_str):
    1    parts = version_str.split('-')
    2    # "<version>" or "<version>-platform"
    3    version_base, _, version_os = version_str.partition('-')
    4    version_rc = ""
    5    if version_os.startswith("rc"): # "<version>-rcN[-platform]"
    6        version_rc, _, version_os = version_os.partition('-')
    7
    8    return version_base, version_rc, version_os
    

    Look good to you?


    stickies-v commented at 11:50 am on June 25, 2024:
    Actually: would version, rc, platform be better variable names? The version_ prefix doesn’t really make sense I think.

    stickies-v commented at 11:52 am on June 25, 2024:

    Look good to you?

    No need for the parts = statement I think.

    # "<version>" or "<version>-platform"

    That seems wrong, given that we may also have an rc part still?


    BenWestgate commented at 12:02 pm on June 25, 2024:

    All of your comments are right! Glad I asked.

    Looks better with the new local variable names:

    0
    1def parse_version_string(version_str):
    2    # "<version>[-rcN][-platform]"
    3    version_base, _, platform = version_str.partition('-')
    4    rc = ""
    5    if platform.startswith("rc"): # "<version>-rcN[-platform]"
    6        rc, _, platform = platform.partition('-')
    7    # else "<version>" or "<version>-platform"
    8
    9    return version_base, rc, platform
    

    I don’t love my first comment, but it’s less clear without it. Edit: nvm, it grew on me. Comments are good. Squashed and force pushed our changes.

  44. in contrib/verify-binaries/verify.py:514 in a2fc9ddee9 outdated
    508@@ -514,7 +509,9 @@ def cleanup():
    509     # Extract hashes and filenames
    510     hashes_to_verify = parse_sums_file(SUMS_FILENAME, [os_filter])
    511     if not hashes_to_verify:
    512-        log.error("no files matched the platform specified")
    513+        available_versions = ["-".join(line[1].split("-")[2:]) for line in parse_sums_file(SUMS_FILENAME, [])]
    514+        closest_match = difflib.get_close_matches(os_filter, available_versions, cutoff=0, n=1)[0]
    515+        log.error("No files matched the platform specified. Did you mean: "+closest_match)
    


    stickies-v commented at 4:08 pm on June 24, 2024:

    f-string nit

    0        log.error(f"No files matched the platform specified. Did you mean: {closest_match}")
    
  45. stickies-v approved
  46. stickies-v commented at 4:22 pm on June 24, 2024: contributor

    ACK a2fc9ddee9cbaeffb3c460973bab3c2dd734db55

    Nice improvement, a few suggestion nits but not blocking - happy to quickly re-review if you adopt them.

    Tested that all the new examples work too.

  47. BenWestgate force-pushed on Jun 25, 2024
  48. DrahtBot added the label CI failed on Jun 25, 2024
  49. DrahtBot commented at 12:27 pm on June 25, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/26651378072

  50. in contrib/verify-binaries/verify.py:465 in 4a85b154d0 outdated
    461@@ -468,7 +462,7 @@ def cleanup():
    462 
    463     # determine remote dir dependent on provided version string
    464     try:
    465-        version_base, version_rc, os_filter = parse_version_string(args.version)
    466+        version_base, rc, os_filter = parse_version_string(args.version)
    


    stickies-v commented at 1:02 pm on June 25, 2024:
    meta nit: although consistency is good, changing otherwise unaffected lines/functions just to do an unnecessary rename can be counterproductive, so this would have been fine to keep as is

    BenWestgate commented at 3:50 pm on June 25, 2024:

    It was a “change all occurrences” accident. version_os was local to parse_version_string, so I logically assumed version_rc would be too. So I blame the original variable names for this goof. I didn’t see until after I had force pushed.

    Shall I restrict the version_rc to rc rename to only within the parse_version_string function? Or not worth changing back?


    stickies-v commented at 4:26 pm on June 25, 2024:
    I’d prefer limiting it to parse_version_string and will quickly re-ack if you force push

  51. DrahtBot removed the label CI failed on Jun 25, 2024
  52. BenWestgate requested review from stickies-v on Jun 25, 2024
  53. stickies-v approved
  54. stickies-v commented at 3:30 pm on June 25, 2024: contributor
    ACK 4a85b154d0874f7bfa848ae8d7ab341eea754607 - nice work
  55. DrahtBot requested review from willcl-ark on Jun 25, 2024
  56. contrib: Fixup verify-binaries OS platform parsing
    Parse platform strings with "-" or '.' correctly such as "linux-gnu" or
    "x86_64-linux-gnu.tar.gz" to download the matching files or file. String
    partition() is used to tolerate more dashes. Update `VERSION_EXAMPLE`
    with a new string parsed correctly now. Fix "-aarch64" interpreted as a
    release candidate due to sub-string "rc", causing all downloads to fail.
    Now "rc" must immediately follow first "-" to indicate an [-rc] string.
    Local variables `version_rc`, `version_os` renamed to `rc`, `platform`.
    If "-rcN" is specified, `platform` is reassigned to remove the '-rcN'.
    
    Changes are useful to only download one bitcoin core binary on slow
    connections. Making `verify.py pub` more intuitive, robust, and
    versatile. Closes #30145
    
    When user types a platform string not found in any filename lets help
    and say the platform closest to what they typed in a `f"No files
    matched the platform specified. Did you mean: {closest_match}"` log.
    Improves UX when unaware how we name our files.
    Uses the difflib Python built-in which was already imported elsewhere.
    
    Update test.py to test single file verification
    verify-binaries/verify.py can accept an entire filename filter for its
    "-platform" parameter now so let us test that it succeeds and downloads
    and verifies only one file. `verify.py pub 22.0-x86_64-linux-gnu.tar.gz`
    should get and verify only the requested binary. It is placed before the
    existing <version> wide verification as it is a faster test and possibly
    easier to break.
    
    Update doc with examples now possible after bugfix
    Add example to show release candidates now work with "-platform" strings
    containing "-" and string provided can be from the middle of filename:
    `./contrib/verify-binaries/verify.py --json pub 23.0-rc5-linux-gnu`
    Change example 5 to not match example 3.
    New examples to show platform can now be provided specifically enough to
    download only a single binary down to its file extension:
    `./contrib/verify-binaries/verify.py pub 25.2-x86_64-linux`
    `./contrib/verify-binaries/verify.py pub 24.1-rc1-darwin`
    `./contrib/verify-binaries/verify.py pub 27.0-win64-setup.exe`
    This is the most common use if not verifying all files so users see it
    as the first example for "only download the binaries for a certain
    architecture and/or platform". Downloading one file is intuitively what
    most will think this meant and this change delivers on that expectation.
    
    Co-authored-by: stickies-v
    3ab2520190
  57. BenWestgate force-pushed on Jun 25, 2024
  58. BenWestgate requested review from stickies-v on Jun 25, 2024
  59. stickies-v approved
  60. stickies-v commented at 4:37 pm on June 25, 2024: contributor
    re-ACK 3ab25201909bece9066ac6191670bcee09791d54
  61. willcl-ark approved
  62. willcl-ark commented at 7:34 pm on June 25, 2024: member

    re-ACK 3ab25201909bece9066ac6191670bcee09791d54

    Based on range diff since previous review: git range-diff a2fc9ddee9cbaeffb3c460973bab3c2dd734db55...3ab25201909bece9066ac6191670bcee09791d54

  63. fanquake merged this on Jun 26, 2024
  64. fanquake closed this on Jun 26, 2024


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-09-29 01:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me