feature: enable mypy to run without --ignore-imports #22844

pull josibake wants to merge 2 commits into bitcoin:master from josibake:josibake-fix-mypy-import-errors changing 5 files +4 −3
  1. josibake commented at 12:40 PM on August 31, 2021: member

    This relates to #19389 by enabling mypy to run without --ignore-imports. This is the minimum amount of work needed to get mypy running in the linter, after which annotations can be added to scripts in contrib/ and test_framework/ to get the full benefit of running mypy in the linter.

    special notes

    This also involved updating PyZMQ per #19389 (comment) .

    how to test

    after making these changes, I ran ./test/lint/lint-python.sh and verified no errors.

    To be sure mypy would actually catch type errors when they existed, I added the following lines to contrib/devtools/symbol-checker.py:

    def check_imported_symbols(filename: str) -> bool:
        test = filename + 1
        ...
    

    First, filename needed to be annotated as str, or else mypy treats it as Any and Any +1 is allowed. After annotating filename: str and adding an illegal operation, running ./test/lint/lint-python.sh` returned the following error:

    contrib/devtools/symbol-check.py:170:5: F841 local variable 'test' is assigned to but never used
    contrib/devtools/symbol-check.py:170: error: Unsupported operand types for + ("str" and "int")  [operator]
    

    next steps

    Where we do have mypy annotations, they are often incomplete. This causes mypy to evaluate arguments as Any (see example above with symbol-checker.py). I propose a follow-up PR adding annotations. At a minimum, annotations should be added for everything in test_framework/, which can be done fairly easily with monkeytype. The benefit of this is, now that we are running without --ignore-imports, mypy can detect errors in the functional tests even if they don't have annotations, so long as all of the files in test_framework/ are fully annotated.

  2. fanquake commented at 12:51 PM on August 31, 2021: member

    Pretty sure need to install pyzmq into the container where the linter is running:

    test/functional/test_framework/test_framework.py:805: error: Cannot find implementation or library stub for module named 'zmq'  [import]
    test/functional/interface_zmq.py:32: error: Cannot find implementation or library stub for module named 'zmq'  [import]
    test/functional/interface_zmq.py:32: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
    Found 2 errors in 2 files (checked 214 source files)
    ^---- failure generated from test/lint/lint-python.sh
    
  3. josibake commented at 12:59 PM on August 31, 2021: member

    Pretty sure need to install pyzmq into the container where the linter is running:

    ah yes, good catch!

  4. bump pyzmq version to 22.2.1
    mypy stubs were introduced in 21.0.1
    install 22.2.1 (latest as of this commit)
    065c64a4a0
  5. feature: fix mypy import errors
    ignore lief package
    remove --ignore-imports flag from mympy run
    make data/ importable
    47995dddeb
  6. josibake force-pushed on Aug 31, 2021
  7. DrahtBot added the label Scripts and tools on Aug 31, 2021
  8. DrahtBot commented at 7:35 AM on September 1, 2021: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #23148 (build: Fix guix linker-loader path and add check_ELF_interpreter by dongcarl)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  9. practicalswift commented at 1:47 PM on September 4, 2021: contributor

    Concept ACK

  10. fanquake commented at 5:33 AM on September 10, 2021: member

    Can you change the commit message for 065c64a4a06bee3392000e553f508fc7644560c0 to be something like: ci: install pyzmq into lint container. As you're not actually bumping the version, you're adding it to the container for the first time.

    Can you also add a commit bumping mypy to the latest version. Looks like it's 0.910.

    In either commit can you add / update the dependency here as well: https://github.com/bitcoin/bitcoin/tree/master/test#lint-tests.

  11. laanwj commented at 2:32 PM on October 1, 2021: member

    Concept ACK, less exceptions in mypy allows for more thorough checking.

  12. in test/lint/lint-python.sh:105 in 47995dddeb
     101 | @@ -102,7 +102,7 @@ if ! PYTHONWARNINGS="ignore" flake8 --ignore=B,C,E,F,I,N,W --select=$(IFS=","; e
     102 |      EXIT_CODE=1
     103 |  fi
     104 |  
     105 | -if ! mypy --ignore-missing-imports --show-error-codes $(git ls-files "test/functional/*.py" "contrib/devtools/*.py"); then
     106 | +if ! mypy --show-error-codes $(git ls-files "test/functional/*.py" "contrib/devtools/*.py"); then
    


    laanwj commented at 2:37 PM on October 1, 2021:

    Something I've wondered about in the past is whether checking these in the same mypy invocation is a good idea. I think it will consider all the files part of the same top-level module, which might cause inconsistencies of different directories have same-named modules, for example. That said, it works.

  13. fanquake commented at 6:22 AM on October 7, 2021: member

    I've fixed this up / taken it over in #23212.

  14. fanquake closed this on Oct 7, 2021

  15. maflcko referenced this in commit 3bf40d06a2 on Oct 17, 2021
  16. sidhujag referenced this in commit dde8b9a002 on Oct 17, 2021
  17. bitcoin locked this on Oct 30, 2022
  18. josibake deleted the branch on Jan 26, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-18 12:14 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me