Fixes #29972
Use a single script to run the linter locally or in CI.
Works from inside a worktree.
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/34391.
See the guideline for information on the review process.
| Type | Reviewers |
|---|---|
| ACK | maflcko, l0rinc, davidgumberg |
If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.
13+```
14+
15+Or use the included Dockerfile:
16
17 ```sh
18 DOCKER_BUILDKIT=1 docker build --platform=linux --tag=bitcoin-linter --file="./ci/lint_imagefile" ./ && docker run --rm -v $(pwd):/bitcoin -it bitcoin-linter
30+ if not content.startswith("gitdir: "):
31+ return []
32+ gitdir = (repo_root / content.removeprefix("gitdir: ")).resolve()
33+ main_gitdir = gitdir.parent.parent
34+ return [
35+ f"--volume={gitdir}:{gitdir}:ro",
0 f"--volume={gitdir}:{gitdir}",
otherwise:
0fatal: Unable to create '/btc/bitcoin/.git/worktrees/34391/index.lock': Read-only file system
alternatively, could get rid gitdir and just mount main_gitdir (without ro ):
0diff --git a/ci/lint.py b/ci/lint.py
1index ed522c6e90..eff624ce8e 100755
2--- a/ci/lint.py
3+++ b/ci/lint.py
4@@ -25,16 +25,12 @@ def run(cmd, **kwargs):
5 def get_worktree_mounts(repo_root):
6 git_path = repo_root / ".git"
7 if not git_path.is_file():
8- return []
9+ return ""
10 content = git_path.read_text().strip()
11 if not content.startswith("gitdir: "):
12- return []
13- gitdir = (repo_root / content.removeprefix("gitdir: ")).resolve()
14- main_gitdir = gitdir.parent.parent
15- return [
16- f"--volume={gitdir}:{gitdir}:ro",
17- f"--volume={main_gitdir}:{main_gitdir}:ro",
18- ]
19+ return ""
20+ gitdir = (repo_root / content.removeprefix("gitdir: ")).resolve().parent.parent
21+ return f"--volume={gitdir}:{gitdir}"
22
23
24 def main():
25@@ -71,7 +67,7 @@ def main():
26 "--rm",
27 *extra_env,
28 f"--volume={repo_root}:/bitcoin",
29- *get_worktree_mounts(repo_root),
30+ get_worktree_mounts(repo_root),
31 *([] if is_ci else ["-it"]),
32 container,
33 *sys.argv[1:],
but this is slightly less belt and suspenders I guess, no strong feelings about that.
ro from gitdir in a3c44cd2762ce1464bf2ec7abe4e2d0fd539b834
72+ *extra_env,
73+ f"--volume={repo_root}:/bitcoin",
74+ *get_worktree_mounts(repo_root),
75+ *([] if is_ci else ["-it"]),
76+ container,
77+ *sys.argv[1:],
Would be nice to add a note documenting this in the README
e.g. just mentioning that you can do:
0./ci/lint.py ./test/lint/lint-python.py
not blocking at all, feel free to disregard
Also taken in a3c44cd2762ce1464bf2ec7abe4e2d0fd539b834.
Thanks for the review!
cargo run, but I have run into a mismatch with the CI linter doing that.
17+
18+```sh
19+./ci/lint.py ./test/lint/lint-python.py
20 ```
21
22 Building the container can be done every time, because it is fast when the
14+```
15+
16+Or to run a single stage using the container:
17+
18+```sh
19+./ci/lint.py ./test/lint/lint-python.py
This seems confusing, because it seems to be implying to lint python, however it only does checks of little value (like mypy checks), and it is missing the main python lint: ruff.
Also, there are no docs on what lints are available. So there is no easy way to run just (lets say) includes_build_config.
I think it would be better to pass any extra args to cargo run -- ..., so that one can call ./ci/lint.py --help or ./ci/lint.py includes_build_config
execed inside this container, but now we always pass all args to cargo run -- .
680@@ -681,4 +681,4 @@ jobs:
681 cache-provider: ${{ needs.runners.outputs.provider }}
682
683 - name: CI script
684- run: python .github/ci-lint-exec.py
685+ run: ./ci/lint.py
git worktree add ../lint-worktree && cd ... here?
17+ print("+ " + shlex.join(cmd), flush=True)
18+ kwargs.setdefault("check", True)
19+ try:
20+ return subprocess.run(cmd, **kwargs)
21+ except Exception as e:
22+ sys.exit(str(e))
No LLM bug :)
I have switched to ty (to go along with ruff and uv for all-things-astral Python) and when left as e it outputs diagnostic:
0core/worktrees/lint-worktree on lint-worktrees [$!] via △ v4.1.2 via 🐍 v3.13.11 via ❄️ impure (nix-shell-env)
1❯ ty check ci/lint.py
2error[invalid-argument-type]: Argument to function `exit` is incorrect
3 --> ci/lint.py:22:18
4 |
520 | return subprocess.run(cmd, **kwargs)
621 | except Exception as e:
722 | sys.exit(e)
8 | ^ Expected `str | int | None`, found `Exception`
9 |
10info: Function defined here
11 --> stdlib/sys/__init__.pyi:669:5
12 |
13667 | """
14668 |
15669 | def exit(status: _ExitCode = None, /) -> NoReturn:
16 | ^^^^ ------------------------ Parameter declared here
17670 | """Exit the interpreter by raising SystemExit(status).
18 |
19info: rule `invalid-argument-type` is enabled by default
20
21Found 1 diagnostic
So just thought i’d tidy that up.
ty
Ah interesting. I tried to run it once on the Bitcoin Core codebase, and it printed a lot of stuff, so I gave up looking into it.
Though, I think it is actually broken here. According to https://docs.python.org/3/library/sys.html#sys.exit, the object can be any type:
The optional argument arg can be an integer giving the exit status (defaulting to zero), or another type of object.
But this is probably a refactor, and anything is fine here. I just left the comment, because the exact same code is used in several places:
0$ git grep 'sys.exit(e)'
1.github/ci-lint-exec.py: sys.exit(e)
2.github/ci-test-each-commit-exec.py: sys.exit(e)
3ci/test/02_run_container.py: sys.exit(e)
it could make sense to use the exact same code in all places, but just a nit.
Ah interesting. I tried to run it once on the Bitcoin Core codebase, and it printed a lot of stuff, so I gave up looking into it.
I have a branch with the needed rules: https://github.com/bitcoin/bitcoin/compare/master...willcl-ark:bitcoin:typechecker-migration which I might open after this.
I also have a branch with code fixes (which is still ~ WIP) so we could drop all rules and use the ty defaults.
it could make sense to use the exact same code in all places, but just a nit.
I think I’ll leave it as-is for now, to avoid more pushing.
8+import subprocess
9+import sys
10+import time
11+from pathlib import Path
12+
13+CONTAINER_NAME = "bitcoin-linter"
47+ "buildx",
48+ "build",
49+ f"--tag={container}",
50+ *shlex.split(os.environ.get("DOCKER_BUILD_CACHE_ARG", "")),
51+ f"--file={repo_root}/ci/lint_imagefile",
52+ str(repo_root),
str()? In python you can pass String and OsStr in the same args vector, because there are no types checked at compile time (as opposed to Rust).
Sure you can in the vector, but it breaks parsing with shlex in run(), e.g.:
0core/worktrees/lint-worktree on lint-worktrees [$!⇕] via △ v4.1.2 via 🐍 v3.13.11 via ❄️ impure (nix-shell-env) took 1m22s
1❯ ./ci/lint.py
2Traceback (most recent call last):
3 File "/home/will/src/core/worktrees/lint-worktree/./ci/lint.py", line 81, in <module>
4 main()
5 ~~~~^^
6 File "/home/will/src/core/worktrees/lint-worktree/./ci/lint.py", line 52, in main
7 if run(build_cmd, check=False).returncode != 0:
8 ~~~^^^^^^^^^^^^^^^^^^^^^^^^
9 File "/home/will/src/core/worktrees/lint-worktree/./ci/lint.py", line 15, in run
10 print("+ " + shlex.join(cmd), flush=True)
11 ~~~~~~~~~~^^^^^
12 File "/nix/store/qzc04a3npl70cyyy6flnnrb2ig3kayxm-python3-3.13.11/lib/python3.13/shlex.py", line 318, in join
13 return ' '.join(quote(arg) for arg in split_command)
14 ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15 File "/nix/store/qzc04a3npl70cyyy6flnnrb2ig3kayxm-python3-3.13.11/lib/python3.13/shlex.py", line 318, in <genexpr>
16 return ' '.join(quote(arg) for arg in split_command)
17 ~~~~~^^^^^
18 File "/nix/store/qzc04a3npl70cyyy6flnnrb2ig3kayxm-python3-3.13.11/lib/python3.13/shlex.py", line 327, in quote
19 if _find_unsafe(s) is None:
20 ~~~~~~~~~~~~^^^
21TypeError: expected string or bytes-like object, got 'PosixPath'
shlex
Ah, my bad. I tested this locally, but only with subprocess.run, and not with shlex.
lgtm, but there are a few nits that can be addressed, if you don’t mind.
review ACK a3c44cd2762ce1464bf2ec7abe4e2d0fd539b834 🐐
Signature:
0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: review ACK a3c44cd2762ce1464bf2ec7abe4e2d0fd539b834 🐐
3YjwaCc8IYb3Ge1tArtslHvOjSRSZTjUk6dqN7JXKwcjIqne05AKZ2BAInR3QDWrXKCGMrd/GeRPLovxCH81ECA==
Add a ci/lint.py script to run the linter both locally or inside the CI
(replacing .github/ci-lint-exec.py) which supports running from a
worktree.
Determines whether we are in a worktree, and mounts the real `.git`
directory as a read-only volume if we are.
This looks great and fixes not only the worktree bug, but makes running the lint ci container easier and less confusing.
review ACK 5aeaa71c77ac31a1b05f8361356d2810cfc5bc28 🔒
Signature:
0untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
1RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
2trusted comment: review ACK 5aeaa71c77ac31a1b05f8361356d2810cfc5bc28 🔒
3uMQF+LcwFndYWlC9EoLKixpd6iHFOGudWD2pCfxDfN+BnOLIVJX9uqWXmhOwLN1tpyCVqRYV5nYihrpCFgZFDQ==
11@@ -12,8 +12,4 @@ git config --global --add safe.directory /bitcoin
12
13 export PATH="/python_build/bin:${PATH}"
14
15-if [ -z "$1" ]; then
16- bash -ic "./ci/lint/06_script.sh"
17-else
18- exec "$@"
19-fi
20+./ci/lint/06_script.sh "$@"
My understanding is that this is a deliberate behavior change, right?
Previously, when args were provided, the entrypoint would run an arbitrary command in the container (not sure how useful that was).
Now we always forward those args to the test_runner binary (seems deliberate in the last commit).
So e.g. docker run --rm -v "$(pwd):/bitcoin" -it bitcoin-linter bash would start a shell before, but now prints the test_runner usage/help.
Yes that’s correct, it’s an intentional behaviour change. With the changes in here you’d need to override the entrypoint.sh script to drop into a bash shell with the --entrypoint flag, e.g.
0docker run --rm --entrypoint bash -v "$(pwd):/bitcoin" -it bitcoin-linter
edit: meant to say I think the change here would be more useful 99% of the time, and so worth making getting a bash shell inside the lint container (what for?) slightly more difficult :)
My understanding is that this is a deliberate behavior change, right? Previously, when args were provided, the entrypoint would run an arbitrary command in the container (not sure how useful that was). Now we always forward those args to the
test_runnerbinary (seems deliberate in the last commit).So e.g.
docker run --rm -v "$(pwd):/bitcoin" -it bitcoin-linter bashwould start a shell before, but now prints thetest_runnerusage/help.
The only use-case for allowing bash would be to drop in a shell and develop new linters? Or maybe to use the bash alias lint ( echo 'alias lint="./ci/lint/06_script.sh"' >> ~/.bashrc && \).
However, both use-cases are now redundant:
./ci/lint.py seems shorter, more consistent and also faster. You can even call ./ci/lint.py my_new_lint, and it should not have any visible overhead.lint alias inside the container does not seem a lot shorter than ./ci/lint.py outside the container. Also, the outside one has the benefit to refresh the docker image cache, if it becomes stale.So I think the behavior change here makes sense, and I think we can even go as far and remove the ci/lint/container-entrypoint.sh file and just inline it into ci/lint/06_script.sh. However, I didn’t want to suggest it, because it is just internal refactoring cleanup, which can be done later.
Tested (+ lightly reviewed) ACK 5aeaa71c77ac31a1b05f8361356d2810cfc5bc28
Ran ./ci/lint.py from root - first it failed with and ugly Cannot connect to the Docker daemon (in a follow-up maybe we can add a guard to display a nicer error message since it’s not immediately obvious anymore that this command requires Docker).
Starting docker and executing the linter with and without local errors seems to work fine - thanks for simplifying this!
0diff --git a/src/merkleblock.h b/src/merkleblock.h
1--- a/src/merkleblock.h(revision 09cf559404cf9aafca06d7710716dea4d19ee162)
2+++ b/src/merkleblock.h(date 1769522839084)
3@@ -3,8 +3,8 @@
4 // Distributed under the MIT software license, see the accompanying
5 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
6
7-#ifndef BITCOIN_MERKLEBLOCK_H
8-#define BITCOIN_MERKLEBLOCK_H
9+#ifndef INVALID_BITCOIN_MERKLEBLOCK_H
10+#define INVALID_BITCOIN_MERKLEBLOCK_H
11
12 #include <common/bloom.h>
13 #include <primitives/block.h>
14@@ -157,4 +157,4 @@
15 CMerkleBlock(const CBlock& block, CBloomFilter* filter, const std::set<Txid>* txids);
16 };
17
18-#endif // BITCOIN_MERKLEBLOCK_H
19+#endif // INVALID_BITCOIN_MERKLEBLOCK_H
successfully finds:
0src/merkleblock.h seems to be missing the expected include guard to prevent the double inclusion problem:
1 #ifndef BITCOIN_MERKLEBLOCK_H
2 #define BITCOIN_MERKLEBLOCK_H
3 ...
4 #endif // BITCOIN_MERKLEBLOCK_H
code review and lightly tested reACK 5aeaa71
I reviewed $ git range-diff a3c44cd27...5aeaa71 and the related discussions, the changes make sense:
lint.py are now passed to the test_runner binary rather than being an arbitrary command executed from the container. (https://github.com/bitcoin/bitcoin/pull/34391#discussion_r2731228678)CONTAINER_NAME is inlined. (https://github.com/bitcoin/bitcoin/pull/34391#discussion_r2731385047)Tested that running the linter works on 5aeaa71 both from a worktree and from the main repository by running ./ci/lint.py and that passing args works as expected:
0$ ./ci/lint.py --lint=tabs_whitespace
1# # [...]
2+ RUST_BACKTRACE=1
3+ cargo run --manifest-path ./test/lint/test_runner/Cargo.toml -- --lint=tabs_whitespace
4 Finished dev [unoptimized + debuginfo] target(s) in 0.01s
5 Running `test/lint/test_runner/target/debug/test_runner --lint=tabs_whitespace`
6Checking commit range (1b079becf14dd5ba2debf2a5bf90832eb42d5604..HEAD):
75aeaa71c77 lint: pass args from lint.py to cargo run in container
8c17a2adb8d lint: upgrade lint scripts for worktrees
9
10+ '[' '' = 1 ']'
39+ repo_root = Path(__file__).resolve().parent.parent
40+ is_ci = os.environ.get("GITHUB_ACTIONS") == "true"
41+ container = "bitcoin-linter"
42+
43+ build_cmd = [
44+ "docker",
docker=podman, so that most things expecting docker just continue to work with Podman. Linting worked with the previous incantation, but will be broken by the changes here. However I can fix in a followup.
--platform=linux here, but this was an issue before as well, so i think i’ll fix this in a follow-up
but this was an issue before as well,
What was the issue previously? Running the linters using podman worked fine for me?
but this was an issue before as well,
What was the issue previously? Running the linters using podman worked fine for me?
It was a bit of an edge case when running s390x qemu containers, see #34427 (comment) for exact steps to reproduce the issue and the fix.
https://github.com/bitcoin/bitcoin/actions/runs/21433387530/job/61718013430:
0error: could not lock config file /home/admin/actions-runner/_work/bitcoin/bitcoin/.git/config: Read-only file system
https://github.com/bitcoin/bitcoin/actions/runs/21433387530/job/61718013430:
0error: could not lock config file /home/admin/actions-runner/_work/bitcoin/bitcoin/.git/config: Read-only file system
I think there are two possible fixes:
:ro. It seems odd anyway to make it read-only when using a worktree, but not when using no worktree.Remove the
:ro. It seems odd anyway to make it read-only when using a worktree, but not when using no worktree.
I think it makes most sense to PR this quick fix now. The ideal fix I think is to have verify-commits.py run also on PRs, but using merge-base with master as the top commit, which I will look at opening a followup for.
* Properly support read-only for all cases and all folders. This should be the ideal solution. However, I am haven't looked into this.
I guess it may be possible by using :ro for all volumes and then rsync them in a next step to the correct destination, but that sounds tedious.
0* Properly support read-only for all cases and all folders. This should be the ideal solution. However, I am haven't looked into this.I guess it may be possible by using
:rofor all volumes and thenrsyncthem in a next step to the correct destination, but that sounds tedious.
Thinking about it more, I think the “correct” fix might be to just move the git configuration https://github.com/bitcoin/bitcoin/blob/feb74a9372be0670b9e40032d7729e43994ddc26/ci/lint/06_script.sh#L29-L30 into the CI yaml file. At this point, i think mounting as :ro will work again, as no git configuration will take place after mounting… (also we never want to modify someone’s git local git config)