[wip] ignore 2603 ci empty #34726
pull maflcko wants to merge 2 commits into bitcoin:master from maflcko:2603-ci-empty changing 16 files +228 −477-
maflcko commented at 9:54 pm on March 3, 2026: member
-
maflcko marked this as a draft on Mar 3, 2026
-
DrahtBot commented at 9:55 pm on March 3, 2026: contributor
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
Reviews
See the guideline for information on the review process.
Type Reviewers Stale ACK furszy If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.
Conflicts
Reviewers, this pull request conflicts with the following ones:
- #32297 (bitcoin-cli: Add -ipcconnect option by ryanofsky)
- #32061 (Replace libevent with our own HTTP and socket-handling implementation by pinheadmz)
- #31723 (qa: Facilitate debugging bitcoind inside functional tests by hodlinator)
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
LLM Linter (✨ experimental)
Possible typos and grammar issues:
-
#'p2p_orphan_handling.py'-># 'p2p_orphan_handling.py'[Missing space after#makes the commented-out line harder to read/identify as a comment.] -
will never time out.->will never timeout.[Typographic inconsistency in the compound noun; “timeout” is the standard form for the concept referenced.]
2026-03-26 16:46:45
-
DrahtBot added the label CI failed on Mar 3, 2026
-
furszy commented at 1:32 am on March 4, 2026: member
ACK de71c9adfb777d3bbfc7b88e06f2f61db0030bf8
Changes look clearly good to me.
-
fanquake commented at 12:32 pm on March 9, 2026: memberStill using this?
-
maflcko force-pushed on Mar 11, 2026
-
maflcko force-pushed on Mar 11, 2026
-
maflcko commented at 7:48 pm on March 11, 2026: memberFunny, how the bug goes away when logging is added?
-
maflcko closed this on Mar 11, 2026
-
maflcko reopened this on Mar 11, 2026
-
maflcko force-pushed on Mar 12, 2026
-
DrahtBot removed the label CI failed on Mar 12, 2026
-
maflcko closed this on Mar 12, 2026
-
maflcko reopened this on Mar 12, 2026
-
maflcko force-pushed on Mar 12, 2026
-
DrahtBot added the label CI failed on Mar 12, 2026
-
maflcko force-pushed on Mar 13, 2026
-
maflcko force-pushed on Mar 13, 2026
-
DrahtBot removed the label CI failed on Mar 13, 2026
-
DrahtBot added the label CI failed on Mar 13, 2026
-
maflcko commented at 7:23 pm on March 13, 2026: memberNo idea. Giving up for now.
-
maflcko closed this on Mar 13, 2026
-
maflcko commented at 7:52 pm on March 13, 2026: memberhmm, let’s try docker
-
maflcko reopened this on Mar 13, 2026
-
maflcko force-pushed on Mar 13, 2026
-
maflcko force-pushed on Mar 13, 2026
-
maflcko force-pushed on Mar 13, 2026
-
maflcko commented at 1:38 pm on March 16, 2026: member
So it does reproduce even inside docker on that machine (d08d96a3ae858a2310674eb1d52094f5974976c9)
Though, I can’t reproduce locally when running the equivalent in a
ubuntu:24.04container:0apt update && apt-get install vim git clang mold ccache build-essential cmake ninja-build pkgconf python3-zmq libevent-dev libboost-dev libsqlite3-dev systemtap-sdt-dev libzmq3-dev qt6-base-dev qt6-tools-dev qt6-l10n-tools libqrencode-dev capnproto libcapnp-dev python3-pip gdb valgrind -y && pip3 install --break-system-packages pycapnp && git clone https://github.com/bitcoin/bitcoin ./b-c-test --depth=3 && cd b-c-test/ && git fetch origin d08d96a3ae858a2310674eb1d52094f5974976c9 --depth=3 && git checkout d08d96a3ae858a2310674eb1d52094f5974976c9 1 2while python3 ./.github/ci-test-each-commit-exec.py ; do echo _ >> /tmp/runs ; doneI wonder if anyone has an AMD Ryzen™ 9 7950X3D locally to spin up an
ubuntu:24.04contaienr and run the above :thinking:In the meantime, I guess I can continue to (ab)use this pull to debug remotely…
-
maflcko force-pushed on Mar 16, 2026
-
maflcko force-pushed on Mar 16, 2026
-
maflcko force-pushed on Mar 16, 2026
-
maflcko force-pushed on Mar 17, 2026
-
maflcko force-pushed on Mar 17, 2026
-
maflcko commented at 3:46 pm on March 17, 2026: memberSo it failed with 70eae8f47457daf62439559f2a483134bd9d8077, but not with the latest push to c0f45a27e117e6384ed10cd60d2b62ed9f78de91? https://github.com/bitcoin/bitcoin/compare/70eae8f47457daf62439559f2a483134bd9d8077..c0f45a27e117e6384ed10cd60d2b62ed9f78de91 ?
-
maflcko closed this on Mar 17, 2026
-
maflcko reopened this on Mar 17, 2026
-
maflcko force-pushed on Mar 17, 2026
-
maflcko commented at 6:22 am on March 18, 2026: member
It get even more confusing now. https://github.com/bitcoin/bitcoin/actions/runs/23218523489/job/67485409619?pr=34726#step:10:201. There is now a print of:
02026-03-18T02:28:11.3902344Z 264/455 - wallet_createwallet.py --usecli passed, Duration: 2 s 12026-03-18T02:28:11.4035626Z 265/455 - wallet_createwallet.py passed, Duration: 2 s 22026-03-18T02:28:11.8908514Z 266/455 - rpc_signrawtransactionwithkey.py passed, Duration: 0 s 32026-03-18T02:28:12.1003495Z 267/455 - interface_rpc.py passed, Duration: 1 s 42026-03-18T02:28:12.1180766Z 268/455 - mempool_reorg.py passed, Duration: 3 s 52026-03-18T02:28:12.2918040Z 269/455 - rpc_whitelist.py passed, Duration: 1 s 62026-03-18T02:28:12.4402935Z 270/455 - tool_wallet.py passed, Duration: 7 s 72026-03-18T02:28:12.5942751Z WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! 82026-03-18T02:28:12.5943199Z 271/455 - p2p_orphan_handling.py failed, Duration: 45 sHowever,
- No earlier failure printed this warning.
- This is a fresh docker, so this warning can’t happen (Edit: The tests run in a loop, so the only way this could happen if there are stale/zombie bitcoind processes left over from an earlier run, or something spawns them)
- The warning appears in the wrong line, because
printhas flush disabled by default: https://docs.python.org/3/library/functions.html#print
-
maflcko force-pushed on Mar 18, 2026
-
DrahtBot removed the label CI failed on Mar 18, 2026
-
maflcko commented at 11:40 am on March 18, 2026: memberOk, 26.04 passed…. I guess I can bisect, but not sure what outcome to expect here …
-
maflcko force-pushed on Mar 18, 2026
-
DrahtBot added the label CI failed on Mar 18, 2026
-
maflcko commented at 4:07 pm on March 18, 2026: member
25.04 bad 26.04 good?
1 revision left to test …
-
maflcko force-pushed on Mar 18, 2026
-
maflcko commented at 7:56 pm on March 18, 2026: member
So toward the passing 26.04, the package version diff is:
Package Name Version (25.10) Version (26.04) git 1:2.51.0-1ubuntu1 1:2.53.0-1ubuntu1 clang 1:20.0-63ubuntu1 1:21.1.6-70 mold 2.37.1+dfsg-1 2.40.4+dfsg-2.1ubuntu1 ccache 4.11.2-2 4.12.3-1 build-essential 12.12ubuntu1 12.12ubuntu2 cmake 3.31.6-2ubuntu6 4.2.3-2ubuntu2 ninja-build 1.12.1-1 1.13.2-1 pkgconf 1.8.1-4build1 1.8.1-4build1 python3-zmq 26.4.0-1 27.1.0-1 libevent-dev 2.1.12-stable-10build1 2.1.12-stable-10build2 libboost-dev 1.88.0.0ubuntu1 1.90.0.0ubuntu3 libsqlite3-dev 3.46.1-8 3.46.1-9 systemtap-sdt-dev 5.1-5 5.3-2 libzmq3-dev 4.3.5-1build2 4.3.5-1build3 qt6-base-dev 6.9.2+dfsg-1ubuntu1 6.10.2+dfsg-3 qt6-tools-dev 6.9.2-1 6.10.2-1 qt6-l10n-tools 6.9.2-1 6.10.2-1 libqrencode-dev 4.1.1-2 4.1.1-2build1 capnproto 1.1.0-2 1.1.0-2.1 libcapnp-dev 1.1.0-2 1.1.0-2.1 python3-pip 25.1.1+dfsg-1ubuntu2 25.1.1+dfsg-1ubuntu2 I guess I’ll try clang both ways, but otherwise, there is not much that can be wiggled.
-
maflcko force-pushed on Mar 19, 2026
-
maflcko force-pushed on Mar 19, 2026
-
maflcko force-pushed on Mar 19, 2026
-
maflcko force-pushed on Mar 19, 2026
-
maflcko commented at 11:11 am on March 19, 2026: member
that failed, so the issue may not be in upstream Python itself?
02026-03-19T10:58:49.9881256Z 266/455 - wallet_createwallet.py passed, Duration: 2 s 12026-03-19T10:58:50.4596451Z WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! 22026-03-19T10:58:50.4597053Z 267/455 - p2p_orphan_handling.py failed, Duration: 45 s 32 -
maflcko commented at 11:14 am on March 19, 2026: memberlet me try 26.04 again, to really confirm it passes once more.
-
maflcko force-pushed on Mar 19, 2026
-
DrahtBot removed the label CI failed on Mar 19, 2026
-
maflcko commented at 5:33 pm on March 19, 2026: memberok, it passed for the third time
-
maflcko commented at 5:34 pm on March 19, 2026: memberLet me check the older python (and clang)
-
maflcko force-pushed on Mar 19, 2026
-
maflcko commented at 7:00 am on March 21, 2026: member
hmm. So it doesn’t seem to be the python version.
Let’s try clang. Otherwise, it must be some implicit dependency like glibc, or the build system that changed between ubuntu versions?
-
maflcko force-pushed on Mar 21, 2026
-
maflcko commented at 10:44 am on March 24, 2026: member
Hmm. clang-20 passed as well. So I guess this is something in the Ubuntu build system (or any of the build deps). Though, I am probably not going to start building Ubuntu from scratch and then bisect.
Let me try to install packages across Ubuntu versions …
-
maflcko force-pushed on Mar 24, 2026
-
maflcko force-pushed on Mar 24, 2026
-
maflcko commented at 6:36 am on March 25, 2026: memberConfirm a forth time that the issue really is inside the python ppa …
-
maflcko force-pushed on Mar 25, 2026
-
maflcko commented at 6:55 am on March 25, 2026: member
I was thinking one of the upstream CVE fixes broke this? Though, none of them hit Ubuntu 24.04, so maybe not?
0python3.14 (3.14.0-1ubuntu0.3) questing-security; urgency=medium 1 2 * SECURITY REGRESSION: Revert patch for CVE-2025-15366 3 - debian/patches/CVE-2025-15366.patch: Reverted. Patch breaks RFC 4 9051 IMAP conformance and introduces behavior regressions avoided 5 by upstream. 6 - CVE-2025-15366 7 * SECURITY REGRESSION: Revert patch for CVE-2025-15367 8 - debian/patches/CVE-2025-15367.patch: Reverted to prevent behavior 9 regressions, aligning with upstream backporting decisions. 10 - CVE-2025-15367 11 * SECURITY REGRESSION: Allow HTAB in wsgiref header values 12 - debian/patches/CVE-2026-0865-2.patch: Permit HTAB in header values 13 (excluding names) in Lib/wsgiref/headers.py, add test coverage. 14 - CVE-2026-0865 15 16 -- Vyom Yadav <vyom.yadav@canonical.com> Tue, 03 Mar 2026 18:06:02 +0530 17 18python3.14 (3.14.0-1ubuntu0.2) questing-security; urgency=medium 19 20 * SECURITY UPDATE: Header injection in email messages where addresses are not 21 sanitized. 22 - debian/patches/CVE-2025-11468.patch: Add escape parentheses and backslash 23 in Lib/email/_header_value_parser.py. Add test in 24 Lib/test/test_email/test__header_value_parser.py. 25 - CVE-2025-11468 26 * SECURITY UPDATE: Quadratic algorithm when building excessively nested XML 27 documents. 28 - debian/patches/CVE-2025-12084-*.patch: Remove _in_document and replace 29 with node.ownerDocument in Lib/xml/dom/minidom.py. Set self.ownerDocument 30 to None in Lib/xml/dom/minidom.py. Add test in Lib/test/test_minidom.py. 31 - CVE-2025-12084 32 * SECURITY UPDATE: OOM and denial of service when opening malicious plist 33 file. 34 - debian/patches/CVE-2025-13837.patch: Add _MIN_READ_BUF_SIZE and _read 35 with checks in Lib/plistlib.py. Add test in Lib/test/test_plistlib.py. 36 - CVE-2025-13837 37 * SECURITY UPDATE: Header injection in user controlled data URLs in urllib. 38 - debian/patches/CVE-2025-15282.patch: Add control character checks in 39 Lib/urllib/request.py. Add test in Lib/test/test_urllib.py. 40 * SECURITY UPDATE: Command injection through user controlled commands in 41 imaplib. 42 - debian/patches/CVE-2025-15366.patch: Add _control_chars and checks in 43 Lib/imaplib.py. Add test in Lib/test/test_imaplib.py. 44 * SECURITY UPDATE: Command injection through user controlled commands in 45 poplib. 46 - debian/patches/CVE-2025-15367.patch: Add control character regex check 47 in Lib/poplib.py. Add test in Lib/test/test_poplib.py. 48 - CVE-2025-15367 49 * SECURITY UPDATE: HTTP header injection in user controlled cookie values. 50 - debian/patches/CVE-2026-0672.patch: Add _control_characters_re and 51 checks in Lib/http/cookies.py. Add test in Lib/test/test_http_cookies.py. 52 - CVE-2026-0672 53 * SECURITY UPDATE: HTTP header injection in user controlled headers and 54 values with newlines. 55 - debian/patches/CVE-2026-0865.patch: Add _control_chars_re and check in 56 Lib/wsgiref/headers.py. Add test in Lib/test/support/__init__.py and 57 Lib/test/test_wsgiref.py. 58 - CVE-2026-0865 59 60 -- Hlib Korzhynskyy <hlib.korzhynskyy@canonical.com> Thu, 22 Jan 2026 16:10:43 -0330 61 62python3.14 (3.14.0-1ubuntu0.1) questing-security; urgency=medium 63 64 * SECURITY UPDATE: HTTP Content-Length denial of service 65 - debian/patches/CVE-2025-13836.patch: Read large data in chunks with 66 geometric reads in Lib/http/client.py and add tests in 67 Lib/test/test_httplib.py 68 - CVE-2025-13836 69 70 -- Vyom Yadav <vyom.yadav@canonical.com> Thu, 08 Jan 2026 19:50:58 +0530 -
DrahtBot added the label CI failed on Mar 25, 2026
-
maflcko force-pushed on Mar 25, 2026
-
maflcko commented at 7:57 am on March 25, 2026: member
Confirm a forth time that the issue really is inside the python ppa …
Ok, that failed again.
Let’s try
python3.13=3.13.7-1from0python3.13 (3.13.7-1ubuntu0.4) questing-security; urgency=medium 1 2 * SECURITY REGRESSION: Revert patch for CVE-2025-15366 3 - debian/patches/CVE-2025-15366.patch: Reverted. Patch breaks RFC 4 9051 IMAP conformance and introduces behavior regressions avoided 5 by upstream. 6 - CVE-2025-15366 7 * SECURITY REGRESSION: Revert patch for CVE-2025-15367 8 - debian/patches/CVE-2025-15367.patch: Reverted to prevent behavior 9 regressions, aligning with upstream backporting decisions. 10 - CVE-2025-15367 11 * SECURITY REGRESSION: Allow HTAB in wsgiref header values 12 - debian/patches/CVE-2026-0865-2.patch: Permit HTAB in header values 13 (excluding names) in Lib/wsgiref/headers.py, add test coverage. 14 - CVE-2026-0865 15 16 -- Vyom Yadav <vyom.yadav@canonical.com> Tue, 03 Mar 2026 17:49:54 +0530 17 18python3.13 (3.13.7-1ubuntu0.3) questing-security; urgency=medium 19 20 * SECURITY UPDATE: Header injection in email messages where addresses are not 21 sanitized. 22 - debian/patches/CVE-2025-11468.patch: Add escape parentheses and backslash 23 in Lib/email/_header_value_parser.py. Add test in 24 Lib/test/test_email/test__header_value_parser.py. 25 - CVE-2025-11468 26 * SECURITY UPDATE: Quadratic algorithm when building excessively nested XML 27 documents. 28 - debian/patches/CVE-2025-12084-*.patch: Remove _in_document and replace 29 with node.ownerDocument in Lib/xml/dom/minidom.py. Set self.ownerDocument 30 to None in Lib/xml/dom/minidom.py. Add test in Lib/test/test_minidom.py. 31 - CVE-2025-12084 32 * SECURITY UPDATE: OOM and denial of service when opening malicious plist 33 file. 34 - debian/patches/CVE-2025-13837.patch: Add _MIN_READ_BUF_SIZE and _read 35 with checks in Lib/plistlib.py. Add test in Lib/test/test_plistlib.py. 36 - CVE-2025-13837 37 * SECURITY UPDATE: Header injection in user controlled data URLs in urllib. 38 - debian/patches/CVE-2025-15282.patch: Add control character checks in 39 Lib/urllib/request.py. Add test in Lib/test/test_urllib.py. 40 * SECURITY UPDATE: Command injection through user controlled commands in 41 imaplib. 42 - debian/patches/CVE-2025-15366.patch: Add _control_chars and checks in 43 Lib/imaplib.py. Add test in Lib/test/test_imaplib.py. 44 * SECURITY UPDATE: Command injection through user controlled commands in 45 poplib. 46 - debian/patches/CVE-2025-15367.patch: Add control character regex check 47 in Lib/poplib.py. Add test in Lib/test/test_poplib.py. 48 - CVE-2025-15367 49 * SECURITY UPDATE: HTTP header injection in user controlled cookie values. 50 - debian/patches/CVE-2026-0672.patch: Add _control_characters_re and 51 checks in Lib/http/cookies.py. Add test in Lib/test/test_http_cookies.py. 52 - CVE-2026-0672 53 * SECURITY UPDATE: HTTP header injection in user controlled headers and 54 values with newlines. 55 - debian/patches/CVE-2026-0865.patch: Add _control_chars_re and check in 56 Lib/wsgiref/headers.py. Add test in Lib/test/support/__init__.py and 57 Lib/test/test_wsgiref.py. 58 - CVE-2026-0865 59 60 -- Hlib Korzhynskyy <hlib.korzhynskyy@canonical.com> Thu, 22 Jan 2026 16:45:57 -0330 61 62python3.13 (3.13.7-1ubuntu0.2) questing-security; urgency=medium 63 64 * SECURITY UPDATE: HTTP Content-Length denial of service 65 - debian/patches/CVE-2025-13836.patch: Read large data in chunks with 66 geometric reads in Lib/http/client.py and add tests in 67 Lib/test/test_httplib.py 68 - CVE-2025-13836 69 70 -- Vyom Yadav <vyom.yadav@canonical.com> Thu, 08 Jan 2026 17:45:45 +0530 71 72python3.13 (3.13.7-1ubuntu0.1) questing-security; urgency=medium 73 74 * SECURITY UPDATE: Possible payload obfuscation 75 - debian/patches/CVE-2025-8291.patch: check consistency of 76 the zip64 end of central dir record in Lib/zipfile.py, 77 Lib/test/test_zipfile.py. 78 - CVE-2025-8291 79 * SECURITY UPDATE: Performance degradation 80 - debian/patches/CVE-2025-6075.patch: fix quadratic complexity 81 in os.path.expandvars() in Lib/ntpatch.py, Lib/posixpath.py, 82 Lib/test/test_genericpatch.py, Lib/test/test_npath.py. 83 - CVE-2025-6075 84 85 -- Hlib Korzhynskyy <hlib.korzhynskyy@canonical.com> Mon, 24 Nov 2025 17:21:28 -0330 86 87python3.13 (3.13.7-1) unstable; urgency=medium 88 89 * Python 3.13.7. 90 * Patch: Use a list for epub_exclude_files in Sphinx config. 91 92 -- Stefano Rivera <stefanor@debian.org> Thu, 21 Aug 2025 00:17:40 +0200 -
maflcko commented at 8:36 am on March 25, 2026: member
-
maflcko commented at 10:19 am on March 25, 2026: memberok, I guess this will be passing. I’ll apply the CVE patch fix next …
-
DrahtBot removed the label CI failed on Mar 25, 2026
-
maflcko force-pushed on Mar 25, 2026
-
maflcko force-pushed on Mar 25, 2026
-
DrahtBot added the label CI failed on Mar 25, 2026
-
maflcko force-pushed on Mar 25, 2026
-
maflcko force-pushed on Mar 25, 2026
-
maflcko commented at 8:10 pm on March 25, 2026: member
Hmm, so the first patch passed …
Let’s go back to 7c01ca981d4f9c725dd6d00116aa18b86ac3984b and try to reduce the runtime a bit.
-
maflcko force-pushed on Mar 25, 2026
-
maflcko force-pushed on Mar 26, 2026
-
maflcko commented at 7:33 am on March 26, 2026: member
I don’t understand why it passed with the CVE patch applied, this means the patch is not relevant here, and some other ppa patch is relevant, but which one? Let’s try the CVE patch again, but
--reverse…Edit: That also fails, So I guess I am either applying the patch incorrectly, or it is unrelated …
-
maflcko force-pushed on Mar 26, 2026
-
maflcko force-pushed on Mar 26, 2026
-
maflcko force-pushed on Mar 26, 2026
-
in depends/patches/libevent/a.patch:12 in 12b3a4ca81
7+ } 8+ 9++ if (req->ntoread > 0) { 10++ /* We are ready to read the body. */ 11++ usleep(5000); // Sleep for 5 milliseconds 12++ }
rkrux commented at 1:16 pm on March 26, 2026:How does sleeping the server (libevent to be precise) before reading help?rkrux commented at 1:22 pm on March 26, 2026: contributorNice customisation of the testing job. nproc is 8 and 16 tests are run in parallel, 9600 p2p_orphan_handling tests are run in total.
Sharing observations from the tcpdump: The last run was on the 207th iteration of the test and it failed at 12:23:33 but the first “win 0” occurred 30secs earlier at 12:23:03, 11 secs after the start of the test. The connection started degrading 30secs earlier.
in .github/ci-test-each-commit-exec.py:58 in cef2ff12d3 outdated
62+ ] * (num_procs * 2 * 2 * 60 * 5) 63 run([ 64 sys.executable, 65 f"./{build_dir}/test/functional/test_runner.py", 66 "-j", 67 str(num_procs * 2),
rkrux commented at 2:01 pm on March 26, 2026:Since this job tests multiple commits (although sequentially I assume) and I noted that in the 4-5 failures attached in the issue, the failure occurred at any commit. It’s only in this customised job I noted it to fail in the first commit itself.
Do you think experimenting with
num_procs // 2for the “test ancestor commits” job would help? Overall the job will be slower though.maflcko force-pushed on Mar 26, 2026exclude+log+skip_ctest+docker25.10+py3.13+clang_20+python3.13+rep+tcpdump+lp+log+factor a82f6ebe88patch+rev_tcpdump 678f2b632amaflcko force-pushed on Mar 26, 2026maflcko closed this on Mar 26, 2026
maflcko deleted the branch on Mar 26, 2026
This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-03-30 00:13 UTC
More mirrored repositories can be found on mirror.b10c.me