test: pycapnp doesn't support free threaded Python

fanquake commented at 8:52 am on October 9, 2025: member

Python 3.14.0 has been released, and “Free-threaded Python is officially supported”. However using Python 3.14.0t and running the functional tests results in a failure in interface_ipc.py:

 0./build/test/functional/test_runner.py interface_ipc.py
 1Temporary test directory at /var/folders/sq/z88fhjzj0b19ftsd2_bjrmjm0000gn/T/test_runner_₿_🏃_20251009_094704
 2Remaining jobs: [interface_ipc.py]
 31/1 - interface_ipc.py failed, Duration: 7 s
 4
 5stdout:
 62025-10-09T08:47:05.067786Z TestFramework (INFO): PRNG seed is: 8793929945784019901
 72025-10-09T08:47:05.069073Z TestFramework (INFO): Initializing test directory /var/folders/sq/z88fhjzj0b19ftsd2_bjrmjm0000gn/T/test_runner_₿_🏃_20251009_094704/interface_ipc_0
 82025-10-09T08:47:05.880408Z TestFramework (INFO): Running echo test
 92025-10-09T08:47:05.883429Z TestFramework (INFO): Running mining test
102025-10-09T08:47:12.059233Z TestFramework (INFO): Stopping nodes
112025-10-09T08:47:12.173668Z TestFramework (INFO): Cleaning up /var/folders/sq/z88fhjzj0b19ftsd2_bjrmjm0000gn/T/test_runner_₿_🏃_20251009_094704/interface_ipc_0 on exit
122025-10-09T08:47:12.173842Z TestFramework (INFO): Tests successful
13[node 0] Cleaning up ipc directory '/var/folders/sq/z88fhjzj0b19ftsd2_bjrmjm0000gn/T/test-ipc-s5k_0gog'
14
15
16stderr:
17<frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'capnp.lib.capnp', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
18
19
20
21TEST             | STATUS    | DURATION
22
23interface_ipc.py | ✖ Failed  | 7 s
24
25ALL              | ✖ Failed  | 7 s (accumulated) 
26Runtime: 7 s

This is using the latest available version of pycapnp, installed from git. i.e https://github.com/capnproto/pycapnp/commit/c14fd7036aaef45cadef8ce4673dc6612ffc3198.

fanquake added the label Tests on Oct 9, 2025

fanquake added the label Upstream on Oct 9, 2025

fanquake commented at 8:52 am on October 9, 2025: member

cc @Sjors @ryanofsky

maflcko commented at 9:29 am on October 9, 2025: member

I’d expect there is a bunch of our own test code which relies on the GIL. I guess there is no way to find such “unsafe” Python code other than to try to run with the GIL disabled and wait for an intermittent issue to pop up at some point in time?

maflcko commented at 9:43 am on October 9, 2025: member

Generally, I’d say we should continue to treat the no-GIL as experimental, possibly accept fixes where they make sense. However, longer term, I wonder what the goal should be. According to https://peps.python.org/pep-0779/#rationale there is a performance and memory overhead of the no-GIL version. Also, the test code is largely single threaded (at least logic wise), with sync and wait loops put basically after every non-sync statement. So my guess is that, if we wanted to make meaningful use of no-GIL, we’d have to re-write a good chunk of the test framework, at which point it could be re-written in another language?

sipa commented at 11:21 am on October 9, 2025: member

@maflcko My understanding is that from the perspective of the interpreter Python code, GIL or no GIL is largely unobservable, apart from a few minor things. How do you think it would affect our test framework?

fanquake commented at 11:22 am on October 9, 2025: member

I’d expect there is a bunch of our own test code which relies on the GIL. I guess there is no way to find such “unsafe” Python code other than to try to run with the GIL disabled and wait for an intermittent issue to pop up at some point in time?

Possibly; however this was the only test I’ve seen fail so far. I’m just assuming devs/people will start using the free-threaded pythons (maybe in the near future) on their machines, and if they do, this test will fail.

ryanofsky commented at 11:33 am on October 9, 2025: contributor

Not sure about the larger issue, but it seems practically speaking it might be nice if test framework just ignored or suppressed the GIL RuntimeWarning for interface_ipc.py if the test otherwise passes.

maflcko commented at 2:11 pm on October 9, 2025: member

@maflcko My understanding is that from the perspective of the interpreter Python code, GIL or no GIL is largely unobservable, apart from a few minor things. How do you think it would affect our test framework?

Python is not thread-safe by design and there are no thread-safety annotations. Historically, this didn’t matter much, because the GIL in the majority of cases will (by accident or by design?) add thread-safety. The tests may or may not pass without the GIL, but race conditions are now certainly observable. (I think we are adding locks where needed on a best-effort basis, but this hasn’t been checked formally)

You can try this yourself locally:

 0$ cat /tmp/a.py   
 1import threading
 2
 3counter = 0
 4
 5def increment():
 6    global counter
 7    for _ in range(100000):
 8        counter += 1
 9
10threads = [threading.Thread(target=increment) for _ in range(2)]
11for t in threads:
12    t.start()
13for t in threads:
14    t.join()
15
16print(counter)  # May not be 200000 due to race condition

0$ python3.10 /tmp/a.py
1200000
2
3$ python3.14t /tmp/a.py    
4129976

Unrelated(?), I am also pretty sure we violate RUF006 (https://docs.astral.sh/ruff/rules/asyncio-dangling-task/). However, I haven’t observed this Heisenbug in reality, so I didn’t bother to ask for review. Generally, review for race fixes in the Python test code is slim at best (https://github.com/bitcoin/bitcoin/pull/33140 is waiting for months).

So generally, I’d exepct that long-term no-GIL will bring us more intermittent test failures and require fixes for them with no observable benefit.

Though, maybe I am missing something?

sipa commented at 2:27 pm on October 9, 2025: member

@maflcko Thanks - of course.

test: pycapnp doesn’t support free threaded Python #33582