test: pycapnp doesn't support free threaded Python #33582

Python 3.14.0 has been released, and "Free-threaded Python is officially supported". However using Python 3.14.0t and running the functional tests results in a failure in interface_ipc.py:

./build/test/functional/test_runner.py interface_ipc.py
Temporary test directory at /var/folders/sq/z88fhjzj0b19ftsd2_bjrmjm0000gn/T/test_runner_₿_🏃_20251009_094704
Remaining jobs: [interface_ipc.py]
1/1 - interface_ipc.py failed, Duration: 7 s

stdout:
2025-10-09T08:47:05.067786Z TestFramework (INFO): PRNG seed is: 8793929945784019901
2025-10-09T08:47:05.069073Z TestFramework (INFO): Initializing test directory /var/folders/sq/z88fhjzj0b19ftsd2_bjrmjm0000gn/T/test_runner_₿_🏃_20251009_094704/interface_ipc_0
2025-10-09T08:47:05.880408Z TestFramework (INFO): Running echo test
2025-10-09T08:47:05.883429Z TestFramework (INFO): Running mining test
2025-10-09T08:47:12.059233Z TestFramework (INFO): Stopping nodes
2025-10-09T08:47:12.173668Z TestFramework (INFO): Cleaning up /var/folders/sq/z88fhjzj0b19ftsd2_bjrmjm0000gn/T/test_runner_₿_🏃_20251009_094704/interface_ipc_0 on exit
2025-10-09T08:47:12.173842Z TestFramework (INFO): Tests successful
[node 0] Cleaning up ipc directory '/var/folders/sq/z88fhjzj0b19ftsd2_bjrmjm0000gn/T/test-ipc-s5k_0gog'


stderr:
<frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'capnp.lib.capnp', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.



TEST             | STATUS    | DURATION

interface_ipc.py | ✖ Failed  | 7 s

ALL              | ✖ Failed  | 7 s (accumulated) 
Runtime: 7 s

This is using the latest available version of pycapnp, installed from git. i.e https://github.com/capnproto/pycapnp/commit/c14fd7036aaef45cadef8ce4673dc6612ffc3198.

cc @Sjors @ryanofsky

I'd expect there is a bunch of our own test code which relies on the GIL. I guess there is no way to find such "unsafe" Python code other than to try to run with the GIL disabled and wait for an intermittent issue to pop up at some point in time?

Generally, I'd say we should continue to treat the no-GIL as experimental, possibly accept fixes where they make sense. However, longer term, I wonder what the goal should be. According to https://peps.python.org/pep-0779/#rationale there is a performance and memory overhead of the no-GIL version. Also, the test code is largely single threaded (at least logic wise), with sync and wait loops put basically after every non-sync statement. So my guess is that, if we wanted to make meaningful use of no-GIL, we'd have to re-write a good chunk of the test framework, at which point it could be re-written in another language?

@maflcko My understanding is that from the perspective of the interpreter Python code, GIL or no GIL is largely unobservable, apart from a few minor things. How do you think it would affect our test framework?

I'd expect there is a bunch of our own test code which relies on the GIL. I guess there is no way to find such "unsafe" Python code other than to try to run with the GIL disabled and wait for an intermittent issue to pop up at some point in time?

Possibly; however this was the only test I've seen fail so far. I'm just assuming devs/people will start using the free-threaded pythons (maybe in the near future) on their machines, and if they do, this test will fail.

Not sure about the larger issue, but it seems practically speaking it might be nice if test framework just ignored or suppressed the GIL RuntimeWarning for interface_ipc.py if the test otherwise passes.

@maflcko My understanding is that from the perspective of the interpreter Python code, GIL or no GIL is largely unobservable, apart from a few minor things. How do you think it would affect our test framework?

Python is not thread-safe by design and there are no thread-safety annotations. Historically, this didn't matter much, because the GIL in the majority of cases will (by accident or by design?) add thread-safety. The tests may or may not pass without the GIL, but race conditions are now certainly observable. (I think we are adding locks where needed on a best-effort basis, but this hasn't been checked formally)

You can try this yourself locally:

$ cat /tmp/a.py   
import threading

counter = 0

def increment():
    global counter
    for _ in range(100000):
        counter += 1

threads = [threading.Thread(target=increment) for _ in range(2)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(counter)  # May not be 200000 due to race condition

$ python3.10 /tmp/a.py
200000

$ python3.14t /tmp/a.py    
129976

Unrelated(?), I am also pretty sure we violate RUF006 (https://docs.astral.sh/ruff/rules/asyncio-dangling-task/). However, I haven't observed this Heisenbug in reality, so I didn't bother to ask for review. Generally, review for race fixes in the Python test code is slim at best (https://github.com/bitcoin/bitcoin/pull/33140 is waiting for months).

So generally, I'd exepct that long-term no-GIL will bring us more intermittent test failures and require fixes for them with no observable benefit.

Though, maybe I am missing something?

@maflcko Thanks - of course.

So I think the correct fix would be to force GIL=1 for now, see #33795 (comment)

Just to clarify, I think the issue may still be visible when running the test directly: ./bld-cmake/test/functional/interface_ipc.py

But I guess no one does this, so it is not worth to keep this issue open over that.