ci: `feature_settings.py` failed in macOS 14 CI

tdb3 commented at 4:15 pm on October 2, 2024: contributor

Is there an existing issue for this?

I have searched the existing issues

Current behaviour

Encountered the following failure in PR #30793 (https://github.com/bitcoin/bitcoin/pull/30793#issuecomment-2387502458) for CI / macOS 14 native, arm64, no depends, sqlite only, gui (pull_request). The failure wasn’t seen on the other CI jobs.

feature_settings.py failed, timed out for Invalid value detected for '-wallet' or '-nowallet'.

Expected behaviour

No failure

Steps to reproduce

CI run with #39793 commit e6853592361341c27103ed74b25470ac1e098d6d

Relevant log output

https://github.com/bitcoin/bitcoin/actions/runs/11135374974/job/30945271145

 0node0 2024-10-02T00:15:45.783638Z [init] [src/noui.cpp:57] [noui_InitMessage] init message: Verifying wallet(s)… 
 1Error:  node0 2024-10-02T00:15:45.783653Z [init] [src/noui.cpp:31] [noui_ThreadSafeMessageBox] [error] Invalid value detected for '-wallet' or '-nowallet'. '-wallet' requires a string value, while '-nowallet' accepts only '1' to disable all wallets 
 2 node0 2024-10-02T00:15:45.783737Z [init] [src/httpserver.cpp:510] [InterruptHTTPServer] [http] Interrupting HTTP server 
 3 node0 2024-10-02T00:15:45.783748Z [init] [src/httprpc.cpp:379] [InterruptHTTPRPC] [rpc] Interrupting HTTP RPC server 
 4 node0 2024-10-02T00:15:45.783756Z [init] [src/rpc/server.cpp:291] [operator()] [rpc] Interrupting RPC 
 5 node0 2024-10-02T00:15:45.783770Z [init] [src/init.cpp:277] [Shutdown] Shutdown: In progress... 
 6 node0 2024-10-02T00:15:45.783782Z [shutoff] [src/httprpc.cpp:384] [StopHTTPRPC] [rpc] Stopping HTTP RPC server 
 7 node0 2024-10-02T00:15:45.783794Z [shutoff] [src/httpserver.cpp:772] [UnregisterHTTPHandler] [http] Unregistering HTTP handler for / (exactmatch 1) 
 8 node0 2024-10-02T00:15:45.783804Z [shutoff] [src/httpserver.cpp:772] [UnregisterHTTPHandler] [http] Unregistering HTTP handler for /wallet/ (exactmatch 0) 
 9 node0 2024-10-02T00:15:45.783810Z [shutoff] [src/rpc/server.cpp:303] [operator()] [rpc] Stopping RPC 
10 node0 2024-10-02T00:15:45.783892Z [shutoff] [src/rpc/server.cpp:309] [operator()] [rpc] RPC stopped. 
11 node0 2024-10-02T00:15:45.783904Z [shutoff] [src/httpserver.cpp:522] [StopHTTPServer] [http] Stopping HTTP server 
12 node0 2024-10-02T00:15:45.783911Z [shutoff] [src/httpserver.cpp:524] [StopHTTPServer] [http] Waiting for HTTP worker threads to exit 
13 node0 2024-10-02T00:15:45.784048Z [http] [src/httpserver.cpp:354] [ThreadHTTP] [http] Entering http event loop 
14 node0 2024-10-02T00:15:45.784129Z [http] [src/httpserver.cpp:357] [ThreadHTTP] [http] Exited http event loop 
15 test  2024-10-02T00:55:45.674000Z TestFramework (ERROR): Assertion failed 
16                                   Traceback (most recent call last):
17                                     File "/Users/runner/work/bitcoin/bitcoin/test/functional/test_framework/test_node.py", line 665, in assert_start_raises_init_error
18                                       ret = self.process.wait(timeout=self.rpc_timeout)
19                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
20                                     File "/opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 1264, in wait
21                                       return self._wait(timeout=timeout)
22                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
23                                     File "/opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 2045, in _wait
24                                       raise TimeoutExpired(self.args, timeout)
25                                   subprocess.TimeoutExpired: Command '['/Users/runner/work/bitcoin/bitcoin/ci/scratch/build-aarch64-apple-darwin23.6.0/src/bitcoind', '-datadir=/Users/runner/work/bitcoin/bitcoin/ci/scratch/test_runner/test_runner_₿_🏃_20241002_001101/feature_settings_11/node0', '-logtimemicros', '-debug', '-debugexclude=libevent', '-debugexclude=leveldb', '-debugexclude=rand', '-uacomment=testnode0', '-logthreadnames', '-logsourcelocations', '-loglevel=trace', '-v2transport=0', '-settings=/Users/runner/work/bitcoin/bitcoin/ci/scratch/test_runner/test_runner_₿_🏃_20241002_001101/feature_settings_11/node0/regtest/settings.json']' timed out after 2400 seconds
26                                   During handling of the above exception, another exception occurred:
27                                   Traceback (most recent call last):
28                                     File "/Users/runner/work/bitcoin/bitcoin/test/functional/test_framework/test_framework.py", line 132, in main
29                                       self.run_test()
30                                     File "/Users/runner/work/bitcoin/bitcoin/ci/scratch/build-aarch64-apple-darwin23.6.0/test/functional/feature_settings.py", line 110, in run_test
31                                       self.test_wallet_settings(settings)
32                                     File "/Users/runner/work/bitcoin/bitcoin/ci/scratch/build-aarch64-apple-darwin23.6.0/test/functional/feature_settings.py", line 39, in test_wallet_settings
33                                       node.assert_start_raises_init_error(expected_msg="Error: Invalid value detected for '-wallet' or '-nowallet'. '-wallet' requires a string value, while '-nowallet' accepts only '1' to disable all wallets",
34                                     File "/Users/runner/work/bitcoin/bitcoin/test/functional/test_framework/test_node.py", line 695, in assert_start_raises_init_error
35                                       self._raise_assertion_error(assert_msg)
36                                     File "/Users/runner/work/bitcoin/bitcoin/test/functional/test_framework/test_node.py", line 197, in _raise_assertion_error
37                                       raise AssertionError(self._node_msg(msg))
38                                   AssertionError: [node 0] bitcoind should have exited within 2400s with expected error Error: Invalid value detected for '-wallet' or '-nowallet'. '-wallet' requires a string value, while '-nowallet' accepts only '1' to disable all wallets
39 test  2024-10-02T00:55:45.690000Z TestFramework (DEBUG): Closing down network thread 
40 test  2024-10-02T00:55:45.894000Z TestFramework (INFO): Stopping nodes 
41 test  2024-10-02T00:55:45.894000Z TestFramework (WARNING): Not cleaning up dir /Users/runner/work/bitcoin/bitcoin/ci/scratch/test_runner/test_runner_₿_🏃_20241002_001101/feature_settings_11 
42 test  2024-10-02T00:55:45.895000Z TestFramework (ERROR): Test failed. Test logging available at /Users/runner/work/bitcoin/bitcoin/ci/scratch/test_runner/test_runner_₿_🏃_20241002_001101/feature_settings_11/test_framework.log 
43 test  2024-10-02T00:55:45.895000Z TestFramework (ERROR): 
44 test  2024-10-02T00:55:45.896000Z TestFramework (ERROR): Hint: Call /Users/runner/work/bitcoin/bitcoin/test/functional/combine_logs.py '/Users/runner/work/bitcoin/bitcoin/ci/scratch/test_runner/test_runner_₿_🏃_20241002_001101/feature_settings_11' to consolidate all logs 
45 test  2024-10-02T00:55:45.897000Z TestFramework (ERROR): 
46 test  2024-10-02T00:55:45.897000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log. 
47 test  2024-10-02T00:55:45.897000Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues 
48 test  2024-10-02T00:55:45.897000Z TestFramework (ERROR):

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v28.99.0-86e8b1bd8d33-dirty

Operating system and version

macOS 14.6.1

Machine specifications

arm64

tdb3 commented at 4:18 pm on October 2, 2024: contributor

At the time of the failure, the PR commits were on top of commit a74bdeea1b8e27b2335f0f7da78006e87ecfb235 (Sept 2nd). I’m probably missing something, but it almost seemed like the CI job was executing code that was introduced in commit ee47ca29d6ef55650a0af63bca817c5d494f31ef (dated Aug 22nd, from PR #30684, but merged with commit fb52023ee69c346dd101770716b6d8c0525c38aa on Sept 9th).

ismaelsadeeq commented at 11:11 am on October 3, 2024: member

@tdb3 it might be helpful to share link to the full C.I logs?

mzumsande commented at 2:39 pm on October 3, 2024: contributor

At the time of the failure, the PR commits were on top of commit https://github.com/bitcoin/bitcoin/commit/a74bdeea1b8e27b2335f0f7da78006e87ecfb235 (Sept 2nd). I’m probably missing something, but it almost seemed like the CI job was executing code that was introduced in commit https://github.com/bitcoin/bitcoin/commit/ee47ca29d6ef55650a0af63bca817c5d494f31ef (dated Aug 22nd, from PR #30684, but merged with commit https://github.com/bitcoin/bitcoin/commit/fb52023ee69c346dd101770716b6d8c0525c38aa on Sept 9th). @tdb3: CI runs always rebase the PR with respect to master, so that is expected (it helps find conflicts). Also, after CI finished, there are re-runs scheduled every now and then that will rebase on new master again.

tdb3 renamed this:
~~ci: macOS 14 CI failure `Invalid value detected for '-wallet' or '-nowallet'`~~
ci: `feature_settings.py` failed with `Invalid value detected for '-wallet' or '-nowallet'` in macOS 14 CI
on Oct 3, 2024

tdb3 commented at 9:01 pm on October 3, 2024: contributor

@tdb3 it might be helpful to share link to the full C.I logs?

Thanks. Link now included in the description.

mzumsande commented at 6:48 pm on October 4, 2024: contributor

I think the problem is that the node got stuck during shutdown for some reason:

0 node0 2024-10-02T00:15:45.784129Z [http] [src/httpserver.cpp:357] [ThreadHTTP] [http] Exited http event loop 
1 test  2024-10-02T00:55:45.674000Z TestFramework (ERROR): Assertion failed

The second line is ~40 minutes after the first, the node hasn’t completed shutdown during that time. The node is expected to fail with Invalid value detected for '-wallet' or '-nowallet' (that’s what the test is testing), so that’s not the problem.

tdb3 renamed this:
~~ci: `feature_settings.py` failed with `Invalid value detected for '-wallet' or '-nowallet'` in macOS 14 CI~~
ci: `feature_settings.py` failed in macOS 14 CI
on Oct 4, 2024

tdb3 commented at 6:53 pm on October 4, 2024: contributor

The node is expected to fail with Invalid value detected for '-wallet' or '-nowallet' (that’s what the test is testing), so that’s not the problem.

Right, the title was poorly worded. Updated

maflcko added the label CI failed on Oct 7, 2024

ci: feature_settings.py failed in macOS 14 CI #31019