RPC tests are failing to start bitcoind

instagibbs commented at 3:50 pm on November 4, 2016: member

On master I’m getting this:

every time I run rpc tests batched with multiple threads(and sometimes with 1), certain tests basically fail to start: Unexpected exception caught during testing: Exception('bitcoind exited with status 1 during initialization',)

Running them individually allows them to pass. This is making it annoying to test changes locally. I have not been able to reproduce this on another machine since I don’t have access to another one at the moment.

This is on a fairly beefy Intel NUC.

fanquake added the label Tests on Nov 4, 2016

MarcoFalke commented at 4:33 pm on November 4, 2016: member

Thanks for reporting the issue! Lately I have the feeling no one is running the rpc tests anymore and travis is our only tester, which is suboptimal, as a segfault slipped through and existed for several weeks.

I think we should work to get any false positives out of the rpc test suite sooner rather than later.

Just a few points, some of which seem to have caused this Exception in the past:

Are there any “zombie” regtest nodes running?
Are there leftover folders/files in the tempdir you are using?
Is there enough space in the tempdir and RAM for the fullblocktest?

instagibbs commented at 4:48 pm on November 4, 2016: member

Are there any “zombie” regtest nodes running?

running ps -aux | grep "bitcoind" leads me to believe not

Are there leftover folders/files in the tempdir you are using?

(assuming we’re talking about /tmp) A bunch. I deleted them and nothing changed.

Is there enough space in the tempdir and RAM for the fullblocktest?

I have 32GB of RAM, and cleaning out tempdir didn’t help.

instagibbs commented at 5:36 pm on November 4, 2016: member

Ok this is also occurring with 0.13.0, which I know used to work fine. Most likely a local-to-me regression.

jnewbery commented at 5:38 pm on November 4, 2016: member

There might be some hints in debug.log to indicate why bitcoind has failed to initialize. rpc-tests.py should output something like this for a failed test:

0prioritise_transaction.py:
1Initializing test directory /tmp/<test_directory>
2INFO messages ...
3Unexpected exception caught during testing: Exception('bitcoind exited with status 1 during initialization',)
4Stopping nodes

and the temporary test directory shouldn’t be cleared up.

Is there anything in <test_directory>/node<n>/regtest/debug.log?

instagibbs commented at 5:40 pm on November 4, 2016: member

2016-11-04 17:34:53.106184 Binding RPC on address ::1 port 16080 2016-11-04 17:34:53.106314 Binding RPC on address ::1 port 16080 failed. 2016-11-04 17:34:53.106327 Binding RPC on address 127.0.0.1 port 16080 2016-11-04 17:34:53.106352 Binding RPC on address 127.0.0.1 port 16080 failed. 2016-11-04 17:34:53.106361 Unable to bind any endpoint for RPC serve

Good call. For node0, no other nodes managed to start

And I see that port is taken up by another process…. let me investigate.

instagibbs commented at 6:03 pm on November 4, 2016: member

Yep, zombie daemons not named bitcoind. Closing.

instagibbs closed this on Nov 4, 2016

MarcoFalke locked this on Sep 8, 2021

RPC tests are failing to start bitcoind #9086