Intermittent travis issue in node shutdown by RPC tests

laanwj commented at 1:43 pm on September 24, 2015: member

 0Stopping nodes
 1Traceback (most recent call last):
 2  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/wallet.py", line 254, in <module>
 3    WalletTest ().main ()
 4  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/test_framework.py", line 134, in main
 5    stop_nodes(self.nodes)
 6  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/util.py", line 218, in stop_nodes
 7    node.stop()
 8  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/authproxy.py", line 138, in __call__
 9    response = self._request('POST', self.__url.path, postdata)
10  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/authproxy.py", line 124, in _request
11    self.__conn.request(method, path, postdata, headers)
12  File "/usr/lib/python2.7/httplib.py", line 958, in request
13    self._send_request(method, url, body, headers)
14  File "/usr/lib/python2.7/httplib.py", line 992, in _send_request
15    self.endheaders(body)
16  File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
17    self._send_output(message_body)
18  File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
19    self.send(msg)
20  File "/usr/lib/python2.7/httplib.py", line 776, in send
21    self.connect()
22  File "/usr/lib/python2.7/httplib.py", line 757, in connect
23    self.timeout, self.source_address)
24  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
25    raise err
26socket.error: [Errno 111] Connection refused

I think the problem is a race condition:

stop command is sent to RPC by client
Node queues Bitcoin server stopping reply (see https://github.com/bitcoin/bitcoin/blob/master/src/rpcserver.cpp#L246)
Node shuts down
Reply never arrives at client
Client re-tries stop because it assumes the connection was slammed due to a timeout (see #6695)
Client encounters closed port, and throws ‘connection refused’ error

The “Shutdown will take long enough that the response should get back” comment may no longer ring true with the new http server.

To solve this we’ll have to make sure that the reply is always sent before the shutdown sequence is initiated.

Edit: can reproduce this reliably by putting a sleep(3) between the StartShutdown and return.

laanwj added the label Tests on Sep 24, 2015

laanwj added this to the milestone 0.12.0 on Sep 24, 2015

laanwj referenced this in commit db82ac01f2 on Sep 24, 2015

laanwj referenced this in commit d04598bd94 on Sep 24, 2015

laanwj commented at 4:35 pm on September 24, 2015: member

Addressed in#6719

laanwj referenced this in commit 5e0c221356 on Sep 28, 2015

laanwj closed this on Sep 28, 2015

DrahtBot locked this on Sep 8, 2021

Intermittent travis issue in node shutdown by RPC tests #6717