Intermittent travis issue in node shutdown by RPC tests #6717

1. 
   laanwj commented at 1:43 PM on September 24, 2015: member

   Stopping nodes
   Traceback (most recent call last):
     File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/wallet.py", line 254, in <module>
       WalletTest ().main ()
     File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/test_framework.py", line 134, in main
       stop_nodes(self.nodes)
     File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/util.py", line 218, in stop_nodes
       node.stop()
     File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/authproxy.py", line 138, in __call__
       response = self._request('POST', self.__url.path, postdata)
     File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/authproxy.py", line 124, in _request
       self.__conn.request(method, path, postdata, headers)
     File "/usr/lib/python2.7/httplib.py", line 958, in request
       self._send_request(method, url, body, headers)
     File "/usr/lib/python2.7/httplib.py", line 992, in _send_request
       self.endheaders(body)
     File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
       self._send_output(message_body)
     File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
       self.send(msg)
     File "/usr/lib/python2.7/httplib.py", line 776, in send
       self.connect()
     File "/usr/lib/python2.7/httplib.py", line 757, in connect
       self.timeout, self.source_address)
     File "/usr/lib/python2.7/socket.py", line 571, in create_connection
       raise err
   socket.error: [Errno 111] Connection refused
   

   I think the problem is a race condition:

   • stop command is sent to RPC by client
   • Node queues Bitcoin server stopping reply (see https://github.com/bitcoin/bitcoin/blob/master/src/rpcserver.cpp#L246)
   • Node shuts down
   • Reply never arrives at client
   • Client re-tries stop because it assumes the connection was slammed due to a timeout (see #6695)
   • Client encounters closed port, and throws 'connection refused' error

   The "Shutdown will take long enough that the response should get back" comment may no longer ring true with the new http server.

   To solve this we'll have to make sure that the reply is always sent before the shutdown sequence is initiated.

   Edit: can reproduce this reliably by putting a sleep(3) between the StartShutdown and return.

2. laanwj added the label Tests on Sep 24, 2015
3. laanwj added this to the milestone 0.12.0 on Sep 24, 2015
4. laanwj referenced this in commit db82ac01f2 on Sep 24, 2015
5. laanwj referenced this in commit d04598bd94 on Sep 24, 2015
6. 
   laanwj commented at 4:35 PM on September 24, 2015: member

   Addressed in#6719

7. laanwj referenced this in commit 5e0c221356 on Sep 28, 2015
8. laanwj closed this on Sep 28, 2015

   ---

9. DrahtBot locked this on Sep 8, 2021

Milestone
0.12.0