[test] [POC] Stop and start bitcoind nodes asynchronously in functional tests #10055

jnewbery commented at 8:03 PM on March 22, 2017: member

This uses the new asyncio syntax in Python3.5, so it may not be suitable for merging, but it's definitely worth considering.

The functional tests spend an large amount of time in stop_node() and start_node(). Many of those calls are through stop_nodes() and start_nodes(), which stop and start the nodes serially. We could speed up the test case execution significantly by stopping and starting the nodes in parallel.

This PR uses asyncio event loops and co-routines to stop and start the nodes in parallel. It changes the internal implementation of stop_nodes() and start_nodes() and exposes the exact same interface to the test cases. I expect some test cases could be rewritten to use {start,start}_nodes() in place of {start,stop}_node() to further improve execution time.

I tested this locally, and running with this PR reduces elapsed run time by about 20% when running 4 jobs in parallel:

Without this PR:

Run 1:

TEST                           | PASSED | DURATION

wallet-hd.py                   | True   | 18 s
fundrawtransaction.py          | True   | 40 s
segwit.py                      | True   | 14 s
p2p-compactblocks.py           | True   | 66 s
wallet-accounts.py             | True   | 3 s
p2p-fullblocktest.py           | True   | 117 s
wallet.py                      | True   | 66 s
wallet-dump.py                 | True   | 5 s
sendheaders.py                 | True   | 27 s
p2p-segwit.py                  | True   | 64 s
listtransactions.py            | True   | 39 s
importmulti.py                 | True   | 10 s
zapwallettxes.py               | True   | 15 s
mempool_limit.py               | True   | 4 s
walletbackup.py                | True   | 167 s
merkle_blocks.py               | True   | 10 s
abandonconflict.py             | True   | 18 s
receivedby.py                  | True   | 22 s
bip68-112-113-p2p.py           | True   | 20 s
rawtransactions.py             | True   | 15 s
mempool_resurrect_test.py      | True   | 3 s
reindex.py                     | True   | 12 s
txn_doublespend.py --mineblock | True   | 10 s
txn_clone.py                   | True   | 10 s
mempool_spendcoinbase.py       | True   | 2 s
rest.py                        | True   | 11 s
httpbasics.py                  | True   | 8 s
multi_rpc.py                   | True   | 3 s
mempool_reorg.py               | True   | 14 s
signrawtransactions.py         | True   | 3 s
decodescript.py                | True   | 2 s
proxy_test.py                  | True   | 11 s
getchaintips.py                | True   | 30 s
disablewallet.py               | True   | 2 s
blockchain.py                  | True   | 5 s
p2p-mempool.py                 | True   | 3 s
keypool.py                     | True   | 7 s
prioritise_transaction.py      | True   | 5 s
invalidblockrequest.py         | True   | 4 s
invalidtxrequest.py            | True   | 3 s
nodehandling.py                | True   | 20 s
signmessages.py                | True   | 3 s
p2p-versionbits-warning.py     | True   | 8 s
preciousblock.py               | True   | 8 s
importprunedfunds.py           | True   | 6 s
nulldummy.py                   | True   | 3 s
rpcnamedargs.py                | True   | 2 s
p2p-leaktests.py               | True   | 8 s
bumpfee.py                     | True   | 15 s
import-rescan.py               | True   | 16 s
zmq_test.py                    | True   | 12 s
listsinceblock.py              | True   | 30 s

ALL                            | True   | 1019 s (accumulated)

Runtime: 269 s

Run 2:

...
ALL                            | True   | 976 s (accumulated)

Runtime: 257 s

Run 3:

ALL                            | True   | 979 s (accumulated)

Runtime: 259 s

Average cumulative: 991s, average elapsed: 262s

With PR:

Run 1:

TEST                           | PASSED | DURATION

wallet-hd.py                   | True   | 16 s
fundrawtransaction.py          | True   | 56 s
segwit.py                      | True   | 8 s
p2p-compactblocks.py           | True   | 61 s
wallet-accounts.py             | True   | 3 s
wallet.py                      | True   | 30 s
wallet-dump.py                 | True   | 5 s
p2p-fullblocktest.py           | True   | 119 s
walletbackup.py                | True   | 128 s
listtransactions.py            | True   | 29 s
importmulti.py                 | True   | 6 s
zapwallettxes.py               | True   | 10 s
p2p-segwit.py                  | True   | 59 s
mempool_limit.py               | True   | 5 s
merkle_blocks.py               | True   | 4 s
sendheaders.py                 | True   | 24 s
receivedby.py                  | True   | 9 s
abandonconflict.py             | True   | 12 s
mempool_resurrect_test.py      | True   | 3 s
rawtransactions.py             | True   | 14 s
txn_doublespend.py --mineblock | True   | 4 s
txn_clone.py                   | True   | 4 s
bip68-112-113-p2p.py           | True   | 19 s
reindex.py                     | True   | 13 s
mempool_spendcoinbase.py       | True   | 3 s
rest.py                        | True   | 5 s
httpbasics.py                  | True   | 3 s
getchaintips.py                | True   | 10 s
multi_rpc.py                   | True   | 3 s
mempool_reorg.py               | True   | 8 s
proxy_test.py                  | True   | 3 s
signrawtransactions.py         | True   | 3 s
decodescript.py                | True   | 3 s
blockchain.py                  | True   | 3 s
disablewallet.py               | True   | 3 s
p2p-mempool.py                 | True   | 3 s
prioritise_transaction.py      | True   | 4 s
keypool.py                     | True   | 7 s
invalidblockrequest.py         | True   | 4 s
nodehandling.py                | True   | 13 s
invalidtxrequest.py            | True   | 4 s
preciousblock.py               | True   | 3 s
importprunedfunds.py           | True   | 3 s
signmessages.py                | True   | 3 s
nulldummy.py                   | True   | 3 s
p2p-versionbits-warning.py     | True   | 9 s
rpcnamedargs.py                | True   | 3 s
import-rescan.py               | True   | 5 s
bumpfee.py                     | True   | 10 s
zmq_test.py                    | True   | 6 s
p2p-leaktests.py               | True   | 8 s
listsinceblock.py              | True   | 10 s

ALL                            | True   | 786 s (accumulated)

Runtime: 201 s

Run 2:

...
ALL                            | True   | 876 s (accumulated)

Runtime: 223 s

Run 3:

ALL                            | True   | 778 s (accumulated)

Runtime: 198 s

Average cumulative: 813s, average elapsed: 207s

h/t to @ryanofsky for suggesting parallelizing the stop/start functions.

Stop and start bitcoind nodes asynchronously in functional tests 3c42fa6bf8

Install python3.5 on travis 7a27d9f5a9

in test/functional/test_framework/util.py:349 in 3c42fa6bf8 outdated

 350 |      url = rpc_url(i, rpchost)
 351 | -    wait_for_bitcoind_start(bitcoind_processes[i], url, i)
 352 | -    logger.debug("initialize_chain: RPC successfully started")
 353 | +
 354 | +    while True:
 355 | +        if bitcoind_processes[i].poll() is not None:

ryanofsky commented at 8:32 PM on March 22, 2017:

This whole block of code except the sleep is duplicated from wait_for_bitcoind_start. It would be good to factor out the common code into a function to avoid duplication.

ryanofsky commented at 8:56 PM on March 22, 2017: member

I think the parallelization is good, but would point out that there's actually no reason to use coroutines and asyncio here, because this doing no asynchronous io, only polling and calling asyncio.sleep.

So you could achieve the exact same effect with simpler control flow and no asyncio dependency just by doing:

processes = [Popen(args) for args in node_args]
wait_urls = [rpc_url(i) for i in range(num_nodes)]
while wait_urls:
    wait_urls = [url for url in wait_urls if not is_bitcoind_ready(url)]
    time.sleep(0.25)

MarcoFalke commented at 9:41 PM on March 22, 2017: member

I think it is good to keep python 3.4 as minimum version for now. Especially, since we plan to run the functional tests on make check.

fanquake added the label Tests on Mar 22, 2017

jnewbery closed this on Mar 23, 2017

DrahtBot locked this on Sep 8, 2021