When configured with --enable-debug, many tests become dramatically slower. These slow downs are particularly noticed in tests that generate a lot of blocks in separate calls, make a lot of RPC calls, or send a lot of data from the test framework’s P2P connection. This PR aims to improve the runtime of some of the slower tests and improve the overall runtime of the test runner. This has improved the runtime of the test runner from ~400s to ~140s on my computer.
The slowest test by far was wallet_import_rescan.py. This was taking ~320s. Most of that time was spent waiting for blocks to be mined and then synced to the other nodes. It was generating a new block for every new transaction it was creating in a setup loop. However it is not necessary to have one tx per block. By mining a block only every 10 txs, the runtime is improved to ~61s.
The second slowest test was feature_fee_estimation.py. This test spends most of its time waiting for RPCs to respond. I was able to improve its runtime by batching RPC requests. This has improved the runtime from ~201s to ~140s.
In feature_taproot.py, the test was constructing a Python CScript using a very large list of OP_CHECKSIGs. The constructor for the Python implementation of CScript was iterating this list in order to create a bytes from it even though a bytes could be created from it without iterating. By making the bytes before passing it into the constructor, we are able to improve this test’s runtime from ~131s to ~106s.
Although interface_rpc.py was not typically a slow test, I found that it would occasionally have a super long runtime. It typically takes ~7s, but I have observed it taking >400s to run on occasion. This longer runtime occurs more often when --enable-debug. This long runtime was caused by the “exceeding work queue” test which is really just trying to trigger a race condition. In this test, it would create a few threads and try an RPC in a loop in the hopes that eventually one of the RPCs would be added to the work queue while another was processing. It used getrpcinfo for this, but this function is fairly fast. I believe what was happening was that with --enable-debug, all of the code for receiving the RPC would often take longer to run than the RPC itself, so the majority of the requests would succeed, until we got lucky after 10’s of thousands of requests. By changing this to use a slow RPC, the race condition can be triggered more reliably, and much sooner as well. I’ve used waitfornewblock with a 500ms timeout. This improves the runtime to ~3s consistently.
The last test I’ve changed was rpc_packages.py. This test was one of the higher runtime variability tests. The main source of this variation appears to be waiting for the test node to relay a transaction to the test framework’s P2P connection. By whitelisting that peer, the variability is reduced to nearly 0.
Lastly, I’ve reordered the tests in test_runner.py to account for the slower runtimes when configured with --enable-debug. Some of the slow tests I’ve looked at were listed as being fast which was causing overall test_runner.py runtime to be extended. This change makes the test runner’s runtime be bounded by the slowest test (currently feature_fee_estimation.py with my usual config (-j 60).
