[qa] travis: cover more tests #7019

Any reason to not let travis run #3960?

What is the reason for not running all of them in Travis? Are some too slow? Perhaps just one of the test combinations could run them all?

Are some too slow?

One uses 4 GB disk space and at least half an hour time: https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/pruning.py#L9-l11

In theory travis could handle those within the 120 minute time limit but I am not sure about other limits.

Travis appears to fail running the extended tests with

Warning! This test requires 4GB of disk space and takes over 30 mins (up to 2 hours)
Mining a big blockchain of 995 blocks
Unexpected exception caught during testing: [Errno 111] Connection refused

Which is not due to a full disk, I guess. This would throw Error: Error: Disk space is low!

Running the -extended tests during a CI build/check seems to be to expansive. Especially pruning.py uses a lot of resources (and time).

The idea is to enables as much tests as possible over the non-extended tests list.

Enabling -extended in Travis seems against the concept. Adding more tests to the non-extended list should be the way to go.

The idea of the extended list is to have an option to run expansive tests locally (and it also serves as a overview of all available tests).

@jonasschnelli : Reverted to inital commit.

ACK.

utACK

On Sunday, November 15, 2015, Jonas Schnelli notifications@github.com wrote:

ACK.

— Reply to this email directly or view it on GitHub #7019 (comment).

utACK

I like the approach here -- was just about to dive into writing some additional RPC tests when I found that we have coverage in the extended tests for a few that're reported as uncovered. We should probably do an audit to figure out which of the -extended tests can reasonably be run on travis, then add some smoke tests for those that remain uncovered.

@jamesob I am working on this right now. (Needs some refactor, though)

What is the reason for not running all of them in Travis? Are some too slow? Perhaps just one of the test combinations could run them all?

The problem is that a pull request check only finishes when all combinations are finished. CI is not very useful if you have to wait hours for a travis result to come in every time, so it relies on fast tests.

The idea of the extended list is to have an option to run expansive tests locally (and it also serves as a overview of all available tests).

I think it would be useful to run the extended tests, say, daily on master. No idea if Travis can be configured like that.

I think it would be useful to run the extended tests, say, daily on master. No idea if Travis can be configured like that.

I think there is a tools that can trigger a nightly build: https://nightli.es , although not sure if its easy to provide different arguments so that the build will execute the -extended tests.

There is also a discussion here: https://github.com/travis-ci/travis-ci/issues/582

That looks useful!

By the way: ACK, receivedby.py seems to finish in 11 seconds when the cache is already there, a few seconds longer if not. So it's not close to problematic.