[NO MERGE] Travis: Run nightly test suite #7148

Ideally we want to detect fails in the extended suite early (i.e. in the PR which caused the fail). Though, this would double the runtime of the rpc tests in every PR from 5 to 10 minutes (with pruning.py disabled).

Also, we'd want to collect --coverage data until all rpcs are covered and then fail on uncovered (new) rpcs.

For now, every 24 hours a fresh nightly build is triggered within this PR.

Current status:

• Disable usual rpc tests and make check, otherwise travis will block the build exceeding the 2 hour max CPU time.
• Disable pruning.py
• Run extended coverage rpc tests once with CPPFLAGS=-DDEBUG_LOCKORDER
• Verify subtrees
• Enable qt4 build

Travis vanilla will not support nightly builds, but there are 3rd party apps available if this is a direction to take: https://github.com/travis-ci/travis-ci/issues/582#issuecomment-10366023 https://github.com/travis-ci/travis-ci/issues/582#issuecomment-123484647

Am i right: every 24h this PR get's updated with the current master (over an external script) and makes this PR act as a nightly build?

Yes, pretty much like that right now.

Seems like a good idea.

Nice, looks like this caught it's first fish. (I should set up an alert or something)

https://travis-ci.org/bitcoin/bitcoin/builds/95840859 failed with

Running 2nd level testscript maxuploadtarget.py ...
start_node: bitcoind started, calling bitcoin-cli -rpcwait getblockcount
start_node: calling bitcoin-cli -rpcwait getblockcount returned
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11523
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11523
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11523
Assertion failed: 68 != 69
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/test_framework.py", line 135, in main
    self.run_test()
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/maxuploadtarget.py", line 203, in run_test
    assert_equal(test_nodes[0].block_receive_map[big_old_block], i+1)
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/util.py", line 394, in assert_equal
    raise AssertionError("%s != %s"%(str(thing1),str(thing2)))
Stopping nodes
Cleaning up
Failed
Traceback (most recent call last):
  File "qa/pull-tester/rpc-tests.py", line 255, in <module>
    runtests()
  File "qa/pull-tester/rpc-tests.py", line 174, in runtests
    rpcTestDir + testScriptsExt[i] + flags, shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/maxuploadtarget.py --srcdir /home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/src --coveragedir /tmp/coveragexHdFgP ' returned non-zero exit status 1

@sdaftuar maxuploadtarget failed again. Mind to look at this? (I couldn't reproduce locally)

Running 2nd level testscript maxuploadtarget.py ...
start_node: bitcoind started, calling bitcoin-cli -rpcwait getblockcount
start_node: calling bitcoin-cli -rpcwait getblockcount returned
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11593
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11593
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11593
Assertion failed: 2 != 3
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/test_framework.py", line 135, in main
    self.run_test()
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/maxuploadtarget.py", line 203, in run_test
    assert_equal(test_nodes[0].block_receive_map[big_old_block], i+1)
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/util.py", line 394, in assert_equal
    raise AssertionError("%s != %s"%(str(thing1),str(thing2)))
Stopping nodes
Cleaning up
Failed
Traceback (most recent call last):
  File "qa/pull-tester/rpc-tests.py", line 256, in <module>
    runtests()
  File "qa/pull-tester/rpc-tests.py", line 175, in runtests
    rpcTestDir + testScriptsExt[i] + flags, shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/maxuploadtarget.py --srcdir /home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/src --coveragedir /tmp/coveragesoVFH4 ' returned non-zero exit status 1

@MarcoFalke Strange, I don't understand how that error can happen, but one thing I noticed is that there should be a with mininode_lock: guarding the assert_equal(test_nodes[0].block_receive_map[big_old_block]) line. Still, I can't come up with a logical scenario where there's actually a race condition there (but acquiring the lock is the right way to ensure there is no race, since both threads are updating that variable).

If we can't come up with other ideas, perhaps we can make that fix and see if the problem recurs...

@MarcoFalke can you please include #7142 into this?

@arowser Have you even looked at this patch yourself? It is not intended to be merged, as it is automatically retriggered every 24h to detect bugs only caught by the extended tests. Please stop pinging every single pull request. Repository maintainers merge when there is sufficient agreement and review done, but they are not the only ones who do reviews. If you want to help merging things, please review yourself.

@arowser Furthermore, I consider the many comments you've left as spam, and I'll delete them when I encounter them. Please don't do that again, or I'll report it as abuse.

@sipa Sorry to everyone, I just try to create a jenkins build robot in for every pull request to support more build type that not build in travis like PPC/MIPS, but the jenkins robot auto add comments by error configure. I have remove all these comments now, sorry again for these spam comments.

Noting failure in feefilter:

  File "/home/travis/build/bitcoin/bitcoin/build/../qa/rpc-tests/p2p-feefilter.py", line 101, in run_test

    assert(allInvsMatch(txids, test_node))

@MarcoFalke I figured out the feefilter issue. Looks like sometimes the second-to-last test generates transactions which don't relay within 10 seconds, and then get relayed after the feefilter is updated in the last test.

@MarcoFalke, what is the status of this pull request? Do you wish to rebase?

This pull is not intended to be merged, it just re-pushes (and re-bases) regularly to run Travis w/ full test suite, which is too slow for automatic use.

Maybe the PR title should be updated to indicate this?

On February 20, 2017 4:54:46 AM EST, "Wladimir J. van der Laan" notifications@github.com wrote:

This pull is not intended to be merged, it just re-pushes (and re-bases) regularly to run Travis w/ full test suite, which is too slow for automatic use.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/bitcoin/bitcoin/pull/7148#issuecomment-281034479

Now that Travis supports cron jobs, can this be replaced with something like #10052? I'm happy to help merging them if you think that's useful.

I believe this can be closed now?

I think so - @MarcoFalke please reopen if necessary.

This seems to include tests that aren't covered by the travis cron job, namely valgrind, testing salvagewallet and a new compatibility functional test. @MarcoFalke - do you think those should be merged into master to be run by the cron job? If so, I'm happy to help with merging/reviewing.

@jnewbery Most of the code here is just a reminder for myself to look further into that topic. Not really ready to be merged or put into our travis yaml, as the diff needs to be adjusted every couple of weeks.