[test] Run extended tests once daily in Travis #10052

Travis has beta support for a new feature called cron jobs. We should hook into this to run our extended functional test suite once per day.

This requires two changes: merging the change to .travis.yml in this PR, and then enabling daily builds of the master branch through Travis settings.

For an example of a cron build see https://travis-ci.org/jnewbery/bitcoin/builds/213827649. This currently fails because:

• rpcbind_tests.py requires ipv6, which Travis currently doesn't support. I have a workaround for that on the way.
• pruning.py times out. I've now updated the .travis.yml file to exclude running pruning.

The granularity for cron jobs is very coarse - currently Travis only allows jobs to be run daily, weekly or monthly. There have been lots of requests for more flexible cron schedules in this thread so Travis may add this as an enhancement at some point.

Do you know if emails will be sent out to the maintainers on failing builds? I think we'd want to avoid that, as the extended tests on travis could end up being rather flaky.

utACK 88e3aa0 otherwise.

On Wed, Mar 22, 2017 at 4:11 PM, John Newbery notifications@github.com wrote:

Travis has beta support for a new feature called cron jobs. We should hook into this to run our extended functional test suite once per day.

This requires two changes: merging the change to .travis.yml in this PR, and then enabling daily builds of the master branch through Travis settings.

For an example of a cron build see https://travis-ci.org/jnewbery/bitcoin/builds/213827649. This currently fails because:

rpcbind_tests.py requires ipv6, which Travis currently doesn't support. I have a workaround for that on the way. pruning.py times out. I've now updated the .travis.yml file to exclude running pruning.

The granularity for cron jobs is very coarse - currently Travis only allows jobs to be run daily, weekly or monthly. There have been lots of requests for more flexible cron schedules in this thread so Travis may add this as an enhancement at some point.

---

You can view, comment on, or merge this pull request online at:

#10052

Commit Summary

Run extended tests once daily

File Changes

M .travis.yml (3)

Patch Links:

#10052.patch #10052.diff

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

I think email notifications for cron jobs are the same as for regular branch builds.

In any case, I'm not sure whether that's a bad thing. If the test is really broken, then it's better to know immediately that something's broken it, rather than whenever someone happens to run the extended tests. And if you're worried about the extended tests being unreliable, then we should make them more robust.

It is fine when the failure can be reproduced locally, though there have been cases where travis just fails out of the blue and as soon as you try to debug, it will behave as if nothing was wrong.

As you can see in the log of the cron job in your fork, bip9-softforks sometimes fails: https://travis-ci.org/jnewbery/bitcoin/jobs/213827655

I tried to reproduce this failure locally but it always passes for me...

On Wed, Mar 22, 2017 at 8:13 PM, John Newbery notifications@github.com wrote:

I think email notifications for cron jobs are the same as for regular branch builds.

In any case, I'm not sure whether that's a bad thing. If the test is really broken, then it's better to know immediately that something's broken it, rather than whenever someone happens to run the extended tests. And if you're worried about the extended tests being unreliable, then we should make them more robust.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/pull/10052#issuecomment-288508365, or mute the thread https://github.com/notifications/unsubscribe-auth/AGGmv7HhG2DsFdvilONDhkuGpOhlEREqks5roXLIgaJpZM4MlWf4 .

Yes - that could get annoying. I plan to improve travis logging by having it upload detailed logs from failed test cases to an S3 bucket. In the long run we should definitely try to make sure travis runs of extended test cases are reliable, but for now you could manually exclude those test cases which you think are unreliable?

Concept ACK

Great! Now I need to fix bip9-softforks.py :)

I believe #10072 should help with bip9-softforks.