The scheduler_tests/mockforward test introduced in #18037 sometimes fails on the x86_64 Linux machine with no wallet.
This is an attempt to fix by changing the order of events in the tests & starting the scheduler thread before queueing events.
Unfortunately I'm unable to reproduce locally, so I'm opening this PR & would like to get a few green runs to see if this is actually the fix.
Just as a general hint for the approach of "get a few green runs", it's also possible to register at travis-ci.org with your github account and add your forked bitcoin repository there: then on every push the CI will be triggered as well. This proved to be very helpful for me personally, as check before I opened PRs, as I wrongly assumed for some reason that the CI runs can only be triggered here on the main bitcoin repository.
thanks for the tip @theStack !
so you were able to get travis passing on your forked repo? when I tried previously I wouldn't get meaningful results, eg. they would frequently time out or error for unrelated reasons. But I didn't investigate too closely, so I can take another look
so you were able to get travis passing on your forked repo? when I tried previously I wouldn't get meaningful results, eg. they would frequently time out or error for unrelated reasons. But I didn't investigate too closely, so I can take another look
It could be that on a first build you have to restart the job manually from the Travis dashboard, if a message like this appears at the very bottom of the job log:
Error! Initial build successful, but not enough time remains to run later build stages and tests.
See https://docs.travis-ci.com/user/customizing-the-build#build-timeouts . Please manually re-run
this job by using the travis restart button. The next run should not time out because the build cache
has been saved.
Other than that, after this initial hurdle I think a Travis run here should behave the same as a run from your bitcoin fork -- at least that's my experience :)
171 | @@ -171,8 +172,6 @@ BOOST_AUTO_TEST_CASE(mockforward) 172 | size_t num_tasks = scheduler.getQueueInfo(first, last); 173 | BOOST_CHECK_EQUAL(num_tasks, 3ul); 174 | 175 | - std::thread scheduler_thread([&]() { scheduler.serviceQueue(); }); 176 | - 177 | // bump the scheduler forward 5 minutes 178 | scheduler.MockForward(boost::chrono::seconds(5*60));
It looks like this line (or any line after it, but before the next BOOST_CHECK) still fails. E.g.
Running 3 test cases...
Test cases order is shuffled using seed: 902263820
Entering test module "Bitcoin Core Test Suite"
test/scheduler_tests.cpp(11): Entering test suite "scheduler_tests"
test/scheduler_tests.cpp(156): Entering test case "mockforward"
terminate called after throwing an instance of 'boost::wrapexcept<boost::condition_error>'
what(): boost::condition_variable::do_wait_until failed in pthread_cond_timedwait: Invalid argument
unknown location(0): fatal error: in "scheduler_tests/mockforward": signal: SIGABRT (application abort requested)
test/scheduler_tests.cpp(173): last checkpoint
test/scheduler_tests.cpp(156): Leaving test case "mockforward"; testing time: 430us
test/scheduler_tests.cpp(112): Entering test case "singlethreadedscheduler_ordered"
test/scheduler_tests.cpp(112): Leaving test case "singlethreadedscheduler_ordered"; testing time: 7233us
test/scheduler_tests.cpp(38): Entering test case "manythreads"
test/scheduler_tests.cpp(38): Leaving test case "manythreads"; testing time: 8082us
test/scheduler_tests.cpp(11): Leaving test suite "scheduler_tests"; testing time: 15837us
Leaving test module "Bitcoin Core Test Suite"; testing time: 15956us
I tried on three machines and could not reproduce the issue locally
From @jonasschnelli on IRC: Fails also on https://bitcoinbuilds.org/?job=cda0e018-8437-4dea-80b8-283fd445ec8a
@jonasschnelli Can the build be reproduced locally? Is it using depends or system boost?
I looked a bit into it. This PR seems still to fail on bitcoinbuilds.org: https://bitcoinbuilds.org/index.php?job=2f3c8572-cbaf-4cab-a2d2-20809d713084
What I observed was that only the depends-build configuration fail (system-libs configuration works). If I clear the ccache-cache, the build runs successful.
Maybe that helps identifying the issue.
If I clear the ccache-cache, the build runs successful.
I think the failure is intermittent, so it might be hard to draw conclusions early.
I think the failure is intermittent, so it might be hard to draw conclusions early.
It seems to fail on bitcoinbuilds.org 100% on "Linux x86_64 depends" "Linux 32 depends" when using ccache/dependency cache.
I am finally able to reproduce on a single CPU instance on gce n1-standard-1 (1 vCPU, 3.75 GB memory)
I am using depends with NO_QT=1 NO_WALLET=1 and ccache. The bug is only reproducible once after each reboot with: cd bitcoin/ && while sudo bash -c 'rm -f ./src/test/test_bitcoin && (echo 1 > /proc/sys/vm/drop_caches) && make -j 2 check' ; do true ; done
I could not reproduce it when attaching gdb, running in valgrind or when enabling core dumps (in combination with #18183 )
@jonasschnelli @MarcoFalke I'm unable to reproduce :(
What toolchain are you compiling using? Is there any theory as to why the use of ccache would matter?
Also, are you able to trigger it when doing src/test/test_bitcoin -t scheduler_tests?
thank you all for your help with this bug !
I opened issue #18227 to track findings & continue investigation