The scheduler test currently relies on exact timings, this is not workable in the busy Travis environment (or anything but a real-time operating system) resulting in random errors.
Ideally such a test would use mock times, but this is very invasive. For now, fix it by waiting until the number of tasks left to execute is zero before tearing down the threads.