It seems like it would be a easy hack to cache our docker builds (before they look at any local code) as custom docker images.
This way, our docker builds don't need to re-install and fetch a bunch of system depends while building.
The downside is that this change would make it a little trickier to change dependencies, but the time savings would be immense, unless I'm misunderstanding what Travis is able to cache presently.
It would have to download the extra layers (even if you place everything in one layer) because docker images aren't cached, see https://docs.travis-ci.com/user/caching/#things-not-to-cache.
Not having to maintain Dockerfiles etc sounds better?
apt is slower than just downloading all packages. Sometimes it takes longer than 10 minutes.
So it would be something like:
Concept ACK in order to evaluate the real gain.
I am unsure about putting any eggs in the docker-basket. Having docker as a required dependency is not going to solve any of our problems, but rather going to cause more issues. Note that it is possible (while not recommended and experimental) to run the ci system without docker.
All we need is a faster apt, both faster downloading and faster installing packages. All it does is download some binary blobs an put them in a directory tree. It should be possible to do all of this in seconds with a fast connection, and not 10-15 minutes.
There's an apt-fast ppa which parallelizes apt? Maybe worth investigating?
It should be possible to do all of this in seconds with a fast connection, and not 10-15 minutes.
Was this discussed with someone from travis?
apt-fast is slower than vanilla apt, see https://github.com/ilikenwf/apt-fast/issues/163
I think using custom images (docker or not) for CI is a good idea, if it can speed up the builds.
I don't understand either why apt is so slow on Travis, though. Do they have their own APT mirror, or are they hammering the distro servers? If the latter maybe they're throttled somehow.
We are using apt from inside docker, so we use the same mirror that everyone else using docker (or a vanilla ubuntu) is using
If we can figure out if it is a rate limiting issue, maybe we can set up our own mirror and whitelist travis hosts?
Also I've seen that apparently there's an apt add-on which speeds up travis dependencies? https://docs.travis-ci.com/user/installing-dependencies#Installing-Packages-with-the-APT-Addon
Unsure if we can take advantage of this or not.
If we can figure out if it is a rate limiting issue, maybe we can set up our own mirror and whitelist travis hosts?
Running a caching proxy (e.g. apt-cacher-ng) would be less resource intensive than running a mirror.
Maybe even that's overkill, though. Could we download the .deb packages once, then cache them on Travis somehow? dpkg -i from files is a lot faster than apt, generally.
Edit: whoops, they explicitly mention debian packages under "things not to cache" here: https://docs.travis-ci.com/user/caching/#things-not-to-cache
We might want to do this if not for speed, for reliability.
The fewer external servers our CI depends on, the better.
I did some more apt-fast testing and found that it speeds up the download of packages in one setting: x86_64 hardware in a vanilla ubuntu docker container running on travis. All other settings (non-x86_64 hardware or non-ubuntu docker, such as debian), it would slow down. I'd guess that the default amd64 mirrors for ubuntu are over-used, whereas the others are not. So downloading in parallel will help to saturate the connection to the amd64 ubuntu mirror.
Also, for some unknown reasons, s390x is the fastest to download new software (and possibly also run the whole build on travis).
So we might want to switch to s390x for our native builds to try if that speeds up things on travis.
For local runs of the ci system, we'd need to implement some caching mechanism. Even just supporting to cache depends and the ccache would speed up local runs. A custom docker image might also speed things up, but we'd need to make sure that the image is provided for all our supported archs, not only x86_64. I don't know enough about docker on how to achieve that.
The feature request didn't seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.
Closing due to lack of interest. Pull requests with improvements are always welcome.