ci: cache Python wheels built by the CI #22206

Python wheels we build in the CI, such as PyZMQ or LIEF, do not get cached, and are rebuilt every CI run. This wastes time/CPU (LIEF can take some time to build). The versions of these packages rarely change, so we should be caching and reusing build output.

It looks like Cirrus even has a pip_cache key, that we may be able to take advantage of.

Excerpt from the Darwin native job:

./ci/test_run_all.sh
Setting specific values in env
Fallback to default values in env (if not yet set)
Collecting zmq
  Downloading https://files.pythonhosted.org/packages/6e/78/833b2808793c1619835edb1a4e17a023d5d625f4f97ff25ffff986d1f472/zmq-0.0.0.tar.gz
Collecting lief
  Downloading https://files.pythonhosted.org/packages/f4/db/29cf3cecc8b9997a16da6319f64039505861c9cf9ae2394bcb8c495ab3e0/lief-0.11.5.zip (15.7MB)
Collecting pyzmq (from zmq)
  Downloading https://files.pythonhosted.org/packages/99/3b/69360102db726741053d1446cbe9f7f06df7e2a6d5b805ee71841abf1cdc/pyzmq-22.1.0.tar.gz (1.2MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: pyzmq
  Building wheel for pyzmq (PEP 517): started
  Building wheel for pyzmq (PEP 517): finished with status 'done'
  Created wheel for pyzmq: filename=pyzmq-22.1.0-cp38-cp38-macosx_10_14_6_x86_64.whl size=740475 sha256=e3fe3beb65a388bafbc38cd273de33d6e340e3c9bb70edfc073eb3ebe9cd8de4
  Stored in directory: /Users/admin/Library/Caches/pip/wheels/e5/09/86/54436a8590914d725cccf13bfebf6b0dacaa976d68d299c079
Successfully built pyzmq
Building wheels for collected packages: zmq, lief
  Building wheel for zmq (setup.py): started
  Building wheel for zmq (setup.py): finished with status 'done'
  Created wheel for zmq: filename=zmq-0.0.0-cp38-none-any.whl size=1277 sha256=b7d464167b77efa4d800df1ddaa050dc005834f099428c28bba59bc7b30735f3
  Stored in directory: /Users/admin/Library/Caches/pip/wheels/44/7a/7d/ac1d865766b06f9769ac1154bf31dbb5abb3b52ecfe278247a
  Building wheel for lief (setup.py): started
  Building wheel for lief (setup.py): still running...
  Building wheel for lief (setup.py): still running...
  Building wheel for lief (setup.py): still running...

Hi @fanquake . I'd like to contribute to this. Can you assign it to me? Thanks!

You can start working on it if you want. For guidance on contributing, please read CONTRIBUTING.md before opening your pull request.

Hey, I wanted to work on this issue. But there was some problem in setting up the environment. Could you please help me here ?

See https://github.com/bitcoin/bitcoin/tree/master/ci#ci-scripts for documentation

I am getting the same problem. Installed docker.io as explained in the documentation and then tried to run default test stage using ./ci/test_run_all.sh

What does docker run -it --rm ubuntu:focal give?

When i run docker run -it --rm ubuntu:focal it gives - docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: dial unix /var/run/docker.sock: connect: permission denied. When i run it with sudo then nothing happens. The problem still persists.

I was working on the issue and something strange is happening. Even when there is cache hit, it is still rebuilding the wheel. Can anyone help me understand why this could be happening?

When i run docker run -it --rm ubuntu:focal it gives - docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: dial unix /var/run/docker.sock: connect: permission denied. When i run it with sudo then nothing happens. The problem still persists.

Try adding user to docker group using command sudo usermod -a -G docker <username> and then restart your system.

I was working on the issue and something strange is happening. Even when there is cache hit, it is still rebuilding the wheel. Can anyone help me understand why this could be happening?

So i figured out this was happening because default version of pip in macos was 19.2.3 which doesn't caches the built wheels and upgrading it to the newest version should solve the issue. Now, after upgrading, pip is directly fetching pre-built wheels and taking just few seconds to install the packages. So i think instead of caching we should just upgrade the pip version before installing these packages. @fanquake, what are your views?

Hey

When i run docker run -it --rm ubuntu:focal it gives - docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: dial unix /var/run/docker.sock: connect: permission denied. When i run it with sudo then nothing happens. The problem still persists.

Try adding user to docker group using command sudo usermod -a -G docker <username> and then restart your system.

I tried this but it shows this (i m working on wsl ). Could u help me out here?

Hey

When i run docker run -it --rm ubuntu:focal it gives - docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: dial unix /var/run/docker.sock: connect: permission denied. When i run it with sudo then nothing happens. The problem still persists.

Try adding user to docker group using command sudo usermod -a -G docker <username> and then restart your system.

I tried this but it shows this (i m working on wsl ). Could u help me out here?

You have mistyped the command