ci: cache Python wheels built by the CI

fanquake commented at 8:17 am on June 10, 2021: member

Python wheels we build in the CI, such as PyZMQ or LIEF, do not get cached, and are rebuilt every CI run. This wastes time/CPU (LIEF can take some time to build). The versions of these packages rarely change, so we should be caching and reusing build output.

It looks like Cirrus even has a pip_cache key, that we may be able to take advantage of.

Excerpt from the Darwin native job:

 0./ci/test_run_all.sh
 1Setting specific values in env
 2Fallback to default values in env (if not yet set)
 3Collecting zmq
 4  Downloading https://files.pythonhosted.org/packages/6e/78/833b2808793c1619835edb1a4e17a023d5d625f4f97ff25ffff986d1f472/zmq-0.0.0.tar.gz
 5Collecting lief
 6  Downloading https://files.pythonhosted.org/packages/f4/db/29cf3cecc8b9997a16da6319f64039505861c9cf9ae2394bcb8c495ab3e0/lief-0.11.5.zip (15.7MB)
 7Collecting pyzmq (from zmq)
 8  Downloading https://files.pythonhosted.org/packages/99/3b/69360102db726741053d1446cbe9f7f06df7e2a6d5b805ee71841abf1cdc/pyzmq-22.1.0.tar.gz (1.2MB)
 9  Installing build dependencies: started
10  Installing build dependencies: finished with status 'done'
11  Getting requirements to build wheel: started
12  Getting requirements to build wheel: finished with status 'done'
13  Installing backend dependencies: started
14  Installing backend dependencies: finished with status 'done'
15    Preparing wheel metadata: started
16    Preparing wheel metadata: finished with status 'done'
17Building wheels for collected packages: pyzmq
18  Building wheel for pyzmq (PEP 517): started
19  Building wheel for pyzmq (PEP 517): finished with status 'done'
20  Created wheel for pyzmq: filename=pyzmq-22.1.0-cp38-cp38-macosx_10_14_6_x86_64.whl size=740475 sha256=e3fe3beb65a388bafbc38cd273de33d6e340e3c9bb70edfc073eb3ebe9cd8de4
21  Stored in directory: /Users/admin/Library/Caches/pip/wheels/e5/09/86/54436a8590914d725cccf13bfebf6b0dacaa976d68d299c079
22Successfully built pyzmq
23Building wheels for collected packages: zmq, lief
24  Building wheel for zmq (setup.py): started
25  Building wheel for zmq (setup.py): finished with status 'done'
26  Created wheel for zmq: filename=zmq-0.0.0-cp38-none-any.whl size=1277 sha256=b7d464167b77efa4d800df1ddaa050dc005834f099428c28bba59bc7b30735f3
27  Stored in directory: /Users/admin/Library/Caches/pip/wheels/44/7a/7d/ac1d865766b06f9769ac1154bf31dbb5abb3b52ecfe278247a
28  Building wheel for lief (setup.py): started
29  Building wheel for lief (setup.py): still running...
30  Building wheel for lief (setup.py): still running...
31  Building wheel for lief (setup.py): still running...

fanquake added the label Tests on Jun 10, 2021

fanquake added the label good first issue on Jun 10, 2021

erichamers commented at 2:41 pm on June 10, 2021: none

Hi @fanquake . I’d like to contribute to this. Can you assign it to me? Thanks!

MarcoFalke commented at 2:43 pm on June 10, 2021: member

You can start working on it if you want. For guidance on contributing, please read CONTRIBUTING.md before opening your pull request.

MarcoFalke deleted a comment on Jun 21, 2021

aditya113141 commented at 11:58 am on June 24, 2021: none

Hey, I wanted to work on this issue. But there was some problem in setting up the environment. Could you please help me here ? Screenshot from 2021-06-16 15-07-36

MarcoFalke commented at 12:20 pm on June 24, 2021: member

See https://github.com/bitcoin/bitcoin/tree/master/ci#ci-scripts for documentation

aditya113141 commented at 5:33 am on June 26, 2021: none

I am getting the same problem. Installed docker.io as explained in the documentation and then tried to run default test stage using

./ci/test_run_all.sh

MarcoFalke commented at 6:52 am on June 26, 2021: member

What does docker run -it --rm ubuntu:focal give?

aditya113141 commented at 8:29 am on June 26, 2021: none

When i run docker run -it --rm ubuntu:focal it gives -

docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: dial unix /var/run/docker.sock: connect: permission denied.

When i run it with sudo then nothing happens. The problem still persists.

singlatushar07 commented at 8:42 am on June 27, 2021: contributor

I was working on the issue and something strange is happening. Even when there is cache hit, it is still rebuilding the wheel. Can anyone help me understand why this could be happening?

singlatushar07 commented at 8:50 am on June 27, 2021: contributor

When i run docker run -it --rm ubuntu:focal it gives - docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: dial unix /var/run/docker.sock: connect: permission denied. When i run it with sudo then nothing happens. The problem still persists.

Try adding user to docker group using command sudo usermod -a -G docker <username> and then restart your system.

singlatushar07 commented at 5:44 pm on June 27, 2021: contributor

I was working on the issue and something strange is happening. Even when there is cache hit, it is still rebuilding the wheel. Can anyone help me understand why this could be happening?

So i figured out this was happening because default version of pip in macos was 19.2.3 which doesn’t caches the built wheels and upgrading it to the newest version should solve the issue. Now, after upgrading, pip is directly fetching pre-built wheels and taking just few seconds to install the packages. So i think instead of caching we should just upgrade the pip version before installing these packages. @fanquake, what are your views?

smaulik13 commented at 9:28 am on June 28, 2021: none

Hey

When i run docker run -it --rm ubuntu:focal it gives - docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: dial unix /var/run/docker.sock: connect: permission denied. When i run it with sudo then nothing happens. The problem still persists.

Try adding user to docker group using command sudo usermod -a -G docker <username> and then restart your system.

I tried this but it shows this (i m working on wsl ). Could u help me out here? Screenshot (96)

singlatushar07 commented at 5:44 am on June 29, 2021: contributor

Hey

When i run docker run -it --rm ubuntu:focal it gives - docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: dial unix /var/run/docker.sock: connect: permission denied. When i run it with sudo then nothing happens. The problem still persists.

Try adding user to docker group using command sudo usermod -a -G docker <username> and then restart your system.

I tried this but it shows this (i m working on wsl ). Could u help me out here?

You have mistyped the command

fanquake closed this on Jun 30, 2021

sidhujag referenced this in commit deab286659 on Jun 30, 2021

DrahtBot locked this on Aug 18, 2022

ci: cache Python wheels built by the CI #22206