ci: Upgrading pip version in macos environment #22353

pull singlatushar07 wants to merge 1 commits into bitcoin:master from singlatushar07:22206-upgrade-pip changing 1 files +1 −0

singlatushar07 commented at 7:43 PM on June 27, 2021: contributor

During each CI run, in macos native environment, python packages lief and zmq are rebuilt everytime which wastes a lot of resources and time and fixes #22206. The latest version of pip directly fetches pre-built binaries. Through this commit pip version is upgraded in macos environment before installation of these packages.
singlatushar07 renamed this:
~~Upgrading pip version in macos environment~~
ci: Upgrading pip version in macos environment
on Jun 27, 2021
singlatushar07 force-pushed on Jun 27, 2021
DrahtBot added the label Tests on Jun 27, 2021
in ci/test/04_install.sh:14 in f003109a0b outdated
```
  10 | @@ -11,6 +11,7 @@ if [[ $QEMU_USER_CMD == qemu-s390* ]]; then
  11 |  fi
  12 |  
  13 |  if [ "$CI_OS_NAME" == "macos" ]; then
  14 | +  sudo -H pip3 install --upgrade pip
```
fanquake commented at 2:38 AM on June 28, 2021:

Is using sudo -H definitely needed here?

singlatushar07 commented at 6:21 AM on June 28, 2021:

Yes sudo -H is necessary, otherwise if I am upgrading it using --user flag, it is giving some deprecated warning

singlatushar07 commented at 6:24 AM on June 28, 2021:

Also for multiprocess build, pip is directly being downloaded from ubuntu repositories, as will the case for most of the new ubuntu based builds. So, I think we should not upgrade its version in every CI build.

fanquake commented at 2:40 AM on June 28, 2021: member

Concept ACK. From the CI output, it looks like this solves the issue, and that LIEF is being downloaded and installed, rather than compiled. Note that zmq is still built. CI runtime is also reduced:

Collecting pip
  Downloading https://files.pythonhosted.org/packages/47/ca/f0d790b6e18b3a6f3bd5e80c2ee4edbb5807286c21cdd0862ca933f751dd/pip-21.1.3-py3-none-any.whl (1.5MB)
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-21.1.3
Collecting zmq
  Downloading zmq-0.0.0.zip (2.2 kB)
Collecting lief
  Downloading lief-0.11.5-cp38-cp38-macosx_10_14_x86_64.whl (2.9 MB)
Collecting pyzmq
  Downloading pyzmq-22.1.0-cp38-cp38-macosx_10_9_x86_64.whl (1.2 MB)
Building wheels for collected packages: zmq
  Building wheel for zmq (setup.py): started
  Building wheel for zmq (setup.py): finished with status 'done'
  Created wheel for zmq: filename=zmq-0.0.0-py3-none-any.whl size=1275 sha256=b32226d8d3b9ee360cfafa1f11785b47fcacde346dc2970aa9bdc15df7e174b1
  Stored in directory: /Users/admin/Library/Caches/pip/wheels/3d/9b/51/ce9258d99aa48f7d809437392742f760d9ce7537079f5be719
Successfully built zmq
Installing collected packages: pyzmq, zmq, lief
Successfully installed lief-0.11.5 pyzmq-22.1.0 zmq-0.0.0

Through this commit pip version is upgraded in macos environment

Is there any reason to only do this in the macOS CI? Pip is used in the other CIs (i.e multiprocess build, and there will be more in future), and if they aren't already using the latest version, I assume they would also benefit from upgrading? It would also be good to just remain as consistent as possible across the environments.

Note that your commit message can also use the ci: prefix.

ci: Upgrading pip version in macos environment

During each CI run, for macos native environment, python packages lief
and zmq are rebuilt everytime which wastes a lot of resources and time.
The latest version of pip directly fetches pre-built binaries. Through
this commit pip version is upgraded in macos environment before
installation of these packages.

057750c09d

singlatushar07 force-pushed on Jun 28, 2021
hebasto commented at 4:56 PM on June 28, 2021: member

@singlatushar07

The latest version of pip directly fetches pre-built binaries.

Could you provide a link with this feature described?
singlatushar07 commented at 5:40 AM on June 29, 2021: contributor

@singlatushar07

The latest version of pip directly fetches pre-built binaries.

Could you provide a link with this feature described? @hebasto I couldn't find it in pip documentation. I found that pip is directly fetching wheels while installing lief on my system and observed that version of pip in macos environment is 19.2.3 and while my laptop has version 20.0.2. So i decided to upgrade the pip version in the CI environment and it worked.
fanquake commented at 3:26 AM on June 30, 2021: member

@MarcoFalke can you take a look re changes to the CI.
MarcoFalke commented at 6:27 AM on June 30, 2021: member

cr ACK 057750c09d0a8331c33966d2cc2285ef82f08af8
fanquake merged this on Jun 30, 2021
fanquake closed this on Jun 30, 2021
sidhujag referenced this in commit deab286659 on Jun 30, 2021
singlatushar07 deleted the branch on Jun 30, 2021
singlatushar07 restored the branch on Jun 30, 2021
gwillen referenced this in commit f953b53465 on Jun 1, 2022
DrahtBot locked this on Aug 18, 2022

Contributors

Labels

Linked (view graph)

#22206 ci: cache Python wheels built by the CI