CI: Failed pulls from docker.io causing jobs to fail

ismaelsadeeq commented at 9:07 pm on February 4, 2025: member

0[15:50:52.950] STEP 1/7: FROM docker.io/debian:bookworm
1[15:50:52.956] Trying to pull docker.io/library/debian:bookworm...
2[15:51:23.200] time="2025-02-04T15:51:23-05:00" level=warning msg="Failed, retrying in 2s ... (1/3). Error: initializing source docker://debian:bookworm: reading manifest bookworm in docker.io/library/debian: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"
3[15:51:55.409] time="2025-02-04T15:51:55-05:00" level=warning msg="Failed, retrying in 2s ... (2/3). Error: initializing source docker://debian:bookworm: reading manifest bookworm in docker.io/library/debian: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"
4[15:52:57.833] time="2025-02-04T15:52:57-05:00" level=warning msg="Failed, retrying in 2s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://debian:bookworm: reading manifest sha256:f6008b2d1c096556ec301a0c76fe0b40657594ffcfa43aaf63ca9d40767f9a63 in docker.io/library/debian: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"
5[15:53:30.043] Error: creating build container: initializing source docker://debian:bookworm: reading manifest bookworm in docker.io/library/debian: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
6[15:53:30.051] 
7[15:53:30.051] Exit status: 125

https://cirrus-ci.com/task/6014627928080384 https://github.com/bitcoin/bitcoin/pull/31384/checks?check_run_id=36674073808

maflcko commented at 9:28 pm on February 4, 2025: member

This should only happen when a new ci image build is triggered without the images being cached from previous builds. So it should be rare enough to not happen often. When it happens, it should clear itself after an hour or two.

An alternative would be to use (or run) a mirror that has a higher rate limit, but I am not sure if this is worth it.

I’ve re-run the CI tasks for now.

maflcko added the label CI failed on Feb 4, 2025

Sjors commented at 1:12 pm on February 5, 2025: member

@maflcko can you re-run #30975 as well?

maflcko commented at 1:15 pm on February 5, 2025: member

I’ve rebooted the CI machines yesterday and today, which caused the issues here. I’ll keep an eye on the tasks an re-run any that fail due to dockerhub limits for the next few hours. This should fix it once and for all (for now 😅 ).

CI: Failed pulls from docker.io causing jobs to fail #31797