depends: Some fallback downloads are missing #22556

$ wget https://bitcoincore.org/depends-sources/miniupnpc-2.2.2.tar.gz
--2021-07-26 15:44:33--  https://bitcoincore.org/depends-sources/miniupnpc-2.2.2.tar.gz
Resolving bitcoincore.org (bitcoincore.org)... 198.251.83.116, 107.191.99.5
Connecting to bitcoincore.org (bitcoincore.org)|198.251.83.116|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-07-26 15:44:34 ERROR 404: Not Found.

$ wget https://bitcoincore.org/depends-sources/capnproto-c++-0.7.0.tar.gz
--2021-07-26 15:41:57--  https://bitcoincore.org/depends-sources/capnproto-c++-0.7.0.tar.gz
Resolving bitcoincore.org (bitcoincore.org)... 107.191.99.5, 198.251.83.116
Connecting to bitcoincore.org (bitcoincore.org)|107.191.99.5|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-07-26 15:41:58 ERROR 404: Not Found.

Did not test all of dependencies though.

UPDATE: This script allows to verify all downloads.

cc @laanwj

from my testing the following are also missing:

native_mac_alias-v2.2.0 qtbase-everywhere-src-5.12.11 qttranslations-everywhere-src-5.12.11.tar.xz qttools-everywhere-src-5.12.11.tar.xz

See also https://github.com/bitcoin-core/bitcoincore.org/issues/782

Grr, thanks, I'll take a look at why this isn't syncing (again).

Ok, there are a few things going on.

• make download wasn't running with MULTIPROCESS=1, so it wasn't pulling in the multiprocess deps. I've now added it.

• Some wgets are failing because of a cert problem (expired letsencrypt) on the server.

• verify-commits.py is failing on the server and locally for me as well. The sync script runs this first to avoid being tricked into grabbing fake deps. @MarcoFalke it's been a while since I've messed with any of the verify stuff, maybe there's something I'm missing?

For now I've manually uploaded all of the missing files mentioned above.

Edit: once this is fixed, I'll work on moving the sync script into git so that it's not untouchable voodoo :)

Keys regularly get their expiry extended, so you'll need to fetch the latest from a keyserver before running verify-commits. See also commit c702d1fefddcbd8c6465d3dc330fde235d03337e

@MarcoFalke Thanks for the tip, that was indeed the issue :)

I had successfully received keys from a bunch of other keyservers, but pulling from keys.openpgp.org fails on the server. Pulling from there works for me locally (and now so does verify-commits), so I'm guessing the problem with the server receiving keys is related to the other current ssl misconfiguration problem. I'll keep poking at this.

Edit: once this is fixed, I'll work on moving the sync script into git so that it's not untouchable voodoo :)

:+1:

Adding MULTIPROCESS=1 to make download caused this to start failing again because of a bug in depends. I've PR'd the fix: https://github.com/bitcoin/bitcoin/pull/24240

Is this still an issue, and if so, which downloads are still missing?

Did not test all of dependencies though.

This script allows to verify all downloads.

For now, the only one fails -- clang+llvm-10.0.1-aarch64-linux-gnu.tar.xz:

$ wget https://bitcoincore.org/depends-sources/clang+llvm-10.0.1-aarch64-linux-gnu.tar.xz
--2022-07-29 17:04:44--  https://bitcoincore.org/depends-sources/clang+llvm-10.0.1-aarch64-linux-gnu.tar.xz
Resolving bitcoincore.org (bitcoincore.org)... 107.191.99.5, 198.251.83.116
Connecting to bitcoincore.org (bitcoincore.org)|107.191.99.5|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-07-29 17:04:45 ERROR 404: Not Found.

This script allows to verify all downloads.

It might be easier to modify make download to only check the fallback URL

Going to close this as done for now. I think future issues can be opened against https://github.com/bitcoin-core/bitcoincore.org. This is not a bug in Bitcoin Core.