depends: switch to secure download of all dependencies #16049

pull Kemu wants to merge 1 commits into bitcoin:master from Kemu:Berkeley_DB_SSL_source changing 16 files +18 −17
  1. Kemu commented at 8:25 PM on May 19, 2019: contributor

    Even if we potentially check the integrity of the downloaded file via hash comparison, we should make use of SSL since it is available.

  2. fanquake added the label Build system on May 19, 2019
  3. fanquake commented at 4:43 AM on May 20, 2019: member

    Concept ACK

    If we're going to do this, might as well make the change in all packages.

    Looks like http: -> https: will work for everything except miniupnpc. That package could be changed to use the https://miniupnp.tuxfamily.org/files/ mirror.

  4. practicalswift commented at 7:33 AM on May 20, 2019: contributor

    Concept ACK

    Not leaking is better than leaking.

  5. kristapsk commented at 8:25 AM on May 20, 2019: contributor

    Concept ACK

    There is no downside in using HTTPS.

  6. jonasschnelli commented at 9:18 AM on May 20, 2019: contributor

    Tested ACK (manually downloaded curl -OL https://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz and verified the sha256 hash).

  7. jonatack commented at 9:47 AM on May 20, 2019: member

    ACK

    $ curl -OL https://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz

$ sha256sum db-4.8.30.NC.tar.gz
12edc0df75bf9abd7f82f821795bcee50f42cb2e5f76a6a281b85732798364ef  db-4.8.30.NC.tar.gz

    Same hash as the non-SSL http curl from a couple months ago.

  8. jonasschnelli commented at 10:09 AM on May 20, 2019: contributor

    @Kemu Are you interested to also cover the other plain http packages in this pull?

    grep -rnw './depends' -e 'http:'
./depends/packages/xtrans.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/lib/
./depends/packages/libxcb.mk:3:$(package)_download_path=http://xcb.freedesktop.org/dist
./depends/packages/freetype.mk:3:$(package)_download_path=http://download.savannah.gnu.org/releases/$(package)
./depends/packages/xextproto.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/proto
./depends/packages/libX11.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/lib/
./depends/packages/native_cctools.mk:8:$(package)_clang_download_path=http://llvm.org/releases/$($(package)_clang_version)
./depends/packages/zlib.mk:3:$(package)_download_path=http://www.zlib.net
./depends/packages/native_cdrkit.mk:3:$(package)_download_path=http://distro.ibiblio.org/fatdog/source/600/c
./depends/packages/xproto.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/proto
./depends/packages/xcb_proto.mk:3:$(package)_download_path=http://xcb.freedesktop.org/dist
./depends/packages/miniupnpc.mk:3:$(package)_download_path=http://miniupnp.free.fr/files
./depends/packages/fontconfig.mk:3:$(package)_download_path=http://www.freedesktop.org/software/fontconfig/release/
./depends/packages/libXau.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/lib/
./depends/packages/bdb.mk:3:$(package)_download_path=http://download.oracle.com/berkeley-db
./depends/packages/libXext.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/lib/
  9. laanwj commented at 2:43 PM on May 20, 2019: member

    utACK, as issues with apt have shown, if in doubt it's better to use TLS, it's one extra layer of security.

  10. MarcoFalke commented at 9:31 PM on May 20, 2019: member

    Please squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits

  11. Kemu force-pushed on May 20, 2019
  12. dongcarl commented at 10:06 PM on May 20, 2019: member

    Could you add something to depends/packages.md so that future depends packages will be written with this in mind?

  13. depends: switch to secure download of all dependencies
    Some dependency sources were downloaded via http, even though https (SSL/TLS) options are available.
Even if we potentially check the integrity of the downloaded files via hash comparison, we should make
use of this additional security layer.

bdb.mk
fontconfig.mk
freetype.mk
libX11.mk
libXau.mk
libXext.mk
libxcb.mk
native_cctools.mk
native_cdrkit.mk
xcb_proto.mk
xextproto.mk
xproto.mk
xtrans.mk
zlib.mk

miniupnp was switched to official project mirror with SSL support
    d8bc47fde4
  14. Kemu force-pushed on May 20, 2019
  15. Kemu commented at 10:37 PM on May 20, 2019: contributor

    Could you add something to depends/packages.md so that future depends packages will be written with this in mind?

    good idea.

      $(package)_download_path:
   Location of the upstream source, without the file-name. Usually http, https
   or ftp. Secure transmission options like https should be preferred if
   available.
  16. Kemu commented at 10:54 PM on May 20, 2019: contributor

    make download seems to run without issues, except for native_cdrkit:

    Fetching cdrkit-1.1.11.tar.bz2 from https://distro.ibiblio.org/fatdog/source/600/c
...
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

    while directly running: wget https://distro.ibiblio.org/fatdog/source/600/c/cdrkit-1.1.11.tar.bz2 or curl https://distro.ibiblio.org/fatdog/source/600/c/cdrkit-1.1.11.tar.bz2 --output cdrkit-1.1.11.tar.bz2 seem to work just fine.

    Can anyone confirm this behavior in their environment?

    This seems to be a randomly occurring error with curl (only happens with ibiblio.org though): Looks to me like a server configuration issue at ibiblio.org. Probably a loadbalancer with one of the application servers behind it not containing the full cert chain:

    **When not working:**
uli@us02:~$ curl -vs https://distro.ibiblio.org
* Rebuilt URL to: https://distro.ibiblio.org/
*   Trying 152.19.134.43...
* TCP_NODELAY set
* Connected to distro.ibiblio.org (152.19.134.43) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* stopped the pause stream!
* Closing connection 0
---
uli@us02:~$ echo | openssl s_client -connect distro.ibiblio.org:443
CONNECTED(00000003)
depth=0 C = US, postalCode = 27514, ST = NC, L = Chapel HIll, street = 153A Country Club Road, O = University of North Carolina at Chapel Hill, CN = distro.ibiblio.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, postalCode = 27514, ST = NC, L = Chapel HIll, street = 153A Country Club Road, O = University of North Carolina at Chapel Hill, CN = distro.ibiblio.org
verify error:num=21:unable to verify the first certificate
verify return:1
---

**When working:**
uli@us02:~$ curl -vs https://distro.ibiblio.org
* Rebuilt URL to: https://distro.ibiblio.org/
*   Trying 152.19.134.43...
* TCP_NODELAY set
* Connected to distro.ibiblio.org (152.19.134.43) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; postalCode=27514; ST=NC; L=Chapel HIll; street=153A Country Club Road; O=University of North Carolina at Chapel Hill; CN=distro.ibiblio.org
*  start date: Feb  1 00:00:00 2017 GMT
*  expire date: Feb  1 23:59:59 2020 GMT
*  subjectAltName: host "distro.ibiblio.org" matched cert's "distro.ibiblio.org"
*  issuer: C=US; ST=MI; L=Ann Arbor; O=Internet2; OU=InCommon; CN=InCommon RSA Server CA
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: distro.ibiblio.org
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html
< Content-Length: 19655
< Date: Mon, 20 May 2019 23:14:43 GMT
< Server: lighttpd/1.4.53
---
uli@us02:~$ echo | openssl s_client -connect distro.ibiblio.org:443
CONNECTED(00000003)
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA
verify return:1
depth=0 C = US, postalCode = 27514, ST = NC, L = Chapel HIll, street = 153A Country Club Road, O = University of North Carolina at Chapel Hill, CN = distro.ibiblio.org
verify return:1
---

    The depends Makefile includes a fallback download path (https://bitcoincore.org/depends-sources), which is probably why the tests are not failing in any case.

  17. Kemu renamed this:
    depends:Enable SSL download of Berkeley DB source
    depends: switch to secure download of all dependencies
    on May 20, 2019
  18. DrahtBot commented at 1:25 AM on May 21, 2019: member

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #15844 (depends: Purge libtool archives by dongcarl)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  19. jonasschnelli commented at 9:01 AM on May 21, 2019: contributor

    Thanks. utACK d8bc47fde46ca0711fa54a0d70ff5d066c708e50

  20. MarcoFalke added the label Needs gitian build on May 21, 2019
  21. practicalswift commented at 2:52 PM on May 21, 2019: contributor

    Welcome as a contributor @Kemu! :-)

    utACK d8bc47fde46ca0711fa54a0d70ff5d066c708e50

  22. dongcarl commented at 4:28 PM on May 21, 2019: member

    tACK d8bc47fde46ca0711fa54a0d70ff5d066c708e50

  23. DrahtBot commented at 12:43 AM on May 22, 2019: member

    <!--a722867cd34abeea1fadc8d60700f111-->

    Gitian builds for commit 2d1583ee6aff4b68dc46bcb87eb7a85f90b465c3 (master):

    Gitian builds for commit e34ec54e9451ccd3cc41a76f9c1be94d01cff21f (master and this pull):

  24. DrahtBot removed the label Needs gitian build on May 22, 2019
  25. MarcoFalke merged this on May 22, 2019
  26. MarcoFalke closed this on May 22, 2019
  27. MarcoFalke referenced this in commit 953f757677 on May 22, 2019
  28. zkbot referenced this in commit a96942e809 on Dec 11, 2019
  29. deadalnix referenced this in commit f568f26f41 on Mar 19, 2020
  30. ftrader referenced this in commit 074f238ffa on Aug 17, 2020
  31. Munkybooty referenced this in commit 584f8e14df on Oct 17, 2021
  32. Munkybooty referenced this in commit dd5faf4f72 on Oct 22, 2021
  33. Munkybooty referenced this in commit 1db0347f3c on Oct 22, 2021
  34. Munkybooty referenced this in commit 47bef9e3f8 on Oct 23, 2021
  35. Munkybooty referenced this in commit 309debe249 on Oct 26, 2021
  36. Munkybooty referenced this in commit 60ebbdd196 on Oct 28, 2021
  37. Munkybooty referenced this in commit 4d23c24623 on Oct 28, 2021
  38. Munkybooty referenced this in commit 69b122d4d9 on Nov 12, 2021
  39. Munkybooty referenced this in commit e35dc6e017 on Nov 13, 2021
  40. Munkybooty referenced this in commit 02c59f37d6 on Nov 13, 2021
  41. Munkybooty referenced this in commit ed8672a8cb on Nov 14, 2021
  42. Munkybooty referenced this in commit bd4d0dfd1d on Nov 16, 2021
  43. Munkybooty referenced this in commit 4ea11231fc on Nov 16, 2021
  44. Munkybooty referenced this in commit 036e260006 on Nov 18, 2021
  45. Munkybooty referenced this in commit e5817f44a2 on Nov 18, 2021
  46. MarcoFalke locked this on Dec 16, 2021
Contributors
KemufanquakepracticalswiftkristapskjonasschnellijonatacklaanwjMarcoFalkedongcarlDrahtBot
Labels
Build system
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-01 15:14 UTC
This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me