Depends: Bump Qt to 5.4.2 #6314

welshjf commented at 4:44 PM on June 20, 2015: contributor

Discussion in #6267.

Qt 5.4 introduced QtWebEngine which basically embeds the whole of Chromium, growing the monolithic download from 224M to 398M. (Linux distro packagers aren't happy either...)

By switching to submodules, I've reduced the download size to 76M, and disk space needed to build from ~2G to ~600M. (The multi-source support in the depends system leaves a lot to be desired in my opinion, but at least it works.) Another nice result is that any future additional Qt submodules will be skipped by default, rather than having to add more -skip configure options.

depends: split qt config options to separate lines

Helps ease of reading and comparison. Removed some duplicates.

4304929295

depends: update static Qt to 5.4.2

The new version includes the backported fixes qt5-tablet-osx.patch and
qt5-yosemite.patch. Slight adjustments were needed to the xcb include
order patch.

reduce-relocations needs to be explicitly disabled for PIE to work on
GCC 5 (not unique to the new Qt; see #6248).

Accessibility is now included in the base QtWidgets, not a plugin.

CPATH must be set in the Bitcoin build environment to match the way Qt
is configured.

Related discussion: #6267

609e372bf1

Depends: Only download relevant Qt submodules 372893d00a

theuni commented at 7:13 PM on June 20, 2015: member

Great work! I had been planning to switch to submodules with this bump, so thanks for doing that as well. I see a few things that need some work, I'll comment on the individual files.

in depends/packages/qt.mk:None in 372893d00a outdated

  12 |  $(package)_linux_dependencies=freetype fontconfig dbus libxcb libX11 xproto libXext
  13 |  $(package)_build_subdir=qtbase
  14 |  $(package)_qt_libs=corelib network widgets gui plugins testlib
  15 | -$(package)_patches=mac-qmake.conf fix-xcb-include-order.patch qt5-tablet-osx.patch qt5-yosemite.patch
  16 | +$(package)_patches=mac-qmake.conf fix-xcb-include-order.patch
  17 | +qtbase_file_name=qtbase-$(qt_suffix)

theuni commented at 7:23 PM on June 20, 2015:

please use $(package)foo for variables. All files are included by the main makefile, so $(package) acts somewhat as a namespace.

welshjf commented at 9:54 PM on June 20, 2015:

Ah, I figured there was some reason like that... at least it will help consistency, but I'd note that $(package) in this case is "qt" ;)

theuni commented at 7:33 PM on June 20, 2015: member

Looks like their buildsystem changes broke our OSX cross-build again. We'll need to poke at it to see what's changed.

depends: Quick fixes for qt 5.4.3 build for OSX

Addressed:
  - Make sure that the SDK path and version are properly set.
  - Work around a strange CUPS dependency by disabling printing altogether.

Presumably some cups functionality should be hidden behind ifdefs so that
-no-cups turns it off, but in 5.4.3 there's a hard dependency on libcups
regadless of build config. This needs to be investigated and possibly reported
upstream.

4586858ebd

depends: Quick fixes for qt 5.4.3 build for Windows

including comdef.h gets ugly with mingw. Guard it.

453a59adbb

depends: More quick fixes for qt 5.4.3 build for Windows

  Addressed:

  - harfbuzz is enabled by default for Qt now. Make sure that we find a static
    lib, whether it's qt's internal one or a system lib.
  - Temporary hack to drop lqtaccessiblewidgets. This needs to be conditional,
    it's still needed for qt < 5.4.

4ddff59546

in depends/packages/qt.mk:None in 372893d00a outdated

   0 | @@ -1,53 +1,114 @@
   1 |  PACKAGE=qt
   2 | -$(package)_version=5.2.1
   3 | -$(package)_download_path=http://download.qt-project.org/official_releases/qt/5.2/$($(package)_version)/single
   4 | -$(package)_file_name=$(package)-everywhere-opensource-src-$($(package)_version).tar.gz
   5 | -$(package)_sha256_hash=84e924181d4ad6db00239d87250cc89868484a14841f77fb85ab1f1dbdcd7da1
   6 | +$(package)_version=5.4.2
   7 | +$(package)_download_path=http://download.qt.io/official_releases/qt/5.4/$($(package)_version)/submodules

Diapolo commented at 10:52 PM on June 20, 2015:

Is it possible to download via https, just feels better these days...

welshjf commented at 2:38 AM on June 21, 2015:

If you don't trust verification by SHA256 baked into the download code, then you really shouldn't trust HTTPS ;)

laanwj commented at 1:42 PM on June 24, 2015:

I was about to say belt and suspenders to use https, especially as the server supports it. Being slightly paranoid hasn't hurt us in the past. But looks like using https makes no sense, it redirects you to a http mirror:

--2015-06-24 15:40:51--  https://download.qt.io/official_releases/qt/5.4/5.4.2/submodules/qtx11extras-opensource-src-5.4.2.tar.xz
Resolving download.qt.io (download.qt.io)... 77.86.229.90
Connecting to download.qt.io (download.qt.io)|77.86.229.90|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://ftp2.nluug.nl/languages/qt/archive/qt/5.4/5.4.2/submodules/qtx11extras-opensource-src-5.4.2.tar.xz [following]
--2015-06-24 15:40:52--  http://ftp2.nluug.nl/languages/qt/archive/qt/5.4/5.4.2/submodules/qtx11extras-opensource-src-5.4.2.tar.xz
Resolving ftp2.nluug.nl (ftp2.nluug.nl)... 192.87.102.43, 2001:610:1:80aa:192:87:102:43
Connecting to ftp2.nluug.nl (ftp2.nluug.nl)|192.87.102.43|:80... connected.
HTTP request sent, awaiting response... 200 OK

welshjf commented at 4:01 PM on June 24, 2015:

FWIW, the hashes are checked twice, once on download and again on extract.

theuni commented at 11:43 PM on June 20, 2015: member

@welshjf See the top 3 commits at https://github.com/theuni/bitcoin/commits/qt54 .

They're not completely correct (some heavy-handed work-arounds), but it should be enough to have all platforms building again successfully.

laanwj commented at 11:35 AM on June 22, 2015: member

By switching to submodules, I've reduced the download size to 76M, and disk space needed to build from ~2G to ~600M

Nice!

laanwj added the label Build system on Jun 22, 2015

Depends: qt.mk style fixes

Addressed:
  - Prefix all variables with $(package)_
  - Use += instead of \ for line continuation

e3f08386e9

in build-aux/m4/bitcoin_qt.m4:None in e3f08386e9 outdated

 111 | @@ -112,13 +112,11 @@ AC_DEFUN([BITCOIN_QT_CONFIGURE],[
 112 |      if test x$bitcoin_cv_static_qt = xyes; then
 113 |        AC_DEFINE(QT_STATICPLUGIN, 1, [Define this symbol if qt plugins are static])
 114 |        if test x$qt_plugin_path != x; then
 115 | -        QT_LIBS="$QT_LIBS -L$qt_plugin_path/accessible"

laanwj commented at 1:27 PM on June 24, 2015:

Why remove the accessability stuff?

welshjf commented at 3:59 PM on June 24, 2015:

The plugin no longer exists; it was moved to the base library. theuni wants all this conditional though, to support static builds with either version.

laanwj commented at 6:39 AM on June 25, 2015:

OK, fair enough

jonasschnelli commented at 1:33 PM on June 24, 2015: contributor

I almost forgot to post my gitian build: https://builds.jonasschnelli.ch/pulls/6314/ Tested the OSX version and the app can't run. Exits with:

$ /Volumes/Bitcoin-Core/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt 
This application failed to start because it could not find or load the Qt platform plugin "cocoa".

Reinstalling the application may fix this problem.
Abort trap: 6

welshjf commented at 4:24 PM on June 24, 2015: contributor

@jonasschnelli, sounds just like my problem on Linux, #6267 (comment).

We build Qt with <prefix>/include exported in CPATH, which causes it to assume that's the standard system include path, and no longer include it in its pkg-config. Thus when building bitcoin-qt, the system qconfig.h gets included, which among other potential problems makes it think it's dynamic and omit the plugins.

If you have cycles to burn you can try replicating my CPATH hack from gitian-linux.yml. I'll try getting it into config.site per @theuni.

welshjf commented at 4:39 PM on June 24, 2015: contributor

Er, I didn't summarize that quite right... see linked comments for more complicated but correct explanation ;)

theuni commented at 5:06 PM on June 24, 2015: member

The CPATH trick is a hack, and as @welshjf mentioned, needs to be re-worked after 5.3.

I'd like to take a fresh look with this bump, and see if we can work around the issues more cleanly. The patches I posted earlier were just enough to get it building, I'll spend some time this weekend looking at the remaining issues.

welshjf commented at 5:17 PM on June 24, 2015: contributor

Moved CPATH setting to config.site as requested. Tested on Linux; should fix the platform plugin problem on Mac too.

As I mentioned, my first attempt was to remove the CPATH setting from qt.mk which seemed to be the root of the problem, but that broke something about the openssl linkage. I think that would be a good place to start investigating.

theuni commented at 5:56 PM on June 24, 2015: member

CPATH is necessary for the qt build because we need to show qt how to find cross includes. Unfortunately, if you pass it -I/foo/bar in configure, it'll apply those flags to native builds (qmake and friends) as well.

Need to start by seeing if they've fixed up configure to be smarter with cross, allowing separate flags to be passed.

Barring that, it'd be better to just fix up the .pc files. Since qt assumes that the prefix path is included, I'm not sure why they wouldn't list it in Cflags.

theuni commented at 7:21 PM on June 24, 2015: member

@welshjf ok, the problem here was that we were dropping the current flags for the static test. qt's assumption that the pkg-config prefix is already included is a reasonable one.

See https://github.com/theuni/bitcoin/commit/294e35e8869ad07579e99f669477cbef732c22c8 for a fix that should let you drop the CPATH stuff outside of qt. It also fixes up configure for OSX on qt < 5.4. Still need to fix windows though.

welshjf commented at 9:52 PM on June 24, 2015: contributor

@theuni great, thanks, that's a much nicer fix. Simple if you know where to look!

For qt < 5.4 on OSX, wouldn't we still need the tablet crash and yosemite build patches? Although, those were applied from qt.mk, and if you're using that then you must be on 5.4. Still confused about when and how non-depends-based static builds are done :/

theuni commented at 10:22 PM on June 24, 2015: member

@welshjf Bitcoin's buildsystem is completely independent of depends. No matter where the dependencies came from or how they were configured, if they're sane, we attempt to work with them.

The depends system is our way of self-building dependencies, since we need to build everything from source for trusted releases. Once they're built, we setup a config.site so that our configure can use them without having to plug in a bunch of --with-foo=/path/to/foo and etc.

If you're building with libs that were built outside of depends, we can only assume that they're already patched as necessary. Distros frequently patch source releases. Our configure/make doesn't apply any patches, only the depends that we build ourselves, because we're in complete control of those.

Maybe your confusion is coming from the word "static"? Static in this sense means static libraries (.a's) as opposed to shared libraries. It doesn't mean "building a static config".

welshjf commented at 10:44 PM on June 24, 2015: contributor

Yes, when I say "static build" I mean a build using static libraries. On Windows/Mac there are no "distro" patches since Qt comes directly from upstream. Linux distros, to my knowledge, always package Qt as .so shared libraries. So as I see it, either you're on Win/Mac and cannot rely on outdated versions being sane, or you're on Linux doing a dynamic build, or you're building an official release using depends. I acknowledge that there may be more things in heaven and earth than are dreamt of in my philosophy, though. :)

depends: more qt5 build fixes

- Append CPPFLAGS when checking for static plugins. This matches our behavior
  elewhere.
- Only link lqtaccessiblewidgets for OSX for Qt < 5.4.
- Import AccessibleFactory as necessary for Qt < 5.4.

fc9f244c80

gitian-linux.yml: remove CPATH hack added for Qt 5.3+ 0ca52d9355

welshjf force-pushed on Jun 24, 2015

jonasschnelli commented at 11:43 AM on June 25, 2015: contributor

Just built over gitian and tested on max OSX10.10. Works as expected. Nice!

bildschirmfoto 2015-06-25 um 13 43 30

laanwj commented at 3:47 PM on June 25, 2015: member

@welshjf these kinds of builds it needs to be able to handle:

	depends	non-depends
static	×	×
dynamic	-	×

'non-depends' uses any dependencies that the user themselves provides (or their distro, in case of Linux), this can be a wider range of library versions. E.g. we even still support building with Qt4, and with ancient boost.

laanwj commented at 2:46 PM on June 29, 2015: member

@theuni Is this ready to merge?

theuni commented at 3:23 PM on June 29, 2015: member

@laanwj No, not yet. Windows still needs fixing up, and the Linux build needs testing. Need to look over the X include order stuff, since that's subtle. I'll document the issues while I go over them this time.

laanwj commented at 12:51 PM on July 5, 2015: member

I also just got a notification that my suggestion for a flag to force TLS has been implemented and will be in Qt 5.5.0: https://bugreports.qt.io/browse/QTBUG-43168

jonasschnelli commented at 9:25 AM on July 13, 2015: contributor

I assume this might fix #6426. Just built this PR over gitian but the linux build failed. Various errors (grep error):

cc1plus: error: unrecognized command line option â€˜-fuse-ld=goldâ€™
cc1plus: error: unrecognized command line option â€˜-mavx2â€™
Project ERROR: mtdev development package not found
pulseaudio.cpp:34:30: fatal error: pulse/pulseaudio.h: No such file or directory
xrender.cpp:35:36: fatal error: X11/extensions/Xrender.h: No such file or directory
Project ERROR: libdrm development package not found
openvg.cpp:40:23: fatal error: VG/openvg.h: No such file or directory
...
qxcbxsettings.cpp:40:36: fatal error: X11/extensions/XIproto.h: No such file or directory

Full log: https://builds.jonasschnelli.ch/pulls/6314/build-linux.log

Any ideas and anyone willing to have a look at this?

welshjf commented at 9:13 PM on July 13, 2015: contributor

Most of those are just auto-detections failing; XIproto.h is the relevant error.

I guess that answers #6267 (comment).

We used to just strip out that #include, but now qxcbsettings.cpp actually uses some macros from the X headers. Probably we can just define them locally.

welshjf commented at 9:21 PM on July 13, 2015: contributor

@laanwj good news, maybe we should leapfrog 5.4. http://doc.qt.io/qt-5/qssl.html#SslProtocol-enum seems to be where to look, unfortunately doesn't show what's new in 5.5. Looks like it might be secure by default now, but "denying SSLv3 connections that does not upgrade to TLS" doesn't make me feel entirely confident.

depends: qt fixups

- Explicitly disable more functionality we don't use
- CPATH hacks are not necessary with newer qt
- Add back a fixed up version of the XIproto.h include

f7b1db98c0

theuni commented at 2:28 AM on July 17, 2015: member

@welshjf Have a look at https://github.com/theuni/bitcoin/commit/f7b1db98c03ec9d1e1f70e16683c37126e6e49c5 . With that, all platforms work for me as intended. Gitian builds included.

I've confirmed that the Linux include order is still ok. I'm not entirely happy about the osx print workaround, but everything seems to work.

As for jumping to 5.5 that's fine too when it's out+stable, but I don't think we need to hold up this bump for it.

Edit: I didn't realize that 5.5 was already released. I'm grabbing now to see if it's a smooth bump on top of the work here.

welshjf commented at 4:35 AM on July 17, 2015: contributor

Repeating here for clarify: theuni's latest commit is working for me (depends build on Fedora). Travis is failing due to SF outage: "The sourceforge.net website is temporarily in static offline mode. Only a very limited set of project pages are available until the main website returns to service."

welshjf commented at 4:37 AM on July 17, 2015: contributor

Incidentally... are we effectively re-downloading all deps from upstream each time someone submits or updates a PR? That doesn't seem very nice.

theuni commented at 4:45 AM on July 17, 2015: member

No, only when the cache is wiped or the depends are stale and need to be rebuilt. In this case, boost has been bumped in master, so when this PR is merged for testing boost must be rebuilt as well.

Usually the sourceforge problem would not be an issue as we have a fallback download, but annoyingly, they don't respond with an error :\

welshjf commented at 5:02 AM on July 17, 2015: contributor

Would it help if I rebased this on master, then?

theuni commented at 5:03 AM on July 17, 2015: member

I've pushed 2 more commits here https://github.com/theuni/bitcoin/tree/qt54 which bump to 5.5. Quickly sanity checked on all platforms.

theuni commented at 5:04 AM on July 17, 2015: member

Yea, that might work. Though if SF stays down for a while, we'll need to redirect the download anyway, to avoid having a bunch of other PRs fail until it comes back up.

welshjf commented at 2:56 PM on July 20, 2015: contributor

SF is partially back, downloads working at least. @theuni why don't you rename your branch and do a new PR for 5.5. Don't think there's anything left for me to fix here.

laanwj commented at 4:33 PM on July 20, 2015: member

Thanks for your work @welshjf . Yes, let's move directly to 5.5. WIll you file the PR @theuni?

theuni commented at 10:34 PM on July 22, 2015: member

Yes, will do. Maybe the OSX printing problem has been fixed for 5.5 and we can drop that hack.

laanwj closed this on Jul 24, 2015

MarcoFalke locked this on Sep 8, 2021