Upgrade gitian win32 to Precise 12.04. #3029

wtogami commented at 4:25 AM on September 25, 2013: contributor

mingw upgrade allows more hardening, compiler bug fixes and possibily win64 later.

Rename deps .zip files to be more consistent in revision format. Remove "msvc" compat triplet name. LDFLAGS -static-libgcc -static-libstdc++ Upgrade win32 gitian to boost-1.54.0. Workaround 1.54.0 build bug, upstream #9156 Workaround 1.51.0+ human bug, upstream #7262

Upstream 1.51.0 fixed a multiple reference bug that is problematic in 1.50.0. The cross-compile patch is by the Fedora package developer and shipped in Fedora.

Please also note the sha256sum -c thing in boost-win32.yml. devrandom agreed it is a good approach to better ensure the integrity of inputs, at least prior to gitian itself gaining the ability to track and enforce particular input hashes. If folks have no objections to this sha256sum -c approach, I would like to do a different pull request that does it for all inputs.

All unit tests pass.

in contrib/gitian-descriptors/boost-win32.yml:None in dacb59e8bc outdated

  24 | -  tar xjf boost_1_50_0.tar.bz2
  25 | -  cd boost_1_50_0
  26 | -  echo "using gcc : 4.4 : i586-mingw32msvc-g++
  27 | +  tar xjf boost_1_54_0.tar.bz2
  28 | +  cd boost_1_54_0
  29 | +  echo "using gcc : 4.6 : i686-w64-mingw32-g++

Diapolo commented at 5:36 AM on September 25, 2013:

We could just display version info via $compiler-binary$ --version instead of hard-coding 4.6.

wtogami commented at 6:00 AM on September 25, 2013:

"4.6" seems to be cosmetic, I'm not sure what it does if anything. The last part actually does seem to matter though.

wtogami commented at 5:00 AM on September 26, 2013:

@Diapolo What did you mean by $compiler-binary$ ? Gitian doesn't seem to have anything like that. Bash parsed it as $compiler (which is blank) and -binary$ is separate. Unless you indicate exactly what you meant, I'm keeping it as "4.6" which seems to be merely cosmetic anyway.

Diapolo commented at 6:27 AM on September 26, 2013:

I tried to tell you, just use the version output of our MinGW compiler suite, which (for example) could be g++ --version, and don't use the hard-coded 4.6 there, nothing fancy ;).

wtogami commented at 7:41 AM on September 26, 2013:

This works: GCCVERSION=$($HOST-g++ -E -dM $(mktemp --suffix=.h) | grep __VERSION__ | cut -d ' ' -f 3 | cut -d '"' -f 2) echo "using gcc : $GCCVERSION : $HOST-g++

in contrib/gitian-descriptors/boost-win32.yml:None in dacb59e8bc outdated

  47 | +  # Patch Mirror: http://mindstalk.net/host/boost-mingw-gas-cross-compile-2013-03-03.patch
  48 | +  # Patch Mirror: http://rose.makesad.us/~paulproteus/mirrors/boost-mingw-gas-cross-compile-2013-03-03.patch
  49 | +  echo "d2b7f6a1d7051faef3c9cf41a92fa3671d905ef1e1da920d07651a43299f6268  ../boost-mingw-gas-cross-compile-2013-03-03.patch" | shasum -c
  50 | +  patch -p0 < ../boost-mingw-gas-cross-compile-2013-03-03.patch
  51 | +
  52 | +  # Bug Workaround: boost-1.54.0 broke the ability to disable zlib

Diapolo commented at 5:38 AM on September 25, 2013:

Instead of another patch perhaps just build with zlib support ;)?

wtogami commented at 5:57 AM on September 25, 2013:

Edit: I later realized that we already build zlib in deps-win32.yml. Currently boost is not dependent on it, but it could be. Reading through the boost commits it sounds like compression is considered to be a potential DoS vector. I didn't check to see if we actually use that codepath at all, but it just seemed safer to keep it disabled as we didn't need it before.

At least the disabled zlib build workaround is tiny and obviously correct.

theuni commented at 5:18 PM on September 25, 2013:

Bumping to 1.54 here just because of a build-bug in 1.50 is senseless. Backport the fix (https://svn.boost.org/trac/boost/ticket/4258) or discuss the possibility of a bump. But don't just change it arbitrarily.

wtogami commented at 6:38 PM on September 25, 2013:

Thank you. I want to discuss the possibility of a bump of all the deps. Litecoin's gitian used newer versions of all these libraries (except boost) 15 months ago. If folks want to back down to the same version of boost used by 0.8.x gitian win32 then I'll apply the patch instead of this approach. How do others feel about the gitian dep versions?

wtogami commented at 11:08 PM on September 25, 2013: contributor

cfields indicated this ar is not deterministic by default so the hacks need to remain.

http://pastebin.com/raw.php?i=a89Hvm7Q The next update will include this commit to make all mingw builds static.

Discussion in #bitcoin-dev seems to indicate general agreement on upgrading all gitian win32 library deps to stable versions that several distros already ship. To make it easier to review, I will separate the commits to working precise 12.04 gitian with the previous boost-1.50 + build patch, and another commit that upgrades all the libraries.

Before adding this to the next PR update, please provide feedback to indicate that this would be accepted. It is a non-trivial amount of effort to test all this.

wtogami closed this on Sep 25, 2013

wtogami reopened this on Sep 25, 2013

build: add libcrypt32 dep for new mingw. Also ensure that libgcc/libstdc++ are static-linked regardless of the toolchain config c589f5f2bd

gavinandresen commented at 6:18 AM on September 26, 2013: contributor

We currently require 2 VM images to create a release (10.04 32-bit used to compile linux 32-bit and windows binaries, and 10.04 64-bit used to compile linux 64-bit).

Ideally, a single 12.04 64-bit VM could compile everything; if you can get that to work, and produce binaries that work for Windows XP+ and (whatever the equivalent version is for Linux) then I'm all for it, and will gladly do the (not trivial) work of creating a 12.04-based VirtualBox VM for gitian-building.

I'm less enthused if this means anybody who gitian-builds has to create THREE virtual machine images.

RE: updating library dependencies: same criteria, and "widely distributed, not bleeding-edge" is a good guideline, unless, of course, there are critical security fixes that might affect us.

wtogami commented at 7:00 AM on September 26, 2013: contributor

The added storage requirement for base-precise-amd64.qcow2 is only 739M. That is necessary to modernize the toolchain, enable more hardening and possibly performance optimizations later. The mingw back in the 10.04 era was pretty dire compared to what shipped in 12.04.

There was a productive discussion about improvements to the Linux deterministic binary process in #bitcoin-dev today. Some interesting options there were proposed. I would warn however to keep this PR solely on the needs of the win32 build.

"widely distributed, not bleeding-edge" seems to be the reasonable consensus for dep upgrades. I will look at various distros along with patched CVE's if they exist, and pick an appropriate version and patches.

Upgrade gitian win32 to Precise 12.04.

mingw upgrade allows more hardening, compiler bug fixes and possibily win64 later.
Rename deps .zip files to be more consistent in revision format.
Boost #4258 build fix for mingw32

af6c2c71ed

wtogami commented at 11:19 AM on September 26, 2013: contributor

Each commit here results in a working win32 build.

cfield's commit allows the new mingw32 to build without errors and has no effect on the old mingw.
Upgrading to precise 12.04 is literally only that plus the mingw32 compat patch that was later added to boost-1.51.
The following commit upgrades to boost-1.54 and demonstrates a very simple method to check the integrity of source inputs.

Unless there are more improvements to be suggested I believe this is ready for commit.

The next step after commit would be to research the "widely distributed, not bleeding-edge" dep versions, verify that the source tarballs used by multiple distros are identical, then verify that Bitcoin win32 remains working with all those upgrades. I suggest michagogo, with his interest in gitian and learning, to work on that next step as it is time consuming but achievable by someone who is learning software development.

laanwj commented at 1:52 PM on September 26, 2013: member

Changes ACK (though release-process.md has to be updated to download the new dependency boost-mingw-gas-cross-compile-2013-03-03.patch ) - haven't tested yet

Upgrade gitian win32 to boost-1.54.0

Workaround 1.54.0 build bug, upstream #9156
Workaround 1.51.0+ human bug, upstream #7262

This commit also demonstrates a method to verify the integrity of inputs.

f2bc1d35d6

wtogami commented at 2:14 PM on September 26, 2013: contributor

release-process.md has been updated for boost-mingw-gas-cross-compile-2013-03-03.patch

sipa commented at 2:18 PM on September 26, 2013: member

ACK on upgrading win32 gitian to mingw-w64 and upgrading boost.

We probably want to converge to a single VM for everything, with ideally the following properties:

Old glibc, so that the resulting binary works on many systems.
Modern g++, so we can start using for example C++11 features and -flto
Mingw-w64 to have a modern win32 (and perhaps one day win64) environment.

For now, there don't seem to be any readily-available options that combine all properties, so let's stay on Lucid for linux builds for now.

wtogami commented at 9:22 PM on September 26, 2013: contributor

OFF-TOPIC for this PR but responding anyway. @gavinandresen Regarding "Ideally, a single 12.04 64-bit VM could compile everything", consensus is that it is infeasible as its built binaries would be incapable of running on RHEL6, a major server platform on which it is quite difficult to build Bitcoin from source. It would also drop 10.04 which is still supported as a server platform.

The more recent conversation wandered into the desire to have modernized toolchains building to a target old glibc. If that is the case, then we are better off keeping Linux gitian as-is for 0.9 (as it works just fine right now) and looking at future available options. The ideal future single VM would cross-compiling everything from a long-term supported distro that maintains modern toolchains for us. This would allow us to maintain only minimal toolchain additions ourselves, themselves deterministic built on the underlying OS.

On whatever future platform that is chosen we can build deterministic Linux binaries for i686 and x86_64 against a particular target old glibc ABI. Whenever a mac cross-compile toolchain happens, it can be built and deployed there too.

sipa commented at 4:14 PM on September 28, 2013: member

@gavinandresen Agree with merging this now, so we don't have to rely on an ancient GCC with limited hardening anymore for Win32, and postponing the discussion about what to do for the linux build environments later?

gavinandresen referenced this in commit b2180d82a4 on Sep 29, 2013

gavinandresen merged this on Sep 29, 2013

gavinandresen closed this on Sep 29, 2013

Bushstar referenced this in commit 8231255dd3 on Apr 8, 2020

DrahtBot locked this on Sep 8, 2021