Autotools buildsystem

theuni commented at 3:26 am on August 27, 2013: member

Continuation of #2805.

Changes since Round 2:

Rebased on current master Big cleanups Added protoc/libprotobuf support Added win32 installer make target Added Gitian descriptors

I could use some help with the following items:

Documenting windows/osx/mingw build. They should all be working, but I don’t currently have a windows/osx environment for testing. Documentation is a rather large job for this merge, as it requires a pretty extensive overhaul. For the most part, the build docs can be thrown away and replaced by a single procedure, with the exception of platform-specific dependencies setup. There is no longer any need for a distinction between gui/non-gui build procedures.
Edit: Gitian issues fixed.

I’ve briefly updated the documentation for unix. Sadly, my macbook busted last weekend (should be fixed in ~1wk), So I was not able to add the osx dmg build target as I had intended, nor the documentation for building on osx.

Is anyone willing to help me out with the above?

Here is the final go at an Autotools build system replacement. It is meant to be a drop-in replacement for the current system(s), providing the same features with no net changes. It can also live side-by-side with the old system while sharing the same build-related variables in order to facilitate a smooth transition.

I hope the benefits are obvious enough: A single/shared build procedure, portability, ease of packaging, ease for downstreams, ease for repository maintainers, cross-compilation, etc. I don’t vouch for Autotools in any way, in fact, this configure.am is downright ugly (mainly just because of mingw though), but it’s portable and well-established.

This does away with the need for qmake, as the Makefile is capable of generating everything it needs in a portable way.

Building from CLI: For Linux, assuming the dependencies have been met, the build procedure looks like this:

0./autogen.sh
1./configure
2make

Same for OS X, but the pkg-config path needs to be hooked up from macports first:

0echo "/opt/local/share/aclocal" | sudo tee -a /usr/share/aclocal/dirlist

For mingw it’s the same, but you will need to provide lots of paths in the form of:

0./autogen.sh
1export STAGING=/path/to/extracted/libs
2./configure CPPFLAGS=-I$STAGING LDFLAGS=-L$STAGING --prefix=$STAGING --host=i586-mingw32msvc --with-qt-bindir=$STAGING/host/bin --with-qt-plugindir=$STAGING/plugins --with-qt-incdir=$STAGING --with-boost=$STAGING --with-protoc-bindir=$STAGING/host/bin
3make

In addition, there are helpers for qt and boost to help with finding some locations. Use ./configure –help to see the available options.

Native windows built is untested, as I don’t have a windows environment at my disposal.

‘make check’ will run the unit tests and print the results.

I’ve done my best to avoid adding any new behavior or features, and I would much prefer to aim for feature-parity before making any improvements.

jgarzik commented at 3:32 am on August 27, 2013: contributor

ACK

theuni commented at 7:54 am on August 28, 2013: member

I believe I’ve whipped the gitian weirdness into shape enough for initial merge.

Only real blockers as I see now are acks from various platform devs (osx, win32 cross, win32 native), and docs to accompany. I’m afraid I’ve reached my limit now, as far as those things are concerned.

Awaiting some input, though there’s no hurry from my side.

gavinandresen commented at 1:46 am on August 30, 2013: contributor

Doesn’t work for me on OSX 10.7.5 with MacPorts autotools:

 0tully:bitcoin gavin$ autoreconf --version
 1autoreconf (GNU Autoconf) 2.69
 2
 3tully:bitcoin gavin$ autoreconf -vif
 4autoreconf: Entering directory `.'
 5autoreconf: configure.ac: not using Gettext
 6autoreconf: running: aclocal --force -I src/m4
 7autoreconf: configure.ac: tracing
 8autoreconf: configure.ac: not using Libtool
 9autoreconf: running: /opt/local/bin/autoconf --force
10autoreconf: running: /opt/local/bin/autoheader --force
11autoreconf: running: automake --add-missing --copy --force-missing
12configure.ac:624: warning: 'INCLUDES' is the old name for 'AM_CPPFLAGS' (or '*_CPPFLAGS')
13configure.ac:624: warning: 'INCLUDES' is the old name for 'AM_CPPFLAGS' (or '*_CPPFLAGS')
14configure.ac:624: warning: 'INCLUDES' is the old name for 'AM_CPPFLAGS' (or '*_CPPFLAGS')
15/opt/local/share/automake-1.14/am/depend2.am: error: am__fastdepOBJCXX does not appear in AM_CONDITIONAL
16/opt/local/share/automake-1.14/am/depend2.am:   The usual way to define 'am__fastdepOBJCXX' is to add 'AC_PROG_OBJCXX'
17/opt/local/share/automake-1.14/am/depend2.am:   to 'configure.ac' and run 'aclocal' and 'autoconf' again
18src/qt/Makefile.am: error: Objective C++ source seen but 'OBJCXX' is undefined
19src/qt/Makefile.am:   The usual way to define 'OBJCXX' is to add 'AC_PROG_OBJCXX'
20src/qt/Makefile.am:   to 'configure.ac' and run 'autoconf' again.
21configure.ac:624: warning: 'INCLUDES' is the old name for 'AM_CPPFLAGS' (or '*_CPPFLAGS')
22configure.ac:624: warning: 'INCLUDES' is the old name for 'AM_CPPFLAGS' (or '*_CPPFLAGS')
23autoreconf: automake failed with exit status: 1

If I ignore the ‘failed with exit status 1’ and run ./configure, I get ‘configure: error: “Big Endian not supported”’

theuni commented at 3:46 am on August 30, 2013: member

I grabbed automake 1.14 and tested locally. I just veified/pushed the fix for that problem.

New parts for my Macbook should be here tomorrow, Monday at the latest. I can test definitively then if you don’t feel like being a guinea pig.

theuni commented at 4:36 am on August 30, 2013: member

Looks like the problem is trying to build for osx 10.5 from 10.7, which isn’t going to happen without a good amount of local fiddling (likely installing xcode 3).

You might be able to get past it for now by disabling the release compatibility flags: ./configure –enable-debug

fanquake commented at 12:20 pm on August 31, 2013: member

@theuni Do you use MacPorts, HomeBrew or something else on OSX?

theuni commented at 7:42 am on September 1, 2013: member

Just pushed a boatload of fixes. Sorry for yet another round of changes, but this has turned out to be much more complicated than originally anticipated. I can squash down before merge, but I decided to push individual changes in case anyone is following the progress.

Fully verified on osx now that my macbook is up and running again (yay!).

I also scrapped the ‘windows-deploy’ target and replaced it with the generic ‘deploy’ one. ‘make deploy’ on osx will produce a fully-functional dmg. When building for windows, ‘make deploy’ will create a fully-functional win32 installer.

At this point, it’s feeling very very solid. That said, I only have osx 10.6 at my disposal currently, so it hasn’t been tested on 10.7/10.8 yet. @fanquake: Tested extensively with macports and homebrew after today’s changes. The build procedure is pretty much: install macports/homebrew depends as before, but with automake/autoconf added. Additionally, you can install qt directly from macports/homebrew and it should just work (tm). After that, standard configure+make should be all it takes.

I’ll try to get the osx docs written+pushed tomorrow.

sipa commented at 9:48 am on September 1, 2013: member

ACK code changes, and seems to work fine on Ubuntu (13.04). @laanwj still wanted some improvement to the .pro file, though?

gavinandresen commented at 5:26 am on September 2, 2013: contributor

So… … how do I tell autogen.sh and/or configure to use MacPorts? Just running autogen.sh then configure I get:

0checking for SSL... no
1configure: error: openssl  not found.

… with MacPorts openssl in /opt/local/{include,lib}

jgarzik commented at 5:29 am on September 2, 2013: contributor

@gavinandresen Usually I do something like

CXXFLAGS="-O2 -Wall -g -I/opt/local/include" LDFLAGS="-L/opt/local/lib" ./configure

theuni commented at 5:30 am on September 2, 2013: member

Still working on the osx docs.. I spent all day working out possible configs from 10.6-10.8 (managed to get my hands on a 10.8 macbook), both macports and brew.

Short answer: sounds like you need to ‘sudo port install pkgconfig’. After that, it should find everything for you.

wtogami commented at 9:06 pm on September 2, 2013: contributor

10.5 is officially dropped? (Apple has.)

theuni commented at 9:13 pm on September 2, 2013: member

No, I don’t have a 10.5 machine at my disposal. However, release binaries are still built as 32-bit 10.5 compatible after my changes.

Dropping 10.5 would make lots of things much easier, but I don’t wish to mix a policy change (which OSs are supported) with a technical change (buildsystem switch) for fear that this would needlessly delay the process.

gavinandresen commented at 10:50 pm on September 2, 2013: contributor

port install pkgconfig worked nicely. I’d volunteer to help write documentation, but, as you can tell, I don’t know enough about autotools to be helpful.

theuni commented at 11:09 pm on September 2, 2013: member

No worries. The reason I’m hesitant to do the docs is because it essentially needs a rm -f *. The point of autotools is abstraction, so there’s really no need for platform-specific instructions. There’s also really no need for distinction between qt/non-qt builds anymore.

My intention is essentially:

Platform-specific dependencies. A readme for each platform detailing what needs to be done before ./autogen.sh. Basically documenting the part you just went through.
Building bitcoin. How to bootstrap and configure. See ./configure –help for options. Pitfalls. What do I do when X?
Distribution notes. How to package, release, etc.
Testing.

The problem with the above is that I need a good bit of feedback. I’ve been looking at this (and working on other autotools projects) for so long that it’s become hard to tell what’s non-obvious. I don’t think I’ll receive much feedback until it’s merged and there are more devs exposing its warts.

So my proposal is to do a quick readme with monkey-style copy/paste commands for the various platforms, and let that suffice until merge. Post-merge, the above outline could be much more easily worked out.

Thoughts?

jgarzik commented at 11:20 pm on September 2, 2013: contributor

I would propose low merge hurdles, namely

builds qt and daemon on at least one Linux OS
builds qt and daemon on at least one OSX version
builds qt and daemon on Windows, or has a clear path to doing so
pulltester

Anything else can and should be resolved in-tree. Once the system is working “en grosse” merge it, leaving sector experts and motivated parties to hammer out the final details. In particular, Qt will likely need additional attention after merge, it sounds like. That’s OK. We are not going to have 0.9-rc1 tomorrow or next week.

The first three seem to be well on their way to being completed. The last is the toughie, IMO: need to get our pulltester back in working order almost immediately after merge, so we need to beg BlueMatt for some time or hopefully grow some more pulltester experts (I am entirely hands-off Java, not being a fan of the language).

tl;dr: Let’s not endlessly iterate this outside of ‘master’. Merge it.

theuni commented at 11:25 pm on September 2, 2013: member

Whew, that’s great to hear. Getting it merged will will really help.

I’ve been working on pulltester actually, and I"m nearly there. I’ve add lcov support to the buildsystem, as well as support for the java test that pulltester runs (side-rant: I find it troubling when these things are out-of-tree and firmly believe that they should be testable by every dev). So Pulltester essentially becomes a few configure/make and that’s it.

I believe I can have that pushed up tonight.

theuni commented at 8:02 am on September 3, 2013: member

Ok, that should do it for the pull tester. I’ve integrated lcov and the java test into the build. Also added a set of scripts for contrib that pull-tester can run from jenkins. The big benefit is that everything is now in-tree, so if they change in the future, there’s no worry of getting out of sync with the build procedure. Additionally, devs can run the script and yield the same result as the bot. @TheBlueMatt ping. With any luck, the bot just needs to run test-scripts/pull-tester.sh . That in-turn kicks off https://github.com/theuni/bitcoin/blob/dd08a1146f5408af5316651fbd89027575bedf14/contrib/test-scripts/build-tests.sh.in, which is as close to the old behavior as I could get it.

gavinandresen commented at 8:11 am on September 3, 2013: contributor

Nice! If Matt doesn’t get to it before me, I’ll teach the pull-tester machine to Do The Right Thing (if test-scripts/pull-tester.sh exists, run it. Else do what we do now).

theuni commented at 8:20 am on September 3, 2013: member

Sounds good.

It will need a little more love, as it will fail until the win32 deps are updated to the new version. The old ones were scattered around, the new ones are in a prefixed layout. IMO, that should wait until after this is merged, and just let it fail until then.

The old/new win32 deps can coexist to facilitate build-testing of the old kind. There will just need to be 2 copies, one in /mnt/mingw as they currently are, and another set somewhere else.

If running the new script, it should be executed as: contrib/test-scripts/pull-tester.sh /path/to/new/mingw/deps 6 (where 6 is the current make -jX value)

theuni commented at 1:54 am on September 4, 2013: member

@gavinandresen If/when you go to update the bot, the params have changed to accommodate the out-of-tree java tool.

It should now be invoked as:

0contrib/test-scripts/build-tests.sh /path/to/win32/deps /mnt/test-scripts/BitcoinjBitcoindComparisonTool.jar 6

theuni commented at 3:31 am on September 4, 2013: member

Ok, I’ve pushed the last of my changes. I now consider this to be complete. I’m sure it’s full of bugs and facepalms, but I think it now meets @jgarzik’s criteria above.

pull-tester should be working pending teaching the bot its new tricks, and a rebase of this branch. The win32 build will fail until it gets updated deps, so I’ve disabled that build in the test-scripts for now. Any objections to squashing this down now and rebasing to master?

theuni commented at 7:02 am on September 4, 2013: member

Squashed down and ready from my side, with the understanding that it will probably require some patience for a few days/weeks.

The individual commits (plus a few fixes related to rebasing to master) can be seen here if needed: https://github.com/theuni/bitcoin/commits/autotools-final-pre-rebase

I need to step away from this for the rest of the week, but I’m happy to answer any questions that arise.

laanwj commented at 11:16 am on September 5, 2013: member

Nice work, I tested around a bit and it worked wonderfully. One remark: Building from a different directory makes it croak on leveldb, ie.

 0bitcoin$ ./autogen.sh
 1bitcoin$ cd ../bitcoin-build
 2bitcoin-build$ ../bitcoin/configure
 3bitcoin-build$ make -j6
 4....
 5Building LevelDB ...
 6/bin/bash: line 0: cd: leveldb: No such file or directory
 7make[3]: *** [leveldb/libmemenv.a] Error 1
 8make[3]: *** Waiting for unfinished jobs....
 9Building LevelDB ...
10/bin/bash: line 0: cd: leveldb: No such file or directory

I first thought it was caused by parallel build, but it happens with single-threaded build as well.

theuni commented at 11:37 am on September 5, 2013: member

Yea, out-of-tree builds can’t really work without some serious hackery, which I’ve declined to add so far.

You can see how it would work with ‘make distcheck’, which runs an out-of-tree build. The “solution” applied there is to copy leveldb to the build dir.

autotools: switch to autotools buildsystem 35b8af9226

theuni commented at 1:50 am on September 6, 2013: member

squashed with the pull-tester updates and 2 additional fixes:

use c++ for all tests (means that osx framework checks get cxxflags)
bail if pkg-config is needed but not found

BitcoinPullTester commented at 2:56 am on September 6, 2013: none

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/35b8af92265ed74de63c3818e5290c27b3f35df2 for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

jgarzik referenced this in commit 15fd245b9a on Sep 7, 2013

jgarzik merged this on Sep 7, 2013

jgarzik closed this on Sep 7, 2013

shripadk commented at 3:44 am on September 7, 2013: none

Won’t work for me on OSX 10.7.2, get an error in configure.

  0➜  bitcoin git:(master) ./configure 
  1checking build system type... i386-apple-darwin11.2.0
  2checking host system type... i386-apple-darwin11.2.0
  3checking for a BSD-compatible install... /usr/bin/install -c
  4checking whether build environment is sane... yes
  5checking for a thread-safe mkdir -p... src/build-aux/install-sh -c -d
  6checking for gawk... no
  7checking for mawk... no
  8checking for nawk... no
  9checking for awk... awk
 10checking whether make sets $(MAKE)... yes
 11checking whether to enable maintainer-specific portions of Makefiles... no
 12checking if compiling with debugging symbols... checking for g++... g++
 13checking for C++ compiler default output file name... a.out
 14checking whether the C++ compiler works... yes
 15checking whether we are cross compiling... no
 16checking for suffix of executables... 
 17checking for suffix of object files... o
 18checking whether we are using the GNU C++ compiler... yes
 19checking whether g++ accepts -g... yes
 20checking for style of include used by make... GNU
 21checking dependency style of g++... gcc3
 22checking for gcc... gcc
 23checking whether we are using the GNU C compiler... yes
 24checking whether gcc accepts -g... yes
 25checking for gcc option to accept ISO C89... none needed
 26checking dependency style of gcc... gcc3
 27checking how to run the C preprocessor... gcc -E
 28checking for a BSD-compatible install... /usr/bin/install -c
 29checking for gcc... gcc
 30checking whether we are using the GNU Objective C compiler... no
 31checking whether gcc accepts -g... no
 32checking dependency style of gcc... gcc3
 33checking for a thread-safe mkdir -p... src/build-aux/install-sh -c -d
 34checking for a sed that does not truncate output... /usr/bin/sed
 35checking for ar... /usr/bin/ar
 36checking for ranlib... /usr/bin/ranlib
 37checking for windres... no
 38checking for strip... /usr/bin/strip
 39checking for gcov... /Users/shripadk/gcc-4.8/bin/gcov
 40checking for lcov... no
 41checking for java... /usr/bin/java
 42checking for genhtml... no
 43checking for git... /usr/bin/git
 44checking for moc... /usr/local/bin/moc
 45checking for uic... /usr/local/bin/uic
 46checking for rcc... /usr/local/bin/rcc
 47checking for lrelease... /usr/local/bin/lrelease
 48checking for protoc... no
 49checking for ccache... no
 50checking for pkg-config... /usr/local/bin/pkg-config
 51checking pkg-config is at least version 0.9.0... yes
 52checking for port... port
 53checking for brew... brew
 54checking whether the linker accepts -framework Foundation -framework ApplicationServices -framework AppKit... yes
 55checking whether C++ compiler accepts -mmacosx-version-min=10.5... yes
 56checking whether C++ compiler accepts -arch i386... yes
 57checking how to run the C++ preprocessor... g++ -E
 58checking for grep that handles long lines and -e... /usr/bin/grep
 59checking for egrep... /usr/bin/grep -E
 60checking for ANSI C header files... yes
 61checking for sys/types.h... yes
 62checking for sys/stat.h... yes
 63checking for stdlib.h... yes
 64checking for string.h... yes
 65checking for memory.h... yes
 66checking for strings.h... yes
 67checking for inttypes.h... yes
 68checking for stdint.h... yes
 69checking for unistd.h... yes
 70checking whether byte ordering is bigendian... no
 71checking whether pthreads work with -pthread... yes
 72checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
 73checking if more special flags are required for pthreads... -D_THREAD_SAFE
 74checking for PTHREAD_PRIO_INHERIT... yes
 75checking for special C compiler options needed for large files... no
 76checking for _FILE_OFFSET_BITS value needed for large files... no
 77checking whether the linker accepts -Wl,--large-address-aware... no
 78checking whether C++ compiler accepts -Wstack-protector... yes
 79checking whether C++ compiler accepts -fPIE... yes
 80checking whether C++ preprocessor accepts -D_FORTIFY_SOURCE=2... yes
 81checking whether the linker accepts -Wl,--dynamicbase... no
 82checking whether the linker accepts -Wl,--nxcompat... no
 83checking whether the linker accepts -Wl,-z,relro... no
 84checking whether the linker accepts -Wl,-z,now... no
 85checking whether C++ compiler accepts -fno-stack-protector... yes
 86checking whether C++ compiler accepts -fstack-protector-all... yes
 87checking whether the linker accepts -pie... yes
 88checking whether the linker accepts -Wl,-dead_strip... yes
 89checking stdio.h usability... yes
 90checking stdio.h presence... yes
 91checking for stdio.h... yes
 92checking for stdlib.h... (cached) yes
 93checking for unistd.h... (cached) yes
 94checking for strings.h... (cached) yes
 95checking for sys/types.h... (cached) yes
 96checking for sys/stat.h... (cached) yes
 97checking for MSG_NOSIGNAL... no
 98checking db_cxx.h usability... yes
 99checking db_cxx.h presence... yes
100checking for db_cxx.h... yes
101checking for main in -ldb_cxx... yes
102checking for main in -lminiupnpc... yes
103checking for boostlib >= 1.20.0... yes
104checking whether the Boost::System library is available... yes
105checking for exit in -lboost_system-mt... yes
106checking whether the Boost::Filesystem library is available... yes
107checking for exit in -lboost_filesystem-mt... yes
108checking whether the Boost::Program_Options library is available... yes
109checking for exit in -lboost_program_options-mt... yes
110checking whether the Boost::Thread library is available... yes
111checking for exit in -lboost_thread-mt... yes
112checking whether the Boost::Chrono library is available... yes
113checking for exit in -lboost_chrono-mt... yes
114checking whether the Boost::Unit_Test_Framework library is available... yes
115./configure: line 13446: syntax error near unexpected token `fi'
116./configure: line 13446: `  fi'

luke-jr commented at 4:06 am on September 7, 2013: member

Also not working for me, as it isn’t looking for bdb headers in the standard paths :(

jgarzik commented at 4:22 am on September 7, 2013: contributor

“standard path” varies even between Linux OS’s, because BDB 4.x by default just installs into /usr/include/db.h.

theuni commented at 11:06 pm on September 7, 2013: member

@shripadk I think I see it. (Also, you need to install pkg-config from ports).

Could some committer kill the extra ] that snuck in here: https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L411

wtogami commented at 1:05 am on September 8, 2013: contributor

/usr/include/features.h:330:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]

This warning prints very often with a plain ./configure. The default builds without any -O. Is this intended?

jgarzik said: AFAIK the default is -O2 -g warren said: confirmed, plain “./configure” lacks -O or -g

theuni commented at 5:22 am on September 8, 2013: member

Not intended. Now that this has been merged, it’s worth a bit of debate as to what reasonable defaults should be for debug/release configurations should be. Before, they were different per OS. I’d say that for debug, “-g -O2” probably makes sense.

luke-jr commented at 5:46 am on September 8, 2013: member

I find -O2 makes debugging impractical, so I personally prefer “-ggdb -O0”, but if -O1 is needed for security that’s probably good enough (besides, -O0 doesn’t even inline :p).

jonasschnelli commented at 3:11 pm on September 10, 2013: contributor

Getting error while trying to build with Qt. checking if qt should be enabled... configure: error: "qt support requested but qt could not be located. use --without-qt" How can i define the QT dir to use?

I tried --with-qt-incdir=, --with-qt-libdir=, --with-qt-bindir. Would it not be more practical to support --with-qt=/path/to/your/qt/dir?

MarcoFalke referenced this in commit fa13c5ce4f on Sep 21, 2016

michelvankessel referenced this in commit 1a3d5b890d on Dec 23, 2018

michelvankessel referenced this in commit 5c50a8608c on Dec 23, 2018

Bushstar referenced this in commit 53cce815d5 on Apr 8, 2020

laanwj referenced this in commit 1591e35049 on May 10, 2021

DrahtBot locked this on Sep 8, 2021

Autotools buildsystem #2943