build: libtoolize bitcoin core #3961

Minimal changes to make it possible to build shared libraries from Bitcoin Core:

• libbitcoincore_server.so.1
• libbitcoincore_wallet.so.1 (if --enable-wallet)
• libbitcoincore_cli.so.1
• libbitcoincore_common.so.1

Libtool automagically adds --enable-shared and --enable-static options. Bitcoind and the other executables link against the shared libraries if available.

This is a step towards being able to split off the wallet or in general to make it possible to develop other Bitcoin-based stuff outside the core repository. It is a minimal change contained to the build system. The current division into libraries may not be ideal for external usage, this can be improved later.

Default behaviour should be unchanged: building shared libraries is disabled by default. One step in implementing #3882 .

Trivial example Makefile for building bitcoind outside of the repository, using the provided pkg-configs:

LIBS='bitcoincore_cli bitcoincore_server bitcoincore_wallet'
CXXFLAGS=-pthread $(shell pkg-config --cflags ${LIBS})
LDFLAGS=-pthread $(shell pkg-config --libs ${LIBS})

bitcoind: bitcoind.o
        $(CXX) -o $@ $^ $(LDFLAGS)

TODO to make the library more useful, but this can/should be done in later commits:

• Rename -DHAVE_CONFIG_H to -DHAVE_BITCOIN_CONFIG_H to avoid it colliding with a project local config.h
• Make libbitcoincore_server library independent on setting of --enable/disable-wallet used to compile
  • Untangles libbitcoincore_wallet and libbitcoincore_server, there is now a sort of circular dependency between the two if ENABLE_WALLET is enabled
• Make it possible to use P2P / blockchain stuff without importing the RPC (both part of bitcoincore_sever right now...)

And probably quite a few other things.

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/235345d0cdf9ad50ebb9411080f54ebba59c27d0 for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

(Haven't looked deeply into the changes yet).

Does it make sense to do this before defining a stable external API? I certainly +1 the idea in general, but the current headers are very much entangled. I'm nervous that if applications start building against these, we're locked in to a very suboptimal api?

I'm currently working on breaking out the scripting into a clean external api as discussed on irc with @sipa. Worth looking into stubbing out an external api for other areas as well before advertising external linkage?

libtool versioning allows breaking ABI and API compatibility safely.

Libtool versioning allows breaking external ABI compatibility safely. We have no external API/ABI.

Say this is merged as-is. Now I want to write an application watches for transactions (or anything, just a quick example). I have to use all internal headers and structures.

With the next version of libbitcoincore_server, those structures and functions may change in any random way. How do I, as an program developer, build and maintain my program? How do I know which headers match the lib I'm linking against?

Shipping a lib creates a contract between the lib devs and the downstream devs. Most lib projects learn this the hard way. I'd prefer that we don't do the same. Besides that, the next logical step is bindings for other languages. How can those be created with no common API targets?

@theuni Please... I don't want to propose this as something to application developers. As I mention several times this is just a first step. As mentioned this is completely harmless to the normal build.

Even without installing headers (which are the external "interface") this can be useful, it's possible to save a bit of space this way: GUI, cli and daemon can share code.

Do you know of any way to do this as a hidden feature? So unless --enable-experimental-library-mode: use this for building don't install any headers or pkgtool descriptors by default, and don't install the libraries if static?

I need this myself and I don't care about API breakage for now. I like to experiment with externalizing the wallet and some analysis tools that I patch into bitcoind right now.

@laanwj sounds like you're looking for noinst_LTLIBRARIES. That allows for intermediate project linkage without actually installing anything (the way we currently do our .a's). The real libraries would then wrap an external api around those for installing shared libs and headers.

That doesn't work. I've tried that: noinst_ doesn't work for shared libraries, which makes sense, as they're needed at run time.

Anyway I'll change it to not install the headers and pkgtool files. It won't be used for application development. Is it OK then?

Please re-read the above. noinst would be the libtool substitute for our current .a targets. The real libs would be LTLIBRARIES, which combine the (internal) noinst with the (external) interface files.

My argument is that if shared libs are installed, people will use them. I think that's a very worthy goal, but the interfaces really need to be established first.

If the short-term goal is internal testing, what's wrong with just linking to them in the .libs folder?

...but that would mean that our executables don't use the shared libraries themselves, ruining the 'shared code' aspect.

Defining an external interface for general application usage will take years, or may even never happen. Sure, the current interface is a bit of a mess but it is useable if you're used to developing on the project itself.

Closing this, I'll use some other solution. I guess I learned something about autotools...

It's better than nothing... Lack of a sane way to link scripting is what is holding me back from trying to put together a GUI Script debugger... :/

@luke-jr I'm working on breaking out scripting as a standalone lib as discussed with @sipa. I'll ping you when there's something to play with.

@theuni Thanks

@theuni will that also include transaction (de)serialization, or really just the scripting?

@laanwj transactions are a separate project that is almost finished up. This is just scripting. I'm just breaking down the classes to reduce dependencies until it's self-contained.

Having experience with this, getting the in-tree build correct while similar libs and headers exist on the system is a real bear. You can build bitcoind with shared lib just fine, after much configury modification, but you must take care to make sure it is the in-tree shared lib and not the system shared lib.

For now, I'm just copying from bitcoin core what I need instead of bothering with libraries.

That's what I do, when I build small command-line tools that need knowledge of bitcoin binary data structures.

Another development option is more expensive, but potentially useful long term: Maintain a parallel tree, where (a) a select set of files are maintained in a cross-platform, library style yet (b) separate build systems and other source code files will differ.

Over the course of 1-2 years, maintain code in this way, then on a "flag day" delete the library-ized source code from bitcoin.git, and introduce a linking dependency of bitcoin-lib.git.