Allow compiling bitcoind as a shared library: libbitcoind.so #5084

We are all aware of alternative implementations of the bitcoin protocol. I've even helped personally implement two of them. There is an obvious fundamental problem with all alternative implementations that I'm sure everyone here sees: They're not bug-for-bug compatible with Satoshi's original implementation.

As we see more and more production platforms (BitPay, Coinbase, Circle, Blockchain, etc.) that absolutely depend on the adamant stability of their platform, with so much at stake, it is imperative that their backend protocol implementation be 100% compatible with the original protocol. That is non-trivial, and maybe even impossible to do with an alternative implementation.

Platforms and services like this are forced to use bitcoind as a backend on their servers if they want stability. Using just bitcoind with RPC calls has a number of limitations and hurdles.

This pull request alters the autoconf files to allow something like this:

$ git clone git://github.com/bitcoin/bitcoin.git
$ cd bitcoin
$ ./autogen.sh
$ ./configure --enable-daemonlib
$ make
...
$ ls -la src/libbitcoind.so

Any program, or binding for a platform (python, php, node.js, ruby, etc), can now link to the official bitcoin implementation. Luckily bitcoind decided not to use static functions and put function templates in every header file, which means linkage to any function in bitcoind should be fairly easily as every function is exposed to the program which links to it.

An Example

# Write a program which links to libbitcoind
echo '#include "bitcoind.h"' > my_program.cc
echo '#include "..."' >> my_program.cc
echo 'extern bool AppInit(int, char **);' >> my_program.cc
echo 'int main(int argc, char **argv) {' >> my_program.cc
echo '  return AppInit(argc, argv) ? 0 : 1;' >> my_program.cc
echo '}' >> my_program.cc

# Compile our program
gcc -I$HOME/bitcoin/src -I$HOME/bitcoin/src/obj \
  -I$HOME/bitcoin/src/config -I$HOME/bitcoin/src/leveldb/include \
  -DHAVE_CONFIG_H -g -O2 -fexceptions -frtti -fpermissive \
  -o my_program my_program.cc \
  -lboost_system -lboost_filesystem -lboost_program_options \
  -lboost_thread -lboost_chrono -lssl -lcrypto \
  $HOME/bitcoin/src/libbitcoind.so

# Start our process
./my_program -server -daemon

# Bitcoind logs should be spitting out
tail -f ~/.bitcoin/debug.log

A contrived example: Obviously this capability would most likely be used to link to different platforms' modules/bindings with that binding being dynamically loaded into python's/node.js'/php's/ruby's memory.

A More Complete Example

At BitPay, I have written a fully fledged node.js binding which links to libbitcoind.so and can do almost anything an alternative bitcoin implementation can do. It is called bitcoind.js.

Feel free to examine the documentation and source to see the actual benefits something like this would provide.

What this PR does:

• Moves main() out of bitcoind.cpp and into bitcoind_main.cpp. Moves all includes and variables out of bitcoind.cpp and into bitcoind.h so it can be included by both bitcoind.cpp and bitcoind_main.cpp if necessary.
• Exposes two or three functions in header files which will be useful for different platforms linking to libbitcoind. This required creating an rpcwallet.h header file.
• Adds --enable-daemonlib argument to configure.ac - this enables -fPIC for all object files instead of -fPIE. It also sets an AM conditional which is checked for in the automake file. Also resets -fvisibility to its default.
• When compiling the library, automake makes use of this AM_CONDITIONAL such that targets are overridden by one all: target compiling the library, and bitcoin-qt(+tests) is ignored.
• Now that all object files can be compiled as position independent code, the custom Makefile target I've added (vanilla Make syntax - I'm sure automake can do something fancier and compile a shared object through the help of libtool, but I couldn't manage to make it work) can link these object files together into libbitcoind.so with -shared -fPIC.

Issues and potential annoyances:

I figure I'll address these since I'm fairly certain these questions will be raised.

• If you want to compile bitcoind or bitcoin-qt, the object files compiled for libbitcoind are essentially useless since they're not -fPIE. You have to clean and recompile for scratch.

• Since bitcoind was never structured like a proper library, it's essentially one big mess: there is logging and arg parsing all over the place. A coder's best bet is to write their own stripped down version of AppInit2() which starts up the net.cpp threads and listens for RequestShutdown, eventually joining all threads. Currently, bitcoind.js simply starts up AppInit2() on a separate thread.

• Name Mangling: Node.js module bindings are written in C++. Lua and Ruby bindings are canonically written in C, which means in order to use libbitcoind without guessing bizarre function names, functions will have to be exposed with extern "C" on non-OO functions to force the stopping of mangled names. Of course, the workaround would be for Lua and Ruby users to write a C++ wapper and then link to them from their C binding.

• Internal API: When I first started writing bitcoind.js, I forked bitcoin v0.9.0 and turned it into a shared library. Once I rebased it onto the latest master, a lot of functions had changed and took differen params, a lot of classes were changed or added. I had to rewrite several functions. Any program depending on libbitcoind.so will have to deal with this as bitcoind is developed since it is not developed like a library.

• The Build: I'm sure there's a slicker way to compile this as an .so. I'm really not an Automake guru. I've fooled around a bit with the build here, but at the end of the day, someone smarter than me could probably do a better job at writing a src/Makefile.daemon.include.

  The noinst_LIBRARIES libraries and other magical features of automake are simply ignored when --enable-daemonlib is used. The .mm.o target compiles all the object files necessary to link them up into a shared library with a vanilla make target.

• Platforms: I'm not terribly familiar with OSX, but right now, I've only tested this build on Linux (Arch). I've had two friends try to build on OSX with unfortunate results that were apparently common in trying to compile bitcoind on OSX. I'm still unsure if this will work with Macintoshes. I have even less of an idea of how windows works compared to OSX. Maybe it could be made into a .dll somehow.

The Score

There may be better ways of doing this, but I think if the future of bitcoin and the bitcoin world is to be a bright one, bitcoind needs to be reshaped into an optional library in some form. I don't only speak on behalf of my employer here (BitPay), even though I'm sure we all agree. This also happens to be my personal opinion on the matter.

I open this to discussion. I'm willing to do whatever it takes to make Bitcoin into a library.

There is already work in progress to modularise Bitcoin Core. I believe the current plan is to start with a libbitcoinscript and libbitcoinconsensus.

@luke-jr this is @BitPay's implementation, which we'd like to use as a foundation for future work. It'd make sense to consolidate efforts if that is the case – can you please link us to the appropriate place where the existing efforts are being documented?

@martindale eg #4692 #4981 #5081 #5082

Yes, this seems like a very bad idea to structure it this way - exposing lots and lots of bitcoin core's internal interfaces externally is very poor library design as those interfaces are nearly guaranteed to change. Instead, the current approach is to take it slow, first by exposing one simple function - script verification, and hopefully adding more to that as is possible.

See-also: https://github.com/TheBlueMatt/bitcoin/commit/6d697838209a12eb5cba4f5f41caca53899b85d1

Do note that most of the bugs we've seen in alternate implementations have been in the handling of script details, so this should alleviate many of the issues, and others can be addressed over time.

Ideally, we can find a happy medium between @BitPay's need to move quickly with an implementation of a modular daemon library and the overall community's need to expose a resilient interface but not access internals (i.e., the existing work on #4692) – perhaps a solution would be to consider the two distinct needs, such that a daemonlib and libbitcoin (or libbitcoinconsensus?) would exist separately.

What, specifically, does Bitpay need here? Access to most information about blocks, etc is already available via RPC, and I would argue that, with a bit of testing and running it behind bitcoin core proxies, doing a full node implementation based on a libbitcoinconsensus_verify_script should be reasonably secure.

@martindale (note that libbitcoin already exists and is completely independent from Bitcoin Core)

I understand the use case, but I think this is a bad idea without having a stable API we want to commit to.

Either we'll end up breaking the API continuously, with people perhaps sticking to older versions and not upgrading because of incompatibilities, or we'll end up maintaining crufty old APIs because we can't afford the former to happen.

We are working pretty actively towards modularizing, reducing dependencies, and separating different pieces, and help with that is definitely welcome. The planned libscript/consensus/whatever is an example of something with a very simple API and we're really close to tiny near-dependencyless implementation for it.

I really prefer starting with easily-separatable parts here...

NACK from me as well, I agree with @sipa, @luke-jr, @TheBlueMatt completely.

Direct language bindings typically do have access to internal, oft-changing APIs (ex. SQL db world), as what matters is the downstream JS API.

There is no explicit or implicit obligation to make internal APIs stable and maintain them until they are old & crufty.

I would definitely suggest readers look at linked bitcoind.js and the provided example, as some of the comments seem to miss how the submitted code actually operates, and will be used.

This is much closer to a ZMQ-like integration. bitcoind is embedded in the node.js process(es), and provides a rapid, direct and 100%-consensus-compliant interface via JS.

This pull request is not introducing a node.js library though - it's introducing a libbitcoind.so whose API will break with every code change.

Can't we have a C++ top-level API implementation that does what the node.js code is doing, and expose that as a library? That way it would be reusable, and not break compatibility...

After some discussion in IRC, we're going to investigate using the -rdynamic flag to expose this as a dynamic library. If this works out, we will add commits to this branch as we make said changes. Thanks to everyone who is providing feedback.

Yes, this seems like a very bad idea to structure it this way - exposing lots and lots of bitcoin core's internal interfaces externally is very poor library design as those interfaces are nearly guaranteed to change.

Yes, I mentioned the burden of keeping what is normally an internal API stable in my PR. However, I see this as a problem for the platform binding maintainers, not the bitcoin core devs. I would rather change my bitcoind.js project with every update than hamstring bitcoin's improvements by forcing it to follow the traditional model of a library with a stable API.

I originally started writing bitcoind.js based off a fork of bitcoin v0.9.0 (it was just what I was more familiar with at the time, and it happened to be code from roughly 8 months ago). After rebasing onto the latest HEAD, it took me maybe an afternoon to update the code for the new internal functions, classes, types, etc: That's 8 month's worth of API changes updated in an afternoon. I don't think it will be that difficult for library authors to keep up, short of a major, major refactor.

I also want to make it clear, that bitcoind.js, for example, is a full bitcoin node. It simply is bitcoind, just running inside a node.js process. It creates a wallet, finds peers, and downloads the blockchain as soon as it starts. bitcoind.js is sort of a proof-of-concept right now, but I hope to make it production-ready soon. bitcoind.js can access any block or transaction, listen for block and transaction events as they come in, verify blocks and transactions, send bitcoins with automatically constructed transactions based on any available unspent outputs, or build a transaction manually and broadcast it. It has enough features to be useful.

Users of this or other similar libraries essentially can run bitcoind with their own programmatic enhancements. This is a huge step up from simply making RPC calls to a bitcoind server.

Also, the coders using bitcoind.js only see the external documented JS API. They don't have to deal with the nitty gritty bitcoin core API. As long as whatever library (for whatever platform, be it python, ruby, etc) stays up to date with bitcoin's internal interface, the only people who have to worry about it are the library maintainers, like myself. It isn't as difficult as it may seem at first.

I may be wrong, but I don't see this as much of a change. It's all hidden behind a compile-time flag. Very little has to change in the codebase. You can simply forget about it if you want to. Don't bother worrying about maintaining a stable API. Leave that to the platform library maintainers, or maybe there could be one well-maintained global wrapper for it as mentioned. Who knows?

I heard the -export-dynamic/-rdynamic argument mentioned here and there (which maybe only works with gcc? I suppose it doesn't work with clang, now the compiler of choice for Apple machines), which builds a binary which is both linkable/dlopen'able and executable. I think that's a great idea. Anything to make this simpler is good. Like I said, the build could be improved. That's why I posted this.

Personally, I can live with OSX users not being able to use this because I'm a total linux fiend, however, the ability for programmers to test on OSX is also probably important. All that being said, if this option does end up being used, -fvisibility would still most likely have to be set to default on all object files (correct me if I'm wrong). This is mentioned in my post and apparent in the diff.

I've spent the last several months working on two different alternative bitcoin implementations, yet I've decided I want to eventually write my own main wallet based on bitcoind.js or something like it.

My first attempt at a bitcoin wallet was using JSON RPC calls to bitcoind, and it was extremely limited. After two pull requests here trying to improve the RPC calls to make them more reasonable, I was shot down twice. So, I guess I took to more drastic measures: I wanted something better, and something that is not an alternative implementation.

Anyway, that's all I really have to say. Thanks for the consideration at the very least. I was hoping there would be a few more proponents, but whatever the decision, I trust it leads bitcoin in the right direction. I figure most here are likely to be in a better position to determine that than me.

There was a long discussion around this on #bitcoin-dev last night, so I'll refer you to that. But, essentially, if BitPay wants to update bitcoind.js after every release, they might as well simply maintain this patch as a part of that effort (its not that big a patch, so it really should be minimal effort). That very clearly sends the right message of "this is an unsupported way of doing things, and we're only doing it because we have the resources to keep this up-to-date".

The aforementioned stable C++ API is being developed inside-out, small consensus parts first. It's the right way to do things, but it will be a long while before that is a complete solution akin to what embedding bitcoind offers immediately: bug-for-bug match right down to the database & crypto libs.

That is a reasonable solution to offer users; there is nothing BitPay-specific about that solution and use case. It is normal to upstream changes like this, that will see use out in the wild.

Reiterating comments above, bitcoind.js provides a high speed, asynchronous interface to a running bitcoind. It produces something akin to the ZMQ integration.

I very well understand that RPC does not cover all needs - or even just the built-in wallet code as such.

But I never got a good answer as to why the abstraction is done in the Javascript layer rather than the C++ layer. If we had a libbitcoind.so which exposed an API similar to what bitcoind.js is doing now (which seems trivial), it would not break on every change, and it would be usable to more users than just Javascript ones.

The argument brought up yesterday in #bitcoin-dev against this was that doing it this would make it harder to develop the JS library, as that could otherwise be done by javascript developers. I don't understand this, as you still need the glue code to be written, which needs understanding of the (ever changing) internal Bitcoin Core code.

But I never got a good answer as to why the abstraction is done in the Javascript layer rather than the C++ layer.

That's definitely possible, and a good idea. I would love to write a C++ wrapper and abstract that into a library and simply maintain that as frequently as bitcoin itself changes. Libraries for any other language or platform could use that.

bitcoind.js was just sort of a proof-of-concept for now. A well-maintained C++ wrapper library could be possible in the future with the existence of libbitcoind.so.

Reiterating comments above, bitcoind.js provides a high speed, asynchronous interface to a running bitcoind. It produces something akin to the ZMQ integration. Why not implementing a ZMQ equivalent of the RPC (and whetever else you need) and call that from javascript? Thus using ZMQ as the glue.

if BitPay wants to update bitcoind.js after every release, they might as well simply maintain this patch as a part of that effort +1

I would like to close this pull request, given an easier equivalent short-term solution exists (-rdynamic), and that later work can be focussed around more proper encapsulation (either by having a libbitcoind.so, or by building upon the libbitcoinconsensus.so work).

I feel your pain @chjj. I also tried to do this once, see #3961. Even without a stable interface there are valid use-cases for something like this. The drawback is that whatever you're developing needs to be developed in lock-step with bitcoin core's internals, but for some uses that can be acceptable.

For those uses the only feasible way right now is to manually patch bitcoind to yield a library, or re-implement everything (as bitcoin-pythonlib did). I went with using bitcoin-pythonlib in my case. But I'm not opposed to doing this (IF it can be done without impact to other library-zation efforts such as the consensus library).

Edit: -rdynamic sounds like a nice solution though. I hadn't though of that...

Thanks for your sympathy, @laanwj. I really wish this wasn't seemingly dismissed so quickly. I should have figured it's been tried before. I think it can be useful anyway and isn't too intrusive to the current build even if it appears to be a step in the wrong direction at first glance. I do like the way you did it and I'm glad you learned something about libtool (meanwhile I'm still trying to figure all of it out). :)

As of now, I've rewritten the Makefile.daemon.include to make it much simpler, with no hardcoded list of object files, so bitcoin core devs likely don't have to bother with modifying it much in the future. They can simply ignore it. Furthermore, leave the worrying of the API to the devs using this and wrapping it. There's no need to create a documented stable API.

Also, as a minuscule aside, you'll notice I've also renamed bitcoind_main.cpp to bitcoin-main.cpp to match source file naming convention.

(NOTE: It still doesn't build on OSX: OSX seems to require some esoteric magic in order to work. If someone could give insight into that it would be fantastic. I don't have an OSX installation of my own, so I can't really test too often).

Does -rdynamic not work?

@sipa, apparently not on OSX. Which is why I'd like to find a sane way to build this on OSX (people I know have problems just compiling bitcoind itself on Apple machines given OSX's gigantic mess of directory structures, headers, and libraries). I imagine there will be some hurdles.

Closing this for now. bitcoinconsensus has been merged for transaction verification, and more of the consensus will be available as library over time. People that need this can find it (or use -rdynamic).