Decouple build system from source. Use CMake to help newcomers and ecosystem! #19693

Problem

New Developers unable to quickly get up and going in their IDE

The current build system is very brittle and the custom tailored nature of it leaves the development team having to address constant breaking changes and new architectures etc. It also leaves new comers to the project in a state of despair when they realize that the current IDE cannot pick up the project.

For example in this issue #13694 , the user asked "where I could find the guide on how to install an IDE and debug the code?" because his top of the line IDE, CLion works by using CMake. The other well known C/C++ IDEs that are currently used are able to and appear to prefer the CMake build system. Sadly, the sarcastic response (i hope) from @sipa was "Most developers don't use IDEs as far as I know".

For the Windows users out there, in issue #14118 , the issue got alot of support to the idea of adding CMake, as one is able to quickly work on C/C++ projects given Visual Studio's and VSCode support for CMake projects.

Improve ecosystem by having option to build shared libraries

This last point will be a simple theory, but if CMake is added, one could imagine that the ecosystem to grow and flourish and bring in new support and tooling with it. For instance, if i wanted to instead wrap around the C++ code using CFFI and call directly from Python/JS/Whatever I am unable to as where would I even begin to get started on compiling a library as opposed to the executable in a neat manner? Right now the best way to interact with bitcoin is via the rpc it seems, but why not directly ?

The other direct benefits of having a "simple" CMake build system is we can undo the tight coupling of what should be the "build" with the source code. When developing, the sanitizers can be added and help ensure there is strong checks with proper memory usage and addressing etc.

Proper sanitizing and bug catching

For instance, I found a buffer overflow in the tests within the ctaes library (inside crypto) which was found trivially after setting up a proper standalone CMake file for it. This was found within hours of me poking around and was there for a number of years. Makes one wonder how many other bugs are simple low hanging fruit, just waiting to be picked.

Solution

According to The Developer Ecosystem in 2020: How C++ is Doing , the C++ community is moving with conviction to CMake as their build system of choice. Over the last couple years its gone from 34% to 53% of developers choosing it over Makefiles or Visual Studio. If the trend continues, it is imperative the project invests in improving the current status quo.

Begin adding CMake files to cover the major OSs Windows, OSX, Linux on the major processors x86_64. Then expand out to ARM and the smaller targets like Raspberry Pis and the like. Eschew the many optimizations and custom-tailored functions/modules for specific architectures and just add the generic standard library versions until the entire project builds and tests properly.

While this is a "monumental task" as noted by @sipa , the hardest part will be decoupling of the many build steps with source code, but thats more tedious and repetitive than anything.

I can and already have a couple CMake files for some modules like crypto and ctaes to start off. What I am not sure of exactly is the flow of direction between the libraries in the "bitcoin-core" project and this one. Which one follows which?

Hi @whb07, sorry to hear of your difficulties in onboarding. AFAICT it is true that most of the regular developers here don't use an IDE (I don't either). If helpful, when I was getting started, I wrote this little guide to help others begin building this project from source and have been maintaining it: https://jonatack.github.io/articles/how-to-compile-bitcoin-core-and-run-the-tests as well as other guides for the first months of getting started at https://jonatack.github.io/articles.

The current build system is very brittle and the custom tailored nature of it leaves the development team having to address constant breaking changes

Can you point to some examples of it being very brittle, or the constant breaking changes? I'm not sure what you mean by custom tailored, unless you're talking about our depends system? However in that case, it's "tailored" to serve a specific purpose.

Improve ecosystem by having option to build shared libraries if i wanted to instead wrap around the C++ code using CFFI and call directly from Python/JS/Whatever I am unable to as where would I even begin to get started on compiling a library as opposed to the executable in a neat manner

The current build system can and does build shared libraries. i.e libconsensus. There is a large amount of discussion related to libconsensus in this repository. There are also some PRs that demonstrate integration with other languages, including Rust.

When developing, the sanitizers can be added Proper sanitizing and bug catching

Our build system currently has support for debugging/sanitizers/fuzzers etc and they are run as part of our CI. We also have some getting started type documentation for new contributors, i.e https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md. Is there a particular feature or support for a tool that you think is missing?

For instance, I found a buffer overflow in the tests within the ctaes library (inside crypto)

Great. Have you reported this upstream anywhere? See https://github.com/bitcoin/bitcoin/blob/master/SECURITY.md.

In any case, you're likely to get more interest if you actually present a branch with all the changes required, as they are going to be substantial, involve multiple sub projects, third-party libraries, compilers, build tools etc. For what it's worth I'm still Concept NACK.

I agree 100% @jonatack with your statement of " most of the regular developers here don't use an IDE". Where as most of the world does in fact use one and if the goal of bitcoin is to grow adoption then by definition, the metric here is to help these set of users. Please see any number of surveys to include the one linked above.

I don't need help onboarding. I am pointing out the need to move the technology along. It seems clear CMake is what the industry has coalesced around, look at LLVM using it or INSERT_ANY_HIGH_TECH_PROJECT. It is clear CMake isn't a fad and as such, given the massive gains to be had, should be something to work for.

edit: added comment about surveys. insert company flag being escaped somehow

There are no plans to switch build system or add another build system. I think CMake is an okay build system but there are more pressing issues right now.

Our build system currently has support for debugging/sanitizers/fuzzers etc and they are run as part of our CI. We also have some getting started type documentation for new contributors, i.e https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md. Is there a particular feature or support for a tool that you think is missing?

For instance, I found a buffer overflow in the tests within the ctaes library (inside crypto)

Great. Have you reported this upstream anywhere? See https://github.com/bitcoin/bitcoin/blob/master/SECURITY.md.

Clearly this isn't happening as thoroughly as it should be isnt it? The ctaes library goes back a number of years and that bug has yet to be filed for. Perhaps the main executables have the extensive flags and checks on them but not maybe the intermediary objects /libs? Point is, where in the number of files related to GNU's automake, config, Makefiles can one quickly fix this, I'm confused as to where/how to start.

There are no plans to switch build system or add another build system. I think CMake is an okay build system but there are more pressing issues right now.

I agree, what is the will if a CMake build PR was added? Thats more of the general question I suppose I am posing.

To save the time of others here are some of the reasons for Concept NACKs or skepticism previously expressed. Ideally the author would steelman all the opposing arguments or at least address them but in the absence of that I will post them here.

"I'm not really +1 on having to maintain two different build systems simultaneously. Given that most devs here aren't using Windows, most won't think to add/adjust any Windows specific Cmake stuff, because they wont notice while compiling (which will likely hold up review)."

"One issue with cmake is that it's a makefile generator while autoconf is a configure script generator, the configure script created by autoconf is designed to be portable, there isn't really a direct equivalent with cmake AFAIK. In general autoconf projects are considered easier to work with for people packaging software even though the build system may be somewhat more complex to maintain."

"I find autotools to typically have much better cross compilation support than CMake."

"it seems like working on CMake is a waste of time (unless someone can really show that the result will be a lot simpler to manage and understand)"

"I would personally be in favor of switching a build system only if it gives us one of these features: Reduce the complexity of the build system by a lot (making it easier to maintain and adjust over time) Reduce the amount of stuff to maintain (i.e. MSVC stuff). Can show actual real superiority over the current build system (highly doubt that can be the case). A nice feature but I don't think that's enough to switch is somehow faster builds."

I agree, what is the will if a CMake build PR was added? Thats more of the general question I suppose I am posing.

It's not going to be merged, I expect. It's extra maintenance work, as well as for every contributor, to have to update and test three build systems for every added/removed implementation file, or change in folder layout.

(CMake, as it can generate MSVC files, could in principle replace the MSVC build system, and there would still be two… But the MSVC "build system" is very simple. Right no there is no crossplatform logic needed there, a cross-platform CMake build system would be much more complex)

Can you point to some examples of it being very brittle, or the constant breaking changes? I'm not sure what you mean by custom tailored, unless you're talking about our depends system? However in that case, it's "tailored" to serve a specific purpose.

There is literally a "compat" directory dedicated to manually checking for things that should be left off to the build system. Many of the config and makefiles are maintained and carefully crafted rather than declared as to what they should be for the build system to generate.

(CMake, as it can generate MSVC files, could in principle replace the MSVC build system, and there would still be two… But the MSVC "build system" is very simple. Right no there is no crossplatform logic needed there, a cross-platform CMake build system would be much more complex)

Can you explain to me why perhaps the trend for developers globally is to increase their usage of CMake, while the current build system used now is slowly losing share?

CMake is more portable and easier to maintain is one of the many reasons its growing. I am not here to make this a flamewar of opinions, all I'm pointing out is the efficiencies gained to the project by adopting such a tool.

I've seen here and across other issues/threads a common vein of the main argument being "well we dont use an IDE" as a good point to NOT improve the tooling for the project.

Most C++/C devs use an IDE. Most of the younger devs are in fact using IDEs and are using CMake, if Bitcoin is expected to be here for the long run then perhaps some investment should be created for the yet-to-be developers of this project.

Side note: It is possible to develop Bitcoin Core in CLion, see #13694 (comment)

Side note: It is possible to develop Bitcoin Core in CLion, see #13694 (comment) @MarcoFalke it isn't just about using a specific IDE. It is about reducing maintenance costs and increasing portability with a properly setup build system.

I noticed your comment about the CLion and how some documentation should be created to show how to use CLion with this project currently. Now that doc needs to be maintained and then another doc needs to be created for the future competitor of CLion in 5 years. Now you have at least 2 more docs to maintain besides the source code.

Or just like updating the C++ standard used in this project, the build system could be updated.

@whb07 Realistically I think the only way you are going to get CMake usage to happen with bitcoin is to truly "Decouple build system from source" and create an independent "bitcoin-cmake" repository similar to https://github.com/bitcoin-core/packaging that houses cmake scripts and packaging separately from the sources. You could use gitsubmodules or subtrees to help with this or just make the CMake scripts flexible enough to build different versions of bitcoin.

The people who actually devote time and effort to maintaining the bitcoin build system already have many issues to deal with and lots of work to do (supporting reproducible builds, packaging and signing, minimizing dependencies) and don't have a huge amount of experience with CMake. It would be unrealistic to expect them to learn how to use and maintain a CMake build.

Personally, I have a lot of experience with CMake and think it's much nicer than autotools. But I don't maintain the bitcoin build system, and I am glad if the people who do maintain the build system use the tools they're most knowledgeable about.

@ryanofsky I agree with you. I'm not here saying "Hey guys, listen stop what youre working on and start working on CMake". I want to be clear I am not saying that .

I am however asking "assuming a proper PR or solution was created that can use CMake would that be incorporated or used?"

I am for spending ( me/some others) N units of energy/time in such a feature. I am not for spending N + M units, where M is the energy required to promote and start a campaign getting the core devs to change their opinions on whether or not they like to use IDEs 😰 .

As an extra data point, looks like the C++ community in Reddit overwhelmingly prefers CMake https://www.reddit.com/r/cpp/comments/i7825h/build_system_whats_your_favorite/

I am however asking "assuming a proper PR or solution was created that can use CMake would that be incorporated or used?"

No-one is going to give you a guarantee that your (theoretical) changes will be merged. That's just not how open source works.

Feel free to open a PR, and if contributors are interested, it might get some review / feedback. However the project certainly isn't about to change build systems, and has hundreds of other more important issues to deal with right now.

If there is a CMake file that replaces and removes the msvc config without any trade-offs, then I don't see a downside. We already have to maintain the msvc config and python script to autofill the msvc build script, so if instead we have a python script to autofill some CMake file, there is no additional cost. Obviously that CMake file is going to be for developers only (like the current msvc) and won't support fancy build configs (like ./depends).

@sipsorcery @NicolasDorier Would you happen to have any thoughts on this w/re msvc?

@whb07

Can you explain to me why perhaps the trend for developers globally is to increase their usage of CMake, while the current build system used now is slowly losing share?

Following trends is not the business of software that is meant to be stable, in fact it should be as adverse to change as possible. Bitcoin Core development is already a significant challenge to follow as an outsider with only fleeting contact with the project, not because it's not using the fanciest and greatest, but the amount of changes that happen between releases to tooling and interfaces.

@whb07

For instance, if i wanted to instead wrap around the C++ code using CFFI and call directly from Python/JS/Whatever I am unable to as where would I even begin to get started on compiling a library as opposed to the executable in a neat manner? Right now the best way to interact with bitcoin is via the rpc it seems, but why not directly ?

Literally nobody is doing this, or should ever want to be doing this. Those that have tried, failed spectacularly.

If there is a CMake file that replaces and removes the msvc config without any trade-offs, then I don't see a downside.

This is a good idea, better than my suggestion to implement cmake support in a separate repository, if replacing the msvc build this way is acceptable.

@whb07

Can you explain to me why perhaps the trend for developers globally is to increase their usage of CMake, while the current build system used now is slowly losing share?

A properly declared CMake file is more portable and easier to maintain. This isn't some "fad" its many years in the making, I've already noted that notable projects like LLVM are making use of it. I agree that a stable project like this one shouldn't chase after shiny new objects, but this isn't one. If you can follow that reddit poll of preferred build systems, it appears that Makefiles doesnt do "as bad", its because there weren't more options. CMake got nearly 4x what Makefiles did.

Now there are times when a good idea/project are unknown and secret to most of the world, Bitcoin is one of them. Then there are the things everyone knows about and is fully valued, like Makefiles which are being replaced by a better product, like CMake or Meson and others.

@whb07

For instance, if i wanted to instead wrap around the C++ code using CFFI and call directly from Python/JS/Whatever I am unable to as where would I even begin to get started on compiling a library as opposed to the executable in a neat manner? Right now the best way to interact with bitcoin is via the rpc it seems, but why not directly ?

Literally nobody is doing this, or should ever want to be doing this. Those that have tried, failed spectacularly.

No idea how this is a good argument. Perhaps its a consequence of how hard it is to build standalone shared libraries ? Bitcoin is an example of a good outcome amongst the many dead failed attempts to create a decentralized money system. Had you said those exact words right up to the end of 2008/early 2009 you would have been 100% correct.

Personally, and same as when it was discussed in #14118, I'd be interested in trying out a cmake build script. I regularly build on 4 different platforms, where the build on each has its own little quirks. If cmake, or any other build tool, abstracted that away it's hard to see how it wouldn't be worth at least considering further, even if it's only as a supplementary tool rather than an Autoconf replacement. There really needs to be something concrete to test though.

@whb07 sorry to interrupt the discussion here, but you mentioned you believe you found a problem in the ctaes repo. Could you please open an issue there or at least state here specifically what you found so that someone else can open an issue?

As a result of your comment I re-reviewed the entire library then spent a bit of time throwing static analysis tools at it and found nothing particularly noteworthy.

Hey so i just ran it again, i mixed the executable that caused it. Its apparently when running bench which is a relief. Probably why I didn't actually file a ticket the moment i saw it. Here is the sanitizer catching it, found using Clang's Asan : https://gist.github.com/whb07/0850471de745d115a818cd6b6dadf102

In some of my workflows I add a GNUmakefile to the repo that can be used to setup specific build conditions that I like/prefer...

I append

-include Makefile

to the bottom of the GNUmakefile to include Makefile if it exists

REF:https://www.gnu.org/software/make/manual/make.html#How-Make-Works

By default, when make looks for the makefile, it tries the following names, in order: GNUmakefile, makefile and Makefile.

It is possible to overload/override what ever Makefile is generated by configure.

This could be a practical way to customize the workflow of a particular IDE.

In some of my workflows I add a GNUmakefile to the repo that can be used to setup specific build conditions that I like/prefer...

I append

-include Makefile

to the bottom of the GNUmakefile to include Makefile if it exists

REF:https://www.gnu.org/software/make/manual/make.html#How-Make-Works

By default, when make looks for the makefile, it tries the following names, in order: GNUmakefile, makefile and Makefile.

It is possible to overload/override what ever Makefile is generated by configure.

This could be a practical way to customize the workflow of a particular IDE.

Interesting. But still falls short. I guess I wasn't very clear in the main post that easier onboarding and use of newer tools would be a positive side effect of implementing CMake. I didn't mean to say that this project should use CMake to have access to IDEs. Thats just a cherry on top.

There are a number of places where the code does some checking for X environment or Y library. Or you have python scripts and weird patches or whole directories like compat dedicated to custom tailor something to adjust for the infinite hardware /software combinations.

Many of these scripts and whole modules can be done away with by having a more portable build system.

@whb07

For instance, if i wanted to instead wrap around the C++ code using CFFI and call directly from Python/JS/Whatever I am unable to as where would I even begin to get started on compiling a library as opposed to the executable in a neat manner? Right now the best way to interact with bitcoin is via the rpc it seems, but why not directly ?

Literally nobody is doing this, or should ever want to be doing this. Those that have tried, failed spectacularly.

No idea how this is a good argument. Perhaps its a consequence of how hard it is to build standalone shared libraries ? Bitcoin is an example of a good outcome amongst the many dead failed attempts to create a decentralized money system. Had you said those exact words right up to the end of 2008/early 2009 you would have been 100% correct.

I don't get the impression that you've a lot of experience working with this software, there is simply no value to attempting to use Bitcoin in this manor, there is no use case which comes close to justifying doing this. It has been tried before with Bitpay and their "bitcore" application, which resulted in people using tortured, ancient versions of Bitcoin with unsafe accesses through javascript peppered through it.

It doesn't really matter if you think that Bitcoin is revolutionary, exceptional, or standing on the ashes of other decentralized systems; it's simply not a good idea and results in people complaining when the interface they were calling is broken- even when it was never meant to be a public interface in the first place.

@wpeckr I don’t ever presume to be all-knowing about all possible ideas and yet to be created products. I hope that you’re also not insinuating that you are. It is precisely because I don’t know, that making it easier should be a goal.

Either way, that line of reasoning leads nowhere. I’m only stating that a better build system would lead to such positive side effects.

As far as “successful” products/projects that bind around a lower level C/C++ library are far too many to enumerate.

Hey so i just ran it again, i mixed the executable that caused it. Its apparently when running bench which is a relief. Probably why I didn't actually file a ticket the moment i saw it. Here is the sanitizer catching it, found using Clang's Asan : https://gist.github.com/whb07/0850471de745d115a818cd6b6dadf102

Thanks, fix is here: https://github.com/bitcoin-core/ctaes/pull/16

Going to close this. As mentioned, the project has no plans on changing build system right now. If there's a next step, it might be swapping out the existing MSVC build to use CMake. However that should be a PR, and I think it makes sense that any discussion continue there.