Building requires >1GB memory

TheBlueMatt commented at 10:43 pm on September 10, 2015: member

CXX test/test_test_bitcoin-script_tests.o g++: internal compiler error: Killed (program cc1plus)

script_tests requires more than 1GB memory to build, killing a lot of VMs.

theuni commented at 4:24 am on September 11, 2015: member

I don’t think an ICE is really our concern. I assume clang manages ok?

TheBlueMatt commented at 4:26 am on September 11, 2015: member

Hmm? I’d say it being impossible to build Bitcoin Core by default on many VPSes is our concern.

On September 11, 2015 12:24:16 AM EDT, Cory Fields notifications@github.com wrote:

I don’t think an ICE is really our concern. I assume clang manages ok?

Reply to this email directly or view it on GitHub: #6658 (comment)

sipa commented at 4:29 am on September 11, 2015: member

Internal Compiler Errot

luke-jr commented at 4:31 am on September 11, 2015: member

@TheBlueMatt Try using CXXFLAGS="–param ggc-min-expand=1 –param ggc-min-heapsize=32768"

TheBlueMatt commented at 4:32 am on September 11, 2015: member

The ICE in question is OOM killer eating g++.

On September 11, 2015 12:29:35 AM EDT, Pieter Wuille notifications@github.com wrote:

Internal Compiler Errot

Reply to this email directly or view it on GitHub: #6658 (comment)

TheBlueMatt commented at 4:33 am on September 11, 2015: member

@Luke-Jr: I can get it to build fine (–disable-tests), I just care that we fix the actual issue.

On September 11, 2015 12:31:36 AM EDT, Luke-Jr notifications@github.com wrote:

@TheBlueMatt Try using CXXFLAGS="–param ggc-min-expand=1 –param ggc-min-heapsize=32768"

Reply to this email directly or view it on GitHub: #6658 (comment)

theuni commented at 4:34 am on September 11, 2015: member

I’d be concerned if no compiler could build on that VPS. However, I’d be willing to bet that it’s just a matter of an inefficient gcc version. If clang dies the same way, I’ll buy you a beer this weekend :)

luke-jr commented at 4:35 am on September 11, 2015: member

@TheBlueMatt Memory use is not an issue, if it correctly builds with the appropriate CXXFLAGS… IMO

TheBlueMatt commented at 4:38 am on September 11, 2015: member

This is a default gcc on a relatively recent Debian. I really don’t think the response to “I can’t build Bitcoin Core on a vps with 1GB of memory with default parameters” should be “oh, but if you set these flags you can!”

On September 11, 2015 12:34:57 AM EDT, Cory Fields notifications@github.com wrote:

I’d be concerned if no compiler could build on that VPS. However, I’d be willing to bet that it’s just a matter of an inefficient gcc version. If clang dies the same way, I’ll buy you a beer this weekend :)

Reply to this email directly or view it on GitHub: #6658 (comment)

unsystemizer commented at 12:23 pm on September 11, 2015: contributor

0.11 does require more RAM and in my experience --disable-tests doesn’t help. 0.10 could be built on systems with 512MB of RAM (with and without --disable-tests).

jgarzik commented at 12:44 pm on September 11, 2015: contributor

Any instrumentation on why script_test takes so much memory? Would prefer to discern “why” before taking any action.

In general we should be able to build Bitcoin Core on a 1GB VPS.

gmaxwell commented at 12:44 pm on September 11, 2015: contributor

Y’all being overly pedantic. Matt identified a usability problem: User obtains and attempts to compile the software with normal instructions on a common and reasonable platform type, and it crashes.

If you want to suggest that the docs should have instructions with low mem cflags or whatnot, OKAY, but without that it doesn’t stop being an issue because just us experts can work around it. Our memory usage is kind of obnoxious during compile time IMO, and if we could do some minor tweaks to reduce the peak that would be nice too– but at the very least we should provide some instructions.

laanwj commented at 0:47 am on September 18, 2015: member

This is actually known and documented: https://github.com/bitcoin/bitcoin/blob/master/doc/build-unix.md#system-requirements (feel free to add the recommendation for low-memory build flags tho!)

I would have expected that no longer using boost::spirit (for JSON) and boost::asio (for RPC) would reduce compilation memory usage - unfortunate to hear that it became worse.

Anyhow: cross compilation and pseudo-static compilation is extremely easy with the depends system, I’d recommend using that.

laanwj closed this on Sep 18, 2015

laanwj commented at 0:56 am on September 18, 2015: member

Oh wait, the compilation of that file uses more than 1GB of memory? Ok, reopening, that is definitly not good.

laanwj reopened this on Sep 18, 2015

laanwj added the label Build system on Sep 18, 2015

TheBlueMatt commented at 7:46 pm on September 18, 2015: member

Yes, the VPS in question has a total of 1.5GB memory (1GB ram + .5 swap), and nothing else of note was running at the time.

laanwj commented at 5:04 pm on September 29, 2015: member

I measured using gcc 4.8.4 (64 bit): Top 10 memory usage during build:

nr	source	elapsed	maxresident (kB)
1.	main.cpp	0:36.58	1248524
2.	init.cpp	0:24.24	964136
3.	wallet/wallet.cpp	0:22.11	896096
4.	test/script_tests.cpp	0:19.44	859708
5.	net.cpp	0:18.85	813836
6.	wallet/rpcwallet.cpp	0:14.16	727956
7.	wallet/walletdb.cpp	0:13.48	719388
8.	wallet/rpcdump.cpp	0:12.42	697660
9.	qt/walletmodel.cpp	0:11.26	680500
10.	rpcrawtransaction.cpp	0:11.44	666548

Will try with clang some other time.

laanwj commented at 10:05 am on September 30, 2015: member

Same information, but for clang 3.6.0 (also 64-bit):

nr	source	elapsed	maxresident
1	main.cpp	0:28.34	723960
2	wallet/wallet.cpp	0:15.93	522068
3	init.cpp	0:19.06	517080
4	net.cpp	0:14.36	428924
5	wallet/rpcwallet.cpp	0:11.53	357224
6	wallet/walletdb.cpp	0:10.16	356792
7	test/script_tests.cpp	0:12.73	342376
8	qt/walletmodel.cpp	0:08.79	333660
9	test/test_bitcoin.cpp	0:08.21	326100
10	wallet/rpcdump.cpp	0:09.19	322376

The contenders are the (mostly) same, although it is faster and uses significantly less memory in all cases. So we should add a recommendation to use clang on lower-memory systems.

laanwj commented at 11:57 am on September 30, 2015: member

I’m not sure why it is this “bad”. main and init have huge complex implementation files, but of script_tests you’d say it isn’t that bad - just 1016 lines.

Possibly it has to do with embedding the test data as includes:

0410344  test/data/script_invalid.json.h
1530534  test/data/script_valid.json.h

theuni commented at 5:53 pm on September 30, 2015: member

@laanwj I suspect you’re right about the test data. Maybe older gcc re-reads that each time it’s needed and balloons in size as a result. @TheBlueMatt Could you test with https://github.com/theuni/bitcoin/commit/1a107da7ef4b90bff4165357375b58c41f8e01c9 and see if that helps? That moves the test data into cpps instead. Edit: Whoops. Here’s one that actually works: https://github.com/theuni/bitcoin/commit/4b69de1820e19b161f3071538551ad60e20644f2

TheBlueMatt commented at 0:16 am on October 1, 2015: member

Hmm, the VPS has since been upgraded to have more memory (yay pruning means I get more memory and less disk space…) and doesnt kill script_tests anymore :( What may have been happening is the host may have been retroactively killing processes after some other process had casued the VM to hit some high memory bar. In any case, it is just Debian Jessie, so gcc 4.9.2-10.

TheBlueMatt commented at 0:18 am on October 1, 2015: member

Still, main max resident of 1.2GB will probably not build on most VPSes today, but sadly thats a harder one to fix, and probably a very long-term thing.

gmaxwell commented at 0:20 am on October 1, 2015: contributor

Rather than recommending clang (which would send more hosts into a less tested and potentially consensus inconsistent config) we could recommend compiler flags that reduce memory usage.

unsystemizer commented at 4:22 am on October 1, 2015: contributor

I ran a comparison test yesterday measuring the peak memory consumption of the Bash shell session in which I was working. gcc - 835 MB, clang - 823 MB (32-bit Raspbian OS). Not a big saving. (I don’t know how @laanwj measured RAM consumption, otherwise I would have used his approach).

People low on RAM could use binary builds if they don’t have enough RAM to build, or build without tests. Not everyone should be able to build from source on their system (or, in most hosting environments, they can upsize their RAM, build, and then downsize?).

laanwj commented at 7:32 am on October 1, 2015: member

@gmaxwell Possibly both - we’re using clang for OSX builds already, so it’s not that much of an unknown. I’ve switched to clang for my run-of-the-mill builds on Linux to save time. @cfields The solution in https://stackoverflow.com/questions/5479691/is-there-any-standard-way-of-embedding-resources-into-linux-executable-image to use the .incbin directive is also quite nice - create a file blob.S:

0    .global blob
1    .global blob_size
2    .section .rodata
3blob:
4    .incbin "blob.bin"
51:
6blob_size:
7    .int 1b - blob

which can just be compiled with gcc along with the rest of the source files. This avoids any kind of conversion. @unsystemizer see https://gist.github.com/laanwj/108877a28ec03836568a

In general I recommend to use cross-compiling to build for resource-constrained systems. This a) avoids having a compiler toolchain on the target b) is much faster c) Bitcoin Core’s depends system makes this easy.

jarret commented at 4:33 am on January 25, 2016: contributor

After tripping up on this initially, I succeeded in applying Luke-jr’s workaround by giving the CXXFLAGS at configure time:

0$ ./configure --disable-wallet CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768"

On a Raspberry Pi 2 building with a single core:

0$ time make
1(success)
2real    159m41.895s
3user    154m55.090s
4sys     3m1.710s

The memory use seemed to stay below 500 MB throughout, but I didn’t measure that precisely.

jarret referenced this in commit abc3c15ab9 on Jan 27, 2016

jarret referenced this in commit f9298cc60e on Jan 28, 2016

laanwj added the label Resource usage on Feb 16, 2016

rebroad commented at 2:38 pm on February 26, 2016: contributor

Also having trouble compiling main.cpp on a Raspberry Pi. Perhaps this file can be split in two?

laanwj commented at 10:12 am on June 27, 2016: member

@sipa noticed that with recent compilers (gcc 5.3.1 in his case), enabling -flto (link time optimization) reduced compile-time memory usage for main.cpp to 850 MiB, without unduly raising link-time memory requirements. This may be another useful suggestion for limited-memory compiling.

TheBlueMatt commented at 1:23 am on December 2, 2016: member

On gcc 4.9.2, after #9260, objects which fail to build on a kvm host with 1GB of memory (with wallet, no GUI):

libbitcoin_server_a-init.o libbitcoin_server_a-validation.o libbitcoin_server_a-net_processing.o rpc/libbitcoin_server_a-rawtransaction.o wallet/libbitcoin_wallet_a-rpcdump.o wallet/libbitcoin_wallet_a-rpcwallet.o wallet/libbitcoin_wallet_a-wallet.o

TheBlueMatt commented at 3:28 am on December 2, 2016: member

Note that the above list is the same on gcc 6.2.0.

laanwj commented at 3:55 am on December 2, 2016: member

On gcc 4.9.2, after #9260, objects which fail to build on a kvm host with 1GB of memory (with wallet, no GUI):

This is in contrast to: before #9260? So splitting up main makes this worse? That’s not what I would have expected.

TheBlueMatt commented at 4:09 am on December 2, 2016: member

I’m sure its better, but not better enough that both halves dont both end up needing >1GB.

gmaxwell commented at 2:26 pm on December 2, 2016: contributor

I wonder how much less memory it will take once we start eliminating boost in earnest?

TheBlueMatt commented at 5:46 pm on December 2, 2016: member

Considering net_processing only took >1GB of memory after I re-added boost/thread.hpp, probably a lot.

ciconsulting commented at 9:31 am on May 27, 2018: none

I am also unable to build on a 1GB VPS running Ubuntu 18.04.

 0Making all in src
 1make[1]: Entering directory '/home/satoshi/bitcoin-source/bitcoin/src'
 2make[2]: Entering directory '/home/satoshi/bitcoin-source/bitcoin/src'
 3make[3]: Entering directory '/home/satoshi/bitcoin-source/bitcoin'
 4make[3]: Leaving directory '/home/satoshi/bitcoin-source/bitcoin'
 5  CXX      libbitcoin_server_a-init.o
 6
 7cc1plus: out of memory allocating 5028920 bytes after a total of 40685568 bytes
 8Makefile:5606: recipe for target 'libbitcoin_server_a-init.o' failed
 9make[2]: *** [libbitcoin_server_a-init.o] Error 1
10make[2]: Leaving directory '/home/satoshi/bitcoin-source/bitcoin/src'
11Makefile:9462: recipe for target 'all-recursive' failed
12make[1]: *** [all-recursive] Error 1
13make[1]: Leaving directory '/home/satoshi/bitcoin-source/bitcoin/src'
14Makefile:747: recipe for target 'all-recursive' failed
15make: *** [all-recursive] Error 1

Sjors commented at 4:36 pm on June 5, 2018: member

On a related note, as I’m compiling on a Nano Pi with 1 GB RAM and 4 cores, I initially tried -j5, but there are a number of files where that eats into swap and everything grinds to a halt. -j1 seems the only way to go, but this seems wasteful for the many files that don’t require much RAM.

I did use CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768. Is there a way to make the compiler use less than e.g. 200 MB RAM per process? Or perhaps it’s possible to exclude a specific list of files, run with -j5 (or -j3), and then do those more challenging files with -j1?

laanwj commented at 5:55 pm on June 5, 2018: member

Sorry to say, but 1GB of RAM and 4 cores is not useful for compilation. In my experience reserving memory for VMs, 1-1.5GB per -jX parallelism works, more is pushing it, it can be done with lots of swap space but you won’t win time! For ARM I (still) strongly recommend to use cross-compilation.

Sjors commented at 6:23 pm on June 5, 2018: member

I’m looking at Armian builds, since I might as well just build the rest of the OS - and sync the chain - along with cross compiling bitcoind :-)

MarcoFalke deleted a comment on Jun 11, 2018

MarcoFalke commented at 7:21 pm on June 21, 2018: member

Considering that clang uses a lot less memory compared to gcc, why don’t we mention clang as an alternative in https://github.com/bitcoin/bitcoin/blob/000abbb6b07410357a928768d7d56465ba0d3bac/doc/build-unix.md#memory-requirements

Should I create a pull for this?

dufferzafar commented at 5:19 pm on September 1, 2018: none

@MarcoFalke:

why don’t we mention clang as an alternative @gmaxwell said above:

Rather than recommending clang (which would send more hosts into a less tested and potentially consensus inconsistent config) we could recommend compiler flags that reduce memory usage.

laanwj commented at 5:41 pm on September 1, 2018: member

well if clang, a compiler many, many people use, generates consensus incompatible code, bitcoin has a huge problem.

(to be clear, no, I don’t think this is an issue worrying about here - let’s focus on finding configurations that reduce memory usage)

dufferzafar commented at 5:49 pm on September 1, 2018: none

To be honest, I myself use clang (along with ccache) but that’s only because all my work involves running in regtest, so I don’t have to worry about consensus compatibility.

laanwj commented at 5:51 pm on September 1, 2018: member

I mean, it might have been a good argument in 2012, but at this point, many platforms are clang-only (MacOSX, FreeBSD, OpenBSD, at least). There’s nothing that binds bitcoin to gcc specifically.

meeDamian commented at 3:43 pm on September 11, 2018: contributor

Just wanted to leave an update here, that building Bitcoin Core v0.16.2 is possible¹ on Raspberry Pi Zero W, which has 512MB RAM + default of 100MB SWAP.

0./autogen.sh
1./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" --enable-cxx --without-gui --disable-shared --with-pic
2--enable-upnp-default --disable-wallet --disable-tests
3make
4sudo make install

¹ It did take a very long time.

laanwj commented at 4:20 pm on September 11, 2018: member

@meeDamian hahaha nice, that’s quite a thing, building with less than a GB RAM+SWAP

practicalswift commented at 8:38 am on June 27, 2019: contributor

As noted in this issue we are currently using significantly more compile-time memory than we technically have to.

I plan to address this and you can see steps being taken in #16273 and #16278 (the latter is a subset of the former limited to tests only).

People who care about compile-time memory usage might be interested in reviewing those PR:s :-)

practicalswift commented at 9:39 am on June 27, 2019: contributor

These are the 25 worst offenders in terms of compile-time memory usage:

#	File	Max memory usage (RSS)
1	`test/test_test_bitcoin-script_tests.o`	1416 MB
2	`wallet/libbitcoin_wallet_a-wallet.o`	1102 MB
3	`wallet/libbitcoin_wallet_a-rpcwallet.o`	1059 MB
4	`libbitcoin_server_a-validation.o`	953 MB
5	`test/test_test_bitcoin-util_tests.o`	905 MB
6	`wallet/test/test_test_bitcoin-wallet_tests.o`	905 MB
7	`wallet/test/test_test_bitcoin-coinselector_tests.o`	897 MB
8	`libbitcoin_server_a-init.o`	892 MB
9	`qt/test/qt_test_test_bitcoin_qt-wallettests.o`	890 MB
10	`wallet/test/test_test_bitcoin-ismine_tests.o`	874 MB
11	`qt/test/qt_test_test_bitcoin_qt-addressbooktests.o`	863 MB
12	`libbitcoin_server_a-net_processing.o`	857 MB
13	`wallet/test/test_test_bitcoin-psbt_wallet_tests.o`	837 MB
14	`test/test_test_bitcoin-transaction_tests.o`	813 MB
15	`qt/qt_libbitcoinqt_a-sendcoinsdialog.o`	809 MB
16	`wallet/libbitcoin_wallet_a-rpcdump.o`	800 MB
17	`test/test_test_bitcoin-rpc_tests.o`	792 MB
18	`libbitcoin_server_a-validationinterface.o`	770 MB
19	`test/test_bitcoin`	752 MB
20	`rpc/libbitcoin_server_a-blockchain.o`	745 MB
21	`qt/qt_libbitcoinqt_a-rpcconsole.o`	741 MB
22	`test/test_test_bitcoin-sighash_tests.o`	739 MB
23	`rpc/libbitcoin_server_a-rawtransaction.o`	721 MB
24	`qt/qt_libbitcoinqt_a-walletmodel.o`	721 MB
25	`wallet/libbitcoin_wallet_a-walletdb.o`	720 MB
…	…	…
449	`support/libbitcoin_util_a-cleanse.o`	24 MB
450	`crypto/sha256_sse4.cpp`	24 MB
451	`compat/libbitcoin_util_a-glibc_sanity.o`	24 MB
452	`script/script_error.cpp`	23 MB
453	`script/libbitcoin_consensus_a-script_error.o`	23 MB
454	`compat/libbitcoin_util_a-strnlen.o`	18 MB

Measured using gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0.

practicalswift commented at 2:21 pm on June 29, 2019: contributor

People interested in enabling builds on low memory machines might be interested in reviewing and testing #16312:

tests: Reduce compile-time memory usage by 0.5 GB when compiling script_tests.cpp (from 1.4 GB to 0.9 GB)

practicalswift commented at 6:59 am on August 10, 2019: contributor

Chasing {concept,approach} ACK/NACKs for #16331 which tackles this issue.

Let me know if this issue is no longer relevant :-)

laanwj commented at 9:51 am on August 12, 2019: member

It doesn’t seem that proposed solutions to this attract much, or any attention. Most notably, no one building on 1GB menory systems seems to be helping testing.

So my conclusion would be it’s either not relevant anymore or very very low priority, and will only become less so as systems with so little menory become rarer.

Sorry for the wasted effort :/

On Sat, Aug 10, 2019, 09:00 practicalswift notifications@github.com wrote:

Chasing concept ACK or NACKs for #16331 https://github.com/bitcoin/bitcoin/pull/16331 which tackles this issue.

Let me know if this issue is no longer relevant :-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/6658?email_source=notifications&email_token=AAA65NSZXEEQVXG6PXRHVK3QDZRPLA5CNFSM4BPNRUDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4AHRHA#issuecomment-520124572, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA65NVIG2C2TOST5EDXBCTQDZRPLANCNFSM4BPNRUDA .

meeDamian commented at 10:02 am on August 12, 2019: contributor

I’m not sure if I’m not missing something here, but bitcoind was [already building on RBP zero][zero] (512MB ram with just 100MB swap). Plus, I’ve just built [v0.18.1] on RBP3 (1GB ram, no swap) a few days ago without problems even before applying any PRs. It does take its time, however 😅. So I for once, am not sure what else/how to test?

[zero]: #6658 (comment) [v0.18.1]: https://github.com/bitcoin/bitcoin/tree/v0.18.1

Sjors commented at 11:45 am on August 12, 2019: member

@meeDamian does the new PR work without those extra config flags?

MarcoFalke commented at 12:59 pm on August 12, 2019: member

It is a bit concerning that the unit test have to be disabled to compile on low memory machines

meeDamian commented at 1:19 pm on August 13, 2019: contributor

On a fresh Raspbian on RBP3, running (with or without --enable-lowmem flag):

 0sudo swapoff --all
 1
 2sudo apt install git build-essential libtool autotools-dev automake pkg-config bsdmainutils python3 libssl-dev libevent-dev libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-test-dev libboost-thread-dev libminiupnpc-dev libzmq3-dev
 3
 4git clone  -b fix-absurd-memory-usage-when-compiling-script_build  https://github.com/practicalswift/bitcoin.git bitcoin-swift
 5cd bitcoin-swift/
 6
 7./contrib/install_db4.sh $(pwd)
 8export BDB_PREFIX=$(pwd)/db4
 9./autogen.sh
10
11./configure BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include" --enable-lowmem --without-gui

Crashes with:

 0…
 1checking for upnpDiscover in -lminiupnpc... yes
 2checking miniupnpc/miniupnpc.h usability... yes
 3checking miniupnpc/miniupnpc.h presence... yes
 4checking for miniupnpc/miniupnpc.h... yes
 5checking for upnpDiscover in -lminiupnpc... (cached) yes
 6checking miniupnpc/upnpcommands.h usability... yes
 7checking miniupnpc/upnpcommands.h presence... yes
 8checking for miniupnpc/upnpcommands.h... yes
 9checking for upnpDiscover in -lminiupnpc... (cached) yes
10checking miniupnpc/upnperrors.h usability... yes
11checking miniupnpc/upnperrors.h presence... yes
12checking for miniupnpc/upnperrors.h... yes
13checking for upnpDiscover in -lminiupnpc... (cached) yes
14checking whether to build Bitcoin Core GUI... no (Qt5)
15checking for boostlib >= 1.47.0 (104700)... yes
16checking whether the Boost::System library is available... yes
17configure: error: Could not find a version of the library!

Running the same on bitcoin v0.18.1 does not crash.

meeDamian commented at 1:24 pm on August 13, 2019: contributor

Actually… bitcoin master also crashes the same way, so likely unrelated to this. Happy to try again if someone advises how to fix/proceed.

Sjors commented at 3:24 pm on August 13, 2019: member

@meeDamian did you try with practicalswift/low-mem? #16331

meeDamian commented at 4:54 pm on August 13, 2019: contributor

I’m afraid practicalswift/low-mem crashes the same way as master, and fix-absurd-memory-usage-when-compiling-script_build. Any idea what library version might be missing there, or how to fix it?

fanquake referenced this in commit c3b605cf13 on Aug 14, 2019

MarcoFalke commented at 1:27 pm on October 12, 2019: member

The docs are updated and mention that clang can be used, which uses less than 1 GB of memory. Closing for now

https://github.com/bitcoin/bitcoin/blob/561a7d30478b82f5d46dcf0f16e864a9608004f4/doc/build-unix.md#memory-requirements

MarcoFalke closed this on Oct 12, 2019

DrahtBot locked this on Dec 16, 2021

Building requires >1GB memory #6658