Not a lot of headroom for building with 512 MB RAM #7471

I tried out a few options for compiling on low memory systems and collected data on build time and memory usage.

The takeaways first (but please review the data and provide comment):

1. Issue #6658 from September identified the memory requirements of the build as a problem. The concern was around 1 GB, but this data shows that this can be comfortably mitigated with gcc options or by using clang. This also holds for systems with 512 MB memory plus 512 MB swap. Given this, issue 6658 can be closed.

2. Building with 512 MB and no swap (think Raspberry Pi 1 and Zero) is possible, but there isn't a lot of headroom. If the build becomes more memory hungry, it will push this category of users to a different option for getting binaries. Since this is future-looking, this may not warrant this issue remaining open, but it is worth discussing and monitoring going forward.

3. The '--disable-tests' option doesn't reduce memory use.

Trending Importance (Sept. 2015 to Feb. 2016)

The recent new features for pruning, mempool limiting and libsecp256k1 have made bitcoind on low-end hardware more practical. Also very inexpensive single-board computers with 512 MB RAM are becoming widely available, so we can anticipate the number of node-running systems in this category to be growing.

It is desirable to preserve compile-from-source as an option running node with a decent security/convenience trade-off.

Test Configurations

512 MB to 1 GB is assumed as the window of interest. The three system configurations tested are:

1. A Raspberry Pi 1 Model B - 512MB RAM, no disk swap, single-core 700Mhz ARM, Class 10 SD card for the filesystem, Raspbian Jessie Lite, running headless. gcc v4.9.2, clang 3.5.0-10+rpi1

2. A Raspberry Pi 2 Model B - 1GB RAM, no disk swap, quad-core 900Mhz ARM, Class 10 SD card for the filesystem, Raspbian Jessie, running an idle desktop environment. gcc version 4.9.2, clang version 3.5.0-10+rpi1

3. A VirtualBox VM, 512MB RAM, 575MB swap, assigned one core of a AMD 1650Mhz E-450 CPU, virtual drive backed by 5400 RPM 3.5" desktop SATA drive for swap and filesystem, Debian Jessie amd64, running headless. gcc version 4.9.2, clang version 3.5.0-10

Test Method

• All builds are on the branch v0.12.0rc2
• All building is done with a single process to use a single core.
• The GUI wallet is not part of the build
• The build is with the stock distro libdb-dev and libdb++-dev, which requires --with-incompatible-bdb flag for './configure'
• The elapsed time for 'make' is measured with the console 'time' utility, for real time, user CPU time and system CPU time.
• Fields marked FAIL are for when the build was not successful under the parameters of the trial.

The measurements for memory used is obtained by polling the /proc file for the complier pid and recording the VmPeak value like so:

$ for((i=0;;++i)) { echo $i ` grep VmPeak /proc/\`pidof cc1plus\`/status
| grep -o '[0-9]*'`; sleep 1 || break; } > VmPeak.txt

For clang, 'clang' is substituted for 'cc1plus':

$ for((i=0;;++i)) { echo $i ` grep VmPeak /proc/\`pidof clang\`/status
| grep -o '[0-9]*'`; sleep 1 || break; } > VmPeak.txt

The highest value is then extracted like so:

$ cat VmPeak.txt |awk '{print $2}' |sort -g |uniq -c |tail -n 1

Trial 1.A - GCC without tuning

$ ./configure --with-incompatible-bdb

Pi 1 died with error:

 virtual memory exhausted: Cannot allocate memory

Pi 2 and the VM died with error:

 g++: internal compiler error: Killed (program cc1plus)

This is Matt's original 6658 error and it happened as memory usage approached 100%.

Trial 1.B - GCC without tuning, tests disabled

$ ./configure --with-incompatible-bdb --disable-tests

Same result as 1.A

Pi 1 died with error:

 virtual memory exhausted: Cannot allocate memory

Pi 2 and the VM died with error

 g++: internal compiler error: Killed (program cc1plus)

Trial 2.A - Clang

$ ./configure --with-incompatible-bdb CC=clang CXX=clang++

Pi 1 died with error:

clang: error: unable to execute command: Segmentation fault

This was when memory use was at 100%.

Trial 2.B - Clang, tests disabled

$ ./configure --with-incompatible-bdb --disable-tests CC=clang CXX=clang++

Same result as 2.A

Pi 1 died with error:

clang: error: unable to execute command: Segmentation fault

Trial 3.A - GCC tuned with recommended options

$ ./configure --with-incompatible-bdb CXXFLAGS=" --param ggc-min-expand=1 --param ggc-min-heapsize=32768"

Trial 3.B - GCC tuned with recommended options, tests disabled

$ ./configure --with-incompatible-bdb --disable-tests CXXFLAGS=" --param ggc-min-expand=1 --param ggc-min-heapsize=32768"

Trial 4 - GCC tuned with aggressive ggc-min-heapsize

$ ./configure --with-incompatible-bdb CXXFLAGS=" --param ggc-min-expand=1 --param ggc-min-heapsize=4096"

Trial 5 - GCC tuned with aggressive ggc-min-expand

$ ./configure --with-incompatible-bdb CXXFLAGS=" --param ggc-min-expand=0 --param ggc-min-heapsize=32768"

This also makes the compiler EXTREMELY slow. I let it run for a day and it barely got anywhere - probably less than 10% of the way through. It will probably complete eventually, this is not a practical option.

Trial 6 - GCC tuned with aggressive ggc-min-expand and ggc-min-heapsize

$ ./configure --with-incompatible-bdb CXXFLAGS=" --param ggc-min-expand=0 --param ggc-min-heapsize=4096"

This also runs at a milli-crawl. It also ran for a day before I killed it and it made less progress than above.

Your thoughts?

Thanks for the testing. C++ is memory hungry, absolutely. I don't recommend building on 512MB at all. There was always a recommendation to use at least 1GB but it appears it got lost in f9298cc60e093533ce109aedd7d54d59e87865cd. It may be worded ackwardsly but the gcc tweaking is already necessary when compiling with 1 GB of memory. 512MB is no-chance territory.

As I've mentioned repeatedly the best solution, which any embedded developer can tell you, against this frustration is to cross compile from say, an Ubuntu VM on a fast PC. For ARM:

sudo apt-get install g++-arm-linux-gnueabi # install toolchain

cd depends
make -j10 HOST=arm-unknown-linux-gnueabihf NO_QT=1 # NO_WALLET=1 NO_UPNP=1 ...
cd ..
./configure --prefix=$PWD/depends/arm-unknown-linux-gnueabihf  --enable-glibc-back-compat --enable-reduce-exports LDFLAGS=-static-libstdc++
make -j10
scp src/bitcoind src/bitcoin-cli src/bitcoin-tx src/test/test_bitcoin $TARGET:

And voila, you have a src/bitcoind and src/bitcoin-cli etc to copy to the target.

Do you know of any plans to release signatured arm binaries through bitcoin.org? Stuff like ROKOS Core OS is great, but it is not ideally secure. I don't have a market study, but I am doubtful that the average Pi user is also a native, fluent Linux user on their main machine.

No plans, though if someone were to add armhf executable build to the gitian build system I'd certainly not be opposed.

Thanks for looking into this. Is this something to rework f9298cc?

I am not tied to the wording in the docs, so don't avoid changing it on my account. It felt more proper to at least capture the data and record it, so we have a reference point down the road.

The overall Pi situation is a bit of a newbie trap, but I agree the ultimate thing to do in-repo would be to get armhf into the gitian system. Out-of-repo, I am going to take a stab at putting up a better guide than the ones I found along the way.

BTW for anyone working on documentation this warrants mentioning: RPi1 has always been too weak to run a full node (not only to compile it). It may work but it is really non-optimal, balancing on the edge of crashing all the time. RPi2 with 4 cores and at least 1GB of memory is able to handle it better.

I am working on getting the data on that too ;)

Right now, my Pi 1 is syncing at the rate of a couple blocks a minute in the mid-200ks with 287 MB resident memory. The CPU has been pegged at 100% since starting, and it hasn't crashed yet. I do have a proper power supply that can keep up with the sustained current draw, though.

It probably needs some options set to work as a synced node, but I am optimistic so far. The Pi 2 is definitely getting it done.

@jarret did you get your Pi 1 to catch up with the blockchain? Glad to see the Pi 1 is still able to run a full node - with Qt? I am also in favour of being able to compile and run bitcoin-qt on a Pi 1 (and zero). Thanks to @luke-jr for the CCFLAGS suggestion in #6658 that seems to solve it - and yes, let's get these added to either the documentation, or automatically set if it's detected that these flags will be needed.

@rebroad Not yet. My Pi 1 is still slowly grinding away in the low 300ks. In 24 hours, it is making much more than 144 blocks progress, so it should catch up eventually, however the ETA is still probably a month or two or three out. One can always cheat and copy in the block database from another node, but it is a bit trickier if you are using the prune=xxx setting to fit on the SD card. I also have a Pi 2 syncing with the same settings, so I will probably end up cheating when it finishes.

I have not done Qt with the Pi 1 or 2. The problem is that the heavy SD card accesses for the blockchain stutters the whole system for a couple seconds, so using the desktop at the same time is frustrating. It is near-impossible on a Pi 1 with the 100% single-CPU consumption factor on top of that. A dedicated, headless pi node appears to be the way to go. After learning the bitcoin-cli commands, it is much more interesting and informative to monitor on the console that way.

The flag suggestions were added to the master branch build-unix.md in f9298cc60e093533ce109aedd7d54d59e87865cd, which didn't make it in time for the 0.12 release

@laanwj Did you mean the package g++-arm-linux-gnueabi or g++-arm-linux-gnueabihf?

and how does one cross-compile the bitcoin-qt also please?

Also, when I run bitcoind, I get the following error: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.21' not found

Did you mean the package g++-arm-linux-gnueabi or g++-arm-linux-gnueabihf?

Depends on the system. hf for most modern ARM processors, which have hardware floating point units. Which are probably all that run bitcoind in the first place.

Also, when I run bitcoind, I get the following error: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.21' not found

Using the same configure options as gitian may solve that:

./configure --enable-glibc-back-compat --enable-reduce-exports LDFLAGS=-static-libstdc++

Closing in favor of #6658 , I think one "compilation uses a lot of memory" open issue is enough.