Self-compiled bitcoind 29.0 much slower than self-compiled 28.0 on my system

hMsats commented at 6:08 am on May 9, 2025: none

I always compile the bitcoin software myself (Ubuntu 24.04.2 LTS). For bitcoin-29.0 I compiled it (of course) for the first time with cmake (with and without berkeley-db).

For both (with and without berkeley-db), bitcoind-29.0 is much slower than bitcoin-28.0 which gives me problems on my server.

To investigate I wrote a shell script that determines the time difference in seconds between “Saw new header” and “UpdateTip”:

02025-05-09T05:24:11Z Saw new header hash=0000000000000000000213e630619be8945d471d06b0395fb6adca797877527d height=895920
12025-05-09T05:24:11Z Saw new cmpctblock header hash=0000000000000000000213e630619be8945d471d06b0395fb6adca797877527d peer=5197
22025-05-09T05:24:12Z UpdateTip: new best=0000000000000000000213e630619be8945d471d06b0395fb6adca797877527d height=895920 version=0x2a4aa000 log2_work=95.598602 tx=1188522869 date='2025-05-09T05:23:59Z' progress=1.000000 cache=216.6MiB(1587712txo)

which in this bitcoind-28.0 example is only 1 second. This is typical for bitcoind-28.0 on my system where it is between 0 and 1 seconds with only very occasionally a bit longer. Now for bitcoind-29.0 it’s much longer:

Block nr: delta t in seconds 895579: 3 895580: 3 895581: 4 895582: 2 895583: 16 895584: 65 895585: 8 895586: 280 895587: 81 895588: 133 895589: 124 895590: 3 895591: 73 895592: 153 895593: 284 895594: 528 895595: 17 895596: 8 895597: 4 895598: 2 895599: 3 895600: 4 895601: 6 895602: 3 895603: 2 895604: 5 895605: 5 895606: 34 895607: 2 895608: 6

This is true whether it’s compiled with or without berkeley-db. So mometimes it takes several minutes!

So I did some investigations. The first thing I found was that my bitcoind-29.0 and others are much bigger than the pre-compiled ones:

Precompiled 29.0 (du in MB): 3 bitcoin-cli 16 bitcoind 42 bitcoin-qt 5 bitcoin-tx 3 bitcoin-util 10 bitcoin-wallet 28 test_bitcoin

For my self-compiled 29.0: 21 bitcoin-cli 273 bitcoind 47 bitcoin-tx 21 bitcoin-util 134 bitcoin-wallet 511 test_bitcoin

Although normally I use some compiler options, here for comparison compiled according to the docs: (cmake -B build; cmake --build build; cmake --install build)

Both (my and pre-compiled) test_bitcoin work fine. Note that also my (not too slow) bitcoind-28.0 (etc) is about the same too big a size (269 MB) as my bitcoind-29.0. So I had this before but never noticed.

So my first question before I investigate any further is: how come my bitcoind is so big compared to the pre-compiled one?

fanquake commented at 6:41 am on May 9, 2025: member

So my first question before I investigate any further is: how come my bitcoind is so big compared to the pre-compiled one?

This will be the debug symbols. If you run ‘strip’ on the binaries. i.e ‘strip bitcoind’ this debug information will be removed, and the size of your binaries should shrink 80-90%.

hMsats commented at 7:38 am on May 9, 2025: none

@fanquake Thanks a lot! I expected something like this (debug info) but thought it was some compiler option. The strip worked and now my bitcoind is only 12 MB 👍. As I never used strip on bitcoin-28.0 I don’t expect this to make a difference but will try it first on my server anyway or could it?

This is very confusing. I’m a long time bitcoin node runner and linux user but would never have thought of this possibility. Shouldn’t this be mentioned in the doc/build-unix.md?

maflcko commented at 8:15 am on May 9, 2025: member

What are your settings?

Can you share the debug log?

Can you run with the debug categories bench, blockstorage, lock, prune, validation, and possibly rpc, if there is a background caller?

maflcko added the label Resource usage on May 9, 2025

maflcko added the label Questions and Help on May 9, 2025

hMsats commented at 8:25 am on May 9, 2025: none

@maflcko I will but it will take some time as I’m on vacation right now but will investigate when I have time.

Created bitcoind via:

cmake -B build -DWITH_ZMQ=ON -DBUILD_GUI=ON

Settings (same as for 28.0):

0txindex=1
1rpcthreads=4
2rpcworkqueue=64
3dbcache=2500
4maxconnections=100
5permitbaremultisig=0
6datacarrier=0

sipa commented at 1:31 pm on May 9, 2025: member

Unsure if this matters in your case, but the cache being warm or not impacts validation speed significantly. If you have a node that is running for a long time, and then stop and start it again, the speed will be slower for a while until the cache warms up again.

hMsats commented at 5:16 am on May 10, 2025: none

@sipa these were only the last 30 blocks. In total I let bitcoin-29.0 run for 56 blocks but the performance is so bad compared to when I restart bitcoin-28.0 that something else must be going on. Maybe the debug.log gives you more information?

hMsats commented at 5:28 am on May 10, 2025: none

@maflcko I found time to run 29.0 with the following debug catagories:

0debug=bench
1debug=blockstorage
2debug=cmpctblock
3debug=tor
4debug=validation
5debug=rpc
6debug=zmq

You can find it here and more system information on my website.

There are background callers. Besides a full node I run: Fulcrum (public electrum server), 2 CLN lightning nodes, electrum personal server. It’s maybe a lot but 28.0 has no problems with it. At the beginning of my debug test I started these as well.

hMsats commented at 5:30 pm on May 11, 2025: none

Although the jury is still out, it might be related to what @sipa suggested: that the cache still hasn’t warmed up enough and validation speed is temporarily slow. I’m testing different options like running with the pre-compiled bitcoind but every test takes quite a long time to come to a clear conclusion (days).

At least I got an answer to my questions of why my executables are so big. Closing for now, will reopen if this issue persists.

hMsats closed this on May 11, 2025

hMsats commented at 10:18 pm on May 21, 2025: none

@sipa ~~After a long and systematic research of my problem, I was probably able to find the cause:~~

~~In validation.cpp else if (!check()) was changed into else if (auto result = check(); result.has_value()). Shouldn’t that be else if (auto result = check(); !result.has_value())?~~

hMsats reopened this on May 21, 2025

achow101 commented at 10:29 pm on May 21, 2025: member

In validation.cpp else if (!check()) was changed into else if (auto result = check(); result.has_value()). Shouldn’t that be else if (auto result = check(); !result.has_value())?

Why do you think that’s the error?

If this were incorrect, you wouldn’t be seeing that validation be slow; it would instead be failing on a bunch of blocks.

The code on that line is correct. It was changed from a bool in 28.x to a std::optional<std::pair<ScriptError, std::string>> in 29.x. If the return value has a value, then that means an error occurred and the value contained is the error. On success, it returns no value.

hMsats commented at 10:35 pm on May 21, 2025: none

@achow101 somewhere near that commit I think my server becomes slow. I didn’t fully understand the code, so sorry. I will investigate further until I do find a problematic commit. Thanks!

hMsats commented at 10:45 pm on May 21, 2025: none

For now I think something bad happens between commit:

52fd1511a774d1ff9e747b2ce88aa1fcb778ced8

and

0a159f0914775756dcac2d9fa7fe4e4d4e70ba0c

I’ll try to narrow it down.

Self-compiled bitcoind 29.0 much slower than self-compiled 28.0 on my system #32455