Crash upon RPC v1 connection in v28.0.0 #31041

Is there an existing issue for this?

• I have searched the existing issues

Current behaviour

On RPC v1 connection, after the block sync and mempool read is complete, a crash happens (see logs). I used mempool fork of electrs indexer which did an RPC request.

Original report can be found here: #31039 (comment)

Expected behaviour

No crash is expected

Steps to reproduce

Run Bitcoin Core (I used machine with 8 cores, and 4GB or 8GB of memory)

Relevant log output

Oct 06 17:54:18 core bitcoind[16532]: 2024-10-06T17:54:18Z initload thread exit
Oct 06 17:55:43 core systemd[1]: bitcoind.service: A process of this unit has been killed by the OOM killer.
░░ Subject: A process of bitcoind.service unit has been killed by the OOM killer.
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A process of unit [@UNIT](/bitcoin-bitcoin/contributor/unit/) has been killed by the Linux kernel out-of-memory (OOM)
░░ killer logic. This usually indicates that the system is low on memory and that
░░ memory needed to be freed. A process associated with bitcoind.service has been determined
░░ as the best process to terminate and has been forcibly terminated by the
░░ kernel.
░░ 
░░ Note that the memory pressure might or might not have been caused by bitcoind.service.
Oct 06 17:55:43 core systemd[1]: bitcoind.service: Main process exited, code=killed, status=9/KILL
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ An ExecStart= process belonging to unit bitcoind.service has exited.
░░ 
░░ The process' exit code is 'killed' and its exit status is 9.
Oct 06 17:55:43 core systemd[1]: bitcoind.service: Failed with result 'oom-kill'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit bitcoind.service has entered the 'failed' state with result 'oom-kill'.
Oct 06 17:55:43 core systemd[1]: bitcoind.service: Consumed 3min 18.749s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit bitcoind.service completed and consumed the indicated resources.
Oct 06 17:55:43 core systemd[1]: bitcoind.service: Scheduled restart job, restart counter is at 1.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ Automatic restarting of the unit bitcoind.service has been scheduled, as the result for

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v28.0.0

Operating system and version

Debian Bookworm

Machine specifications

8 CPUs; 4GB RAM

With the increase in memory to 8GB the crash has disappeared.

Steps to reproduce

Run Bitcoin Core

What are the exact steps to reproduce? What is the config? Which RPCs were called, in what order, and in what timing (parallel, sequential), ...?

Does it still happen after https://github.com/romanz/electrs/pull/1091 ?

Does it still happen after https://github.com/romanz/electrs/pull/1091 ?

That is for a different fork of electrs. For the mempool/electrs, you can see in the original logs in #31039 that it is already waiting for mempool to load before syncing.

I see. It would be good to know what the batch request is looking like. Also, it would be good to confirm that this works on 27.x and is broken on 28.x. Also, it would be good to confirm that the system has enough memory to fit the whole batch request response (as json) into memory.

I synced mempool's fork of electrs using bitcoin core v28.0 and there were no issues with unconstrained memory. (also wow; I did not notice in the readme this requires > 1.3TB of disk space to sync)

Memory usage as I measured peaked at about 1.9GB for bitcoind, compared with 6.1GB for electrs. This machine has 64GB RAM so total utilization was low. electrs did crash and require me to increase the max file descriptor count (with ulimit -n) which I wasn't expecting, but after that everything worked smoothly.

My bitcoind was already running (and synced) with default mempool size when I began the electrs sync. Perhaps this is the key difference from your methodology @dr-orlovsky ? Otherwise, I am not sure why mine behaved so differently...

I could try with systemd's MemoryMax=bytes or a docker container, to force-limit maximum memory of bitcoind, perhaps?

Memory usage as I measured peaked at about 1.9GB for bitcoind, compared with 6.1GB for electrs.

I'd say that explains the crash happening with 4 GB of RAM, but not with 8GB of RAM (albeit that seems a bit lucky).

Not sure what to do here. I don't think there is anything that can be done to catch an (expected) OOM?

Oh yes, it could do. I had interpreted OP to mean bitcoin core had 4GB (or 8GB) of RAM to itself, but if both of those were trying to share that amount, it seems like it might not work. I note that over in #31039 it was reported that:

The machine I use has 4GB and there is nothing other than Bitcoin Core on it.

I got suspicious of my own earlier results as I've seen bitcoin core using more memory than this previously, so re-ran an electrs sync to block 600,000 only this time. bitcoind had peak memory usage of ~5GB by block 420,000, which then held steady for the remainder of the sync to 600,000.

$  grep VmPeak /proc/(pidof bitcoind)/status
VmPeak:  5039796 kB

I agree that there's not much we can do about catching an OOM. We also don't have any confirmation that this same issue was not present on 27.x. I will try and test this later to see if there is any difference to block 600k again.

We also don't have any confirmation that this same issue was not present on 27.x. I will try and test this later to see if there is any difference to block 600k again.

By block 400,000 bitcoin core v27.0 peak usage hit VmPeak: 5048020 kB, which is approx the same (marginally higher than v28.0) at this stage where it too roughly levels out.

I'm not going to run this further than the block 550k I am up to now, as these results seem so similar. My read is that:

• Electrs takes 50% more ram than bitcoin core (seen max of 8GB)
• Bitcoin Core v27.0 and v28.0 are approximately equal in requiring ~5GB RAM to host an electrs sync (to block 600k)
• Running these both on a system with only 4GB (maybe even 8GB too) should be expected to cause OOM killing or heavy swapping

Thanks for taking a detailed look. If you don't mind, you could run it again through KDE heaptrack to see if there are any obvious low-hanging fruits for improvement of the memory usage.

In any case, I think it is clear that this is not a regression and I've removed the 28.x milestone.

I took a look with heaptrack on a debug build of 28.0 (output file: https://tmp.256k1.dev/heaptrack.bitcoind.3546996.zst )

Whilst number of allocations increase over time as expected:

Consumed heap memory stays pretty stable:

Peak RSS of 6.8GB (this includes heaptrack though now) seems in line with previous checks.

I would expect this to increase a bit more as we go through the chain further through block 870k, but there does not appear to be any obvious memory leaks here.

I may try an assumeutxo sync from block 800,000 with a no-background-sync patch just to excercise those later blocks. But can't guarantee I'll find the time.

Thanks! I don't expect a memory leak either, but I'd find it interesting to see where the heap usage came from. I guess it is all Univalue?

Actually seems to be a mixture, and UniValue doesn't particularly stand out to me: