RPC breakage with v28.0

fanquake commented at 12:15 pm on October 6, 2024: member

If you’re experiencing issues with Bitcoin Core 28.x, and using RPC, please report them here.

https://x.com/ketominer/status/1842592989488771531

fanquake added the label RPC/REST/ZMQ on Oct 6, 2024

sipa commented at 12:27 pm on October 6, 2024: member

Known issues:

Some software sends the JSON-RPC version as a number ("jsonrpc": 1.0) instead of as a string ("jsonrpc": "1.0"), which fails the sanity check introduced with JSON-RPC 2.0 compatibility (introduced in #27101).
The "warnings" field returned by some RPCs (getblockchaininfo, getmininginfo, and getnetworkinfo) is now an array instead of a string. The old behavior can be re-enabled by running with -deprecatedrpc=warning (introduced in #29845).

sipa commented at 12:30 pm on October 6, 2024: member

LND up to 0.18.3 seems to be incompatible with Bitcoin Core 28.0 due to the "warnings" field change: https://github.com/lightningnetwork/lnd/issues/9053, though using -deprecatedrpc=warnings should be enough to fix this.

sipa commented at 12:34 pm on October 6, 2024: member

The bitcoin-rpc NPM package seems to be incompatible with Bitcoin Core 28.0 due to sending an incorrect version type: https://github.com/vansergen/rpc-bitcoin/pull/65.

pinheadmz commented at 12:34 pm on October 6, 2024: member

The value for “version” was never checked and still is not. The value for “jsonrpc” since #27101 MUST be a string. If that string is “1.0” then the old JSONRPC behavior is followed (since some old documentation incorrectly recommended this). If that string is “2.0” then the new JSONRPC behavior is followed.

Sending any other string or value type for “jsonrpc” returns a parsing error.

sipa commented at 12:36 pm on October 6, 2024: member

@pinheadmz Thanks, fixed.

dr-orlovsky commented at 1:19 pm on October 6, 2024: none

I do experience.

Basically mempool electrs v3.0.0 in combination with Bitcoin Core v28.0 kill each other once Bitcoin core stops IBS and mempool load

ketominer commented at 1:26 pm on October 6, 2024: none

The bitcoin-rpc NPM package seems to be incompatible with Bitcoin Core 28.0 due to sending an incorrect version type: vansergen/rpc-bitcoin#65.

Which FWIW in turn affects Dojo which uses this library. Dojo Coder is aware of the issue and working on changing the RPC client library or fixing rpc-bitcoin but it seems pretty much unmaintained.

sipa commented at 1:28 pm on October 6, 2024: member

@dr-orlovsky Does running with -deprecatedrpc=warnings fix your issue?

epiccurious commented at 2:04 pm on October 6, 2024: contributor

Saw someone’s tweet about having to increase rpcworkqueue.

https://x.com/_behodler/status/1842657622018683198

dr-orlovsky commented at 5:55 pm on October 6, 2024: none

~~@sipa yes, it fixes!~~ However this arg is undocumented. Also, I believe that without it Bitcoin Core should not crash but just disconnect the client.

~~Anyway, I have a solution for the urgent problem, the rest waits. Thank you!~~

UPD: Actually it doens’t fix. Bitcoin Core still crashes right after loading the mempool txes - although Electrs doesn’t crashes anymore.

Bitcoin Core on the left, Mempool Electrs - on the right

Screenshot from 2024-10-06 19-58-52

sipa commented at 6:00 pm on October 6, 2024: member

@dr-orlovsky Are you sure it crashes? That sounds unlikely, based on the changes in this release. Does it segfault? What appears in debug.log?

achow101 commented at 6:01 pm on October 6, 2024: member

Basically mempool electrs v3.0.0 in combination with Bitcoin Core v28.0 kill each other once Bitcoin core stops IBS and mempool load

I’m not observing any RPC issues yet with https://github.com/mempool/electrs/commit/055aba1e8d0d8f4cf7080014c89e3b3055e3d0b7, except for it not handling xor’d blocks, but that’s a different problem.

However this arg is undocumented.

It’s in the help for getblockchaininfo, and also in the release notes.

Bitcoin Core still crashes right after loading the mempool txes

That sounds like it’s independent of electrs, and of the RPC issue. Can you open a new issue for that and post the debug.log?

dr-orlovsky commented at 6:03 pm on October 6, 2024: none

I just attached a screenshot to my previous comment: systemd restarts the node, and reports its crash:

 0Oct 06 17:54:18 core bitcoind[16532]: 2024-10-06T17:54:18Z initload thread exit
 1Oct 06 17:55:43 core systemd[1]: bitcoind.service: A process of this unit has been killed by the OOM killer.
 2░░ Subject: A process of bitcoind.service unit has been killed by the OOM killer.
 3░░ Defined-By: systemd
 4░░ Support: https://www.debian.org/support
 5░░ 
 6░░ A process of unit [@UNIT](/bitcoin-bitcoin/contributor/unit/) has been killed by the Linux kernel out-of-memory (OOM)
 7░░ killer logic. This usually indicates that the system is low on memory and that
 8░░ memory needed to be freed. A process associated with bitcoind.service has been determined
 9░░ as the best process to terminate and has been forcibly terminated by the
10░░ kernel.
11░░ 
12░░ Note that the memory pressure might or might not have been caused by bitcoind.service.
13Oct 06 17:55:43 core systemd[1]: bitcoind.service: Main process exited, code=killed, status=9/KILL
14░░ Subject: Unit process exited
15░░ Defined-By: systemd
16░░ Support: https://www.debian.org/support
17░░ 
18░░ An ExecStart= process belonging to unit bitcoind.service has exited.
19░░ 
20░░ The process' exit code is 'killed' and its exit status is 9.
21Oct 06 17:55:43 core systemd[1]: bitcoind.service: Failed with result 'oom-kill'.
22░░ Subject: Unit failed
23░░ Defined-By: systemd
24░░ Support: https://www.debian.org/support
25░░ 
26░░ The unit bitcoind.service has entered the 'failed' state with result 'oom-kill'.
27Oct 06 17:55:43 core systemd[1]: bitcoind.service: Consumed 3min 18.749s CPU time.
28░░ Subject: Resources consumed by unit runtime
29░░ Defined-By: systemd
30░░ Support: https://www.debian.org/support
31░░ 
32░░ The unit bitcoind.service completed and consumed the indicated resources.
33Oct 06 17:55:43 core systemd[1]: bitcoind.service: Scheduled restart job, restart counter is at 1.
34░░ Subject: Automatic restarting of a unit has been scheduled
35░░ Defined-By: systemd
36░░ Support: https://www.debian.org/support
37░░ 
38░░ Automatic restarting of the unit bitcoind.service has been scheduled, as the result for

Seems bitcoin core tries to allocate a lot of memory invalidly reading RPC data. The machine I use has 4GB and there is nothing other than Bitcoin Core on it. Also, if Electrs doesn’t connect, Bitcoin Core runs indefinitely and the crash doesn’t happen

achow101 commented at 6:10 pm on October 6, 2024: member

Seems bitcoin core tries to allocate a lot of memory invalidly reading RPC data.

Do you know which RPC it is using that’s causing the memory allocation? If you turn on debug=rpc, it will be logged in the debug.log.

This sounds like it could be related to #24542 but that should’ve been fixed by #30094 which is in this release.

Were previous releases working on this machine?

dr-orlovsky commented at 6:15 pm on October 6, 2024: none

@achow101 created issue as requested: #31041

Will add RPC info with debug=rpc there

maflcko commented at 8:20 am on October 7, 2024: member

I’m not observing any RPC issues yet with mempool/electrs@055aba1, except for it not handling xor’d blocks, but that’s a different problem.

Just for reference: https://github.com/mempool/electrs/pull/101

fanquake added this to the milestone 28.1 on Oct 22, 2024

fanquake commented at 4:25 pm on October 22, 2024: member

Going to leave this open a little longer to see if anything else comes up that might warrant a backport or other action.

fanquake commented at 4:28 pm on October 22, 2024: member

Possibly relevant: https://github.com/mempool/electrs/issues/102.

RPC breakage with v28.0 #31039