Seek more/different peers when ours all have too high feefilter

sipa commented at 2:17 pm on August 30, 2023: member

While investigating an issue with a non-propagating transaction with @TheBlueMatt, it occurred to us that it can be a problem for a node to get its transactions relayed when it’s surrounded solely by low-mempool-limit peers (which correspondingly have a high announced minfeefilter value). And it appears this is detectable too, as our own mempoolminfee would be lower.

While obviously relay can’t be guaranteed, this is a detectable situation, and it could factor into the outbound replacement logic and/or trigger something similar to the “seek more peers” behavior we exhibit when we appear to be not getting blocks.

sipa added the label Feature on Aug 30, 2023

maflcko added the label P2P on Aug 30, 2023

maflcko added the label Brainstorming on Aug 30, 2023

maflcko removed the label Brainstorming on Aug 30, 2023

TheBlueMatt commented at 3:17 pm on August 30, 2023: contributor

As nodes with limited mempools (presumably some of the RPi node things do this by default?) become more and more common, this is potentially gonna start to be an issue. Of course post-package-relay+mempool-sync (heh) its less of an issue, but lightning nodes really care about this kinda thing…

darosior commented at 8:21 am on September 4, 2023: member

What about discouraging outbound peers with a mempool -say- 80% smaller than ours? It seems useful in general, would solve this situation for transactions with a reasonable fee attached, and for transactions with a fee such as it would be under 60MB of unconfirmed transactions.. it’s the application’s responsibility to prevent this?

I understand for Lightning’s commitment transactions you rely on your peer to cosign a feerate update, but you would most likely have dropped to chain before getting in a situation where you are holding a commit tx whose feerate is lower than that of 60MB worth of unconfirmed transactions?

We should probably check what the node-in-a-box projects set as -maxmempool too.

ariard commented at 11:28 pm on September 8, 2023: contributor

From my understanding of the current state of #27742 (package relay code), m_fee_filter_received is not looked on during the composition of the ANCPKGINFO reply in ProcessGetData / MaybeGetAncPkgInfo, neither in the further processing of complementary missing package txn relay at GETPKGTXNS so effectively package relay deployment should be a good hardening for time-sensitive lightning transactions with a pre-signed feerate, assuming Lightning node / softwares responsibly RBFing / CPFPing their low-feerate txn as timelock expiration is approaching.

I think there is still a longer concern about a majority of low-mempool-limit peers (i.e low-value -maxmempool) screwing up the transaction-relay of low-feerate lightning transactions and therefore delaying the confirmation of those ones, or at least making not seen by the miners mempools and so _artificially fee-bumping them as timelocks expiration is approaching. So low-mempool-limit peers with a fee-filter not representing miners / default full-nodes mempool setting might be an individual source of failure for transaction-relay propagation and holistically a waste in fee-bumping reserves for Lightning nodes.

I think it might makes sense in the long-term to discourage (or preferentially evict them) tx-relay peers with relatively lower mempool size than ours. I guess it would be better to get back on the table #20726 and BIP338 (clean up of features flow negotiated with peers) before we complexify discourage / peers disconnection on local node settings.

petertodd commented at 7:47 am on September 18, 2023: contributor

As nodes with limited mempools (presumably some of the RPi node things do this by default?) become more and more common, this is potentially gonna start to be an issue. Of course post-package-relay+mempool-sync (heh) its less of an issue, but lightning nodes really care about this kinda thing…

Why would this become more common? I’m looking at Amazon right now, and the supermajority of Raspberry Pi’s for sale are 4GB or higher, with only a few 2GB models and just one 1GB model (in the first page of results). The default is 300MB of actual RAM usage, which is just 15% of a 2GB RPi’s available RAM. I’m also not aware of any RPi node software with a non-default max mempool.

I’ve you’ve actually observed a problem that appears to be this, I’d be more inclined to suspect a cause like a sybil attack of badly written chainalysis tx monitoring nodes.

ariard commented at 3:12 pm on September 18, 2023: contributor

The default is 300MB of actual RAM usage, which is just 15% of a 2GB RPi’s available RAM. I’m also not aware of any RPi node software with a non-default max mempool.

While a RPi can dedicate 15% of its available RAM to the mempool, there is no guarantee than actually users are not running with more constrained settings, e.g if they are sharing the platform with other process like lightning nodes or btcpay, consuming too RAM space.

petertodd commented at 4:16 pm on September 18, 2023: contributor

On Mon, Sep 18, 2023 at 08:12:19AM -0700, Antoine Riard wrote:

The default is 300MB of actual RAM usage, which is just 15% of a 2GB RPi’s available RAM. I’m also not aware of any RPi node software with a non-default max mempool.

While a RPi can dedicate 15% of its available RAM to the mempool, there is no guarantee than actually users are not running with more constrained settings, e.g if they are sharing the platform with other process like lightning nodes or btcpay, consuming too RAM space.

I fully recognize that there is no guarantee. But my point is, there isn’t that much motive. So I’d be surprised if there exists large numbers of genuine nodes with lower than default mempool settings. And I don’t see any reason for that situation to change.

mzumsande commented at 9:20 am on September 19, 2023: contributor

Maybe it would be good to collect some empirical data of how frequent peers with significantly higher minfeefilter actually are. If it can be a problem to find peers relaying our transactions, maybe a very simple first step would be to not keep up full outbound connections to peers that don’t support tx relay at all (e.g. because they might be running in -blocksonly mode).

Crypt-iQ commented at 9:21 pm on February 12, 2024: contributor

Maybe it would be good to collect some empirical data of how frequent peers with significantly higher minfeefilter actually are.

I made a network crawler for feefilter (sends version/verack and waits for feefilter) and for today this is the histogram of feefilters (sat/KvB) from ~16.8k nodes:

feefilter-0212-2

Right now, mempool.space says that the minimum fee to get into the mempool is at 4.35 sat/vbyte. There are ~7.3k nodes with minfeefilter set to >= 9.5 sat/vbyte. I’ve observed similar behavior over the past couple weeks where there’s a chunk of peers that have higher feefilters than the standard 300MB mempool.

carnhofdaki commented at 6:45 am on December 5, 2024: contributor

Today I noticed that the mempoolminfee stays higher than 1000 even when my node’s mempool usage is less than maxmempool. Is this expected?

 0chronos:~$ bitcoin-cli getmempoolinfo
 1{
 2  "loaded": true,
 3  "size": 88640,
 4  "bytes": 47693449,
 5  "usage": 262137696,
 6  "total_fee": 0.71013802,
 7  "maxmempool": 300000000,
 8  "mempoolminfee": 0.00001103,
 9  "minrelaytxfee": 0.00001000,
10  "incrementalrelayfee": 0.00001000,
11  "unbroadcastcount": 5973,
12  "fullrbf": true
13}
14chronos:~$ bitcoin-cli getblockcount
15873332
16chronos:~$ bitcoind -version
17Bitcoin Core version v28.0.0
18Copyright (C) 2009-2024 The Bitcoin Core developers
19
20Please contribute if you find Bitcoin Core useful. Visit
21<https://bitcoincore.org/> for further information about the software.
22The source code is available from <https://github.com/bitcoin/bitcoin>.
23
24This is experimental software.
25Distributed under the MIT software license, see the accompanying file COPYING
26or <https://opensource.org/licenses/MIT>

UPDATE: Everything is OK. I am back to 1000 sat/kvB when unbroadcastcount dropped (by rebroadcasting the mempool manually, but maybe it would happen by itself after some time). mempoolminfee is not there anymore now.

glozow commented at 12:22 pm on December 5, 2024: member

Today I noticed that the mempoolminfee stays higher than 1000 even when my node’s mempool usage is less than maxmempool. Is this expected?

It decays back down gradually. See https://github.com/bitcoin/bitcoin/blob/e8cc790fe2a223cfaa6dedfba88a6dc4f336f1d9/src/txmempool.cpp#L1126-L1148

Seek more/different peers when ours all have too high feefilter #28371