I2P: Limit transient addresses

zzzi2p commented at 6:16 pm on December 26, 2022: none

The per-connection transient address feature in 24.0.1 is, we are pretty sure, putting a large load on the I2P network. Starting on Dec. 19 the number of tunnels in the network started to increase, and as measured at one router, it peaked at about 3x normal levels on Dec. 26. While the load is manageable for now, if the transient feature becomes popular, it has the potential to get much much worse.

I2P isn’t really designed to work that way, and some limit on the number of addresses (aka destinations or tunnels) built by the bitcoin client is necessary. I2P Destinations are designed to be long-lived, it’s not like in Tor where you can create tons of circuits. Waiting for tunnels to be built for each connection also adds a huge amount of delay to connection setup time.

A high number of addresses will also result in excessive CPU and bandwidth usage by your router to build the tunnels for each. Additionally, other routers will reject your excessive tunnel building, which will increase your resource usage even more as your router retries. And, of course, the overhead applies to every connection attempt, not just every successful connection.

Depending on your security goals, and the max number of i2p connections you wish to support, there’s a few options for you to consider:

Use a single transient address, created at startup
Hard limit the number of i2p connections if configured for transient
Group several connections into one of several addresses in a pool
Rate limit new connections or creation of new addresses

The i2pd router does not currently limit the number of local destinations, I don’t think. The Java router limits to 100 by default and will reject addresses over that limit.

If you do choose to continue using multiple addresses, we recommend a reasonable maximum number of addresses of around 16 or so, together with a rate limit on how frequently you create new ones.

Unfortunately I did not realize this was happening when I reviewed your transient changes, but your 24.0.1 release notes confirm it. We ask that you please fix and include this in your next point release.

In the meantime, please update your i2p doc to discourage the transient option. Also in that doc, in the bandwidth section at the bottom, please encourage people to share as much bandwidth and transittunnels as they can, to increase their anonymity with the cover traffic, and so that bitcoin users contribute more to the i2p network than they consume.

If a popular application uses more network resources than it contributes, it has the potential to take down the entire network.

Reference files:

https://github.com/bitcoin/bitcoin/blob/master/src/net.cpp#L499 https://github.com/bitcoin/bitcoin/blob/master/doc/i2p.md

Thank you very much for considering this change.

Graph of number of tunnels on a typical high-speed i2p router, Dec. 15-26: (edit, replaced link) https://i2pd.xyz/partTunnels-12days.png

zzzi2p added the label Bug on Dec 26, 2022

zzzi2p commented at 8:32 pm on December 26, 2022: none

Other ideas to limit resource usage if you want to stick with one connection per transient address:

Limit to one tunnel each way with inbound.quantity=1 outbound.quantity=1
Reuse an address if the connection failed
Reuse an address after the connection closes

Last two are subject to your security goals.

bitcoin deleted a comment on Dec 26, 2022

fanquake commented at 9:09 pm on December 26, 2022: member

@zzzi2p thanks for reporting and the writeup / numbers.

cc @vasild @jonatack

zzzi2p commented at 10:05 pm on December 26, 2022: none

Here’s some further background and another picture.

This is a graph of our exploratory tunnel build success rate, network average, which is a key network health metric for us. It correlates (inversely) to the graph linked above.

It’s been cut almost in half, in only a week, to about 24%, which is the lowest it’s been in seven years. We consider anything below 20% to be dire. We of course have multiple layers of defenses to try to prevent congestion collapse but it could still happen.

A couple things that would help us:

If you have an estimate of how many addresses per minute or hour a typical node is creating
If you know or can guess what downstream project in particular might have enabled transient addresses by default or made it easy for users to do so, (the graph below may provide clues) so that we may contact the devs

thanks

https://geti2p.net/_static/images/tunnel.buildExploratorySuccess.10m_31_0.png

jonatack commented at 10:42 pm on December 26, 2022: contributor

Thanks. Discussion started at https://twitter.com/jonatack/status/1605652404862537728. Will reply here tomorrow.

zzzi2p commented at 11:44 pm on December 26, 2022: none

One of our users @Tracachang did some testing, and reported only 4 I2P connections with Tor enabled, and 9 with onlynet=i2p. That’s less than I would have guessed and on its face shouldn’t be a problem.

One theory I have is that i2pd doesn’t have any tunnel build throttling, so it essentially gets into local congestion collapse after its builds get consistently rejected by other routers. I’m in discussions with i2pd’s @orignal about adding some throttling so it doesn’t spam the network when its builds get consistently rejected by other routers. Further investigation required.

zzzi2p commented at 8:39 am on December 27, 2022: none

I’ve refined my thesis, I do think it’s individual i2pd routers getting pushed too hard.

Bitcoin tries to build too many tunnels, and i2pd starts getting all its build requests rejected. I2pd does not properly rate limit build requests, and ends up spamming the network. Please work with @orignal to reproduce and diagnose.

orignal commented at 2:37 pm on December 27, 2022: none

i2pd created 5 inbound and 5 outbound tunnels per destination by default. If bitcoin maintains 16 peer connection, then 160 tunnel build requests at least per 10 minutes.

vasild commented at 6:13 pm on December 27, 2022: contributor

Some quick notes:

Transient/disposable addresses are used only by non-listening nodes. Such nodes make only outbound connections.
It tries to maintain up to 10 outbound connections. Once they are established it would not attempt to make new ones.
For nodes that access other networks too, I2P is probably a small % of those 10 outbound. But for nodes accessing only I2P, then that is all 10 outbound connections.
With a high rate of failure to establish a connection, a node would keep trying to get to 10 outbound connections, making “frequent” attempts.

I guess it makes sense to have a pool of transient addresses + a rate limit on how often entries from the pool are disposed. What would be a reasonable rate limit? Would it depend on the number of bitcoin nodes trying to establish I2P connections?

Limit to one tunnel each way with inbound.quantity=1 outbound.quantity=1

Does that mean max one outbound address at a time?

Reuse an address if the connection failed

That makes perfect sense!

If you have an estimate of how many addresses per minute or hour a typical node is creating

For the last week, my node made ~7 I2P connection attempts per day with ~5 of those attempts ending up with an error. That node is also accessing other networks. For an I2P-only node those numbers would be higher. For the same period that node made ~150 new outbound connections per day (~6 per hour, including to non-I2P addresses).

If bitcoin maintains 16 peer connection, then 160 tunnel build requests at least per 10 minutes.

It is max 10 peer connections (for the relevant case of transient addresses of a non listening node). It is 10 for I2P-only nodes, which I guess the majority are not. Where does the “10 minutes” come from?

I will work on a patch to reuse the address if connection attempt fails, maybe also after connection is closed - aka pool of addresses.

Thanks!

orignal commented at 7:45 pm on December 27, 2022: none

Where does the “10 minutes” come from?

I2P tunnel live time. Just 10 minutes, not more not less. Even if your tunnel is active you must build new one after 10 minutes and switch

jonatack commented at 9:32 pm on December 27, 2022: contributor

Thanks everyone for sharing here. Some quick info:

My node continuously runs all of the networks supported by Bitcoin Core: IPv4, IPv6, Tor, I2P, and CJDNS, all with listening on and no settings to throttle bandwidth. It generally has 0 (most often) to 2 outbound I2P peers out of the possible 10. OTOH, it usually has 10-20 inbound I2P peers and these tend to be stable, long-lasting connections.
Running -addrinfo shows that my node is aware of only a couple hundred I2P peers on the network at this time. Of those, I’m not sure how many run v24 and changed the default behavior of I2P listening on (it is possible that very few node operators are aware of the transient I2P addresses added in v24).

 0$ ./src/bitcoin-cli -rpcwait -addrinfo
 1{
 2  "addresses_known": {
 3    "ipv4": 46047,
 4    "ipv6": 10580,
 5    "onion": 11913,
 6    "i2p": 238,
 7    "cjdns": 3,
 8    "total": 68781
 9  }
10}

A theoretical maximum number of outbound I2P connections for a node that only connects using I2P would be 10-11 automatic, and up to 8 additional manual ones if the user specifies that many. Note, however, that using only I2P with Bitcoin Core is not a good idea for the reasons given in https://jonatack.github.io/articles/using-alternative-p2p-networks-with-bitcoin-core (sybil/eclipse, network partitioning, and slow IBD). These are also partly discussed in doc/i2p.md.

One of our users @Tracachang did some testing, and reported only 4 I2P connections with Tor enabled, and 9 with onlynet=i2p. That’s less than I would have guessed and on its face shouldn’t be a problem.

Yes, those sound about right.

I will work on a patch to reuse the address if connection attempt fails, maybe also after connection is closed - aka pool of addresses.

SGTM.

In the meantime, please update your i2p doc to discourage the transient option. Also in that doc, in the bandwidth section at the bottom, please encourage people to share as much bandwidth and transittunnels as they can, to increase their anonymity with the cover traffic, and so that bitcoin users contribute more to the i2p network than they consume. If a popular application uses more network resources than it contributes, it has the potential to take down the entire network.

That’s fair. I’ll work on your documentation suggestions.

mzumsande commented at 9:35 pm on December 27, 2022: contributor

For nodes that access other networks too, I2P is probably a small % of those 10 outbound. But for nodes accessing only I2P, then that is all 10 outbound connections.

In addition to this, even when all outbound slots are full, bitcoind makes short-lived connections regularly: It makes an additional connection every 2 minutes on average to get to know more possible peers (“feeler connections”), and every 5 minutes to compare the tip (“extra block-relay-only peers”).

jonatack commented at 9:42 pm on December 27, 2022: contributor

even when all outbound slots are full, bitcoind makes short-lived connections regularly

Yes, thus my number of 10-11 (8 full-relay connections, 2 block-relay-only ones, and occasionally 1 short-lived feeler or extra outbound block-relay-only connection). It is true that the latter ones cause connection churn, but on my node they are very rarely connections to I2P or CJDNS nodes, which may not be surprising given how few there are currently. (Edit: running only with I2P would be different, but IBD can take a couple months in that case, i.e. don’t do it.)

ghost commented at 9:47 pm on December 27, 2022: none

For nodes that access other networks too, I2P is probably a small % of those 10 outbound. But for nodes accessing only I2P, then that is all 10 outbound connections.

In addition to this, even when all outbound slots are full, bitcoind makes short-lived connections regularly: It makes an additional connection every 2 minutes on average to get to know more possible peers (“feeler connections”), and every 5 minutes to compare the tip (“extra block-relay-only peers”).

And multiple connections to same i2p address sometimes if it’s a listening node: https://github.com/bitcoin/bitcoin/issues/26537

zzzi2p commented at 2:12 pm on December 29, 2022: none

Thank you to all for your responses.

The Java I2P team is taking the following actions to protect the network:

Continuing to research possible causes and solutions
Communicate analysis of current network state to all
Provide advice to the i2pd team on tunnel build strategies
Tighten our current tunnel throttles in Java I2P to better protect the network
Fix numerous unrelated SSU2 transport bugs that are also contributing to poor network performance
Accelerate our next release by about a month and a half to early January to get fixes out to our users
Enhance our SAM documentation to provide more guidance to application developers

We have another picture for you, of average participating tunnels in the network. https://geti2p.net/_static/images/tunnel.participatingTunnels.60m_31_0.png While the build success rate (in the last graph I posted) is holding steady at about 22%, the tunnel count continues to rise.

One thing I hadn’t realized is that i2pd defaults to 5 tunnels each way per address. Java is 2. So with onlynet=i2p and transient, that’s 2x5x10 = 100 tunnels on i2pd, plus those abandoned when a connection fails. That compares to a typical user that might have 4-10 tunnels. @vasild what you call an address we call a destination or tunnel pool. You configure it with additional “I2CP” options on the SESSION CREATE line. A tunnel pool meant to talk to a single peer really doesn’t need more than two tunnels each way, and in this scenario where you have multiple peers for redundancy anyway, one each way makes sense. By doing:

SESSION CREATE DESTINATION=TRANSIENT inbound.quantity=1 outbound.quantity=1 ….

you will instantly reduce your resource usage by 5x. This does not affect other sessions or limit simulataneous addresses. This configures this session only. You should probably also specify desired quantity for your non-transient sessions, so you get consistent performance with both i2pd and Java I2P.

I agree with the analysis above that the total number of bitcoin+i2p+transient users appears to be small. Additionally, the bitcoin devs are confirming that the connections, once up, are long term, with very few connection attempts per hour.

This is consistent with my thesis that the root cause, for now, is a collapse scenario, where i2pd + bitcoin are sending an even more overwhelming number of tunnel builds after getting rejected for sending an overwhelming number of tunnel builds. It sounds like it might be more likely to happen at node startup. If the common platform for this is an underpowered Raspberry Pi, that may make collapse more likely. A Pi would be a good test case, at least.

The fixes would be both rate limiting of builds in i2pd and rate limiting of address generation in bitcoin. @vasild In bitcoin, a reasonable rate limit might be 1 or 2 every 15 seconds? But the trick might be to also throttle connection attempts, because a failed connection will take much longer to fail. This will naturally throttle things during times of congestion. Once you create a session, i2pd is going to go off and try hard (maybe too hard) to build tunnels for it, no matter what other sessions are doing. I encourage you to work directly with the i2pd team and also use the i2pd web console to see what’s happening during your testing.

Reducing the tunnels per address (inbound.quantity and outbound.quantity), and the number of addresses generated, will also help the short term issue. It’s also important for the long term, to ensure that the application does not consume more network resources than it provides, as the bitcoin+i2p solution becomes more popular.

We’re still looking for any downstream projects that may have enabled transient by default. openoms/raspibiltz maybe? Any others?

Thanks again for your support. This is a complex issue where the causes and possible fixes are only now becoming apparent. We’ll be monitoring the network closely; this weekend might be a wild ride.

The Java I2P team.

zzzi2p commented at 4:23 pm on December 29, 2022: none

Also, full disclosure: While I think I’m on the right track, others differ. After further discussions with the i2pd team: They are unconvinced of my thesis; nor have they reproduced the issue; nor have they offered any comprehensive alternative explanation for the data I’ve presented; nor are they especially concerned about the current network statistics. The best way to work with them is IRC irc.ilita.i2p #dev inside I2P. I’m doing what I can but I’m focused on the Java I2P users getting our next release ready.

jonatack commented at 5:09 pm on December 29, 2022: contributor

Thanks for the update and info! I think your technical suggestions make sense, and moreso in that I reckon many-to-most Bitcoin nodes are using i2pd based on feedback from users.

There was a twitter poll here and here.

We’re still looking for any downstream projects that may have enabled transient by default. openoms/raspibiltz maybe? Any others?

Openoms responded to that poll with this tweet: https://twitter.com/openoms/status/1608109633440284672: “There will be more nodes to connect to over I2P. The next v1.9.0 RaspiBlitz release (and the dev branch now) features i2pacceptincoming=1”. I’m verifying now the thesis that Raspiblitz doesn’t currently support using Bitcoin Core with I2P prior to that new release. Edit: yes, its I2P support is new, uses i2pd and has inbound traffic enabled.

At Adopting Bitcoin in November while doing a session on using I2P and CJDNS with Bitcoin Core, several in the room mentioned that node-in-a-box services were not yet providing I2P support and that they would give feedback to add it.

One hypothesis could be that adoption of I2P by Bitcoin nodes is increasing with the confluence of the v24 release, of recently spreading the message to users and node-in-the-box providers, and perhaps of time over the holidays for updating.

Edit: Umbrel announced on December 22 that it added I2P support to its latest release using i2pd. I believe Umbrel enables I2P inbound connections.

IDK if this is related, but I’m now seeing i2pd failures (with version 2.44.0 (0.9.56)) on my bitcoin core node running current master on macOS 13.1, my I2P local address disappears from -netinfo and getnetworkinfo, all I2P peer connections are lost, and I need to restart i2pd every few hours:

02022-12-29T12:45:45Z [i2p] Creating persistent SAM session bba4fdc409 with 127.0.0.1:7656
12022-12-29T12:45:45Z connect() to 127.0.0.1:7656 failed after wait: Connection refused (61)
22022-12-29T12:45:45Z [i2p] Error listening: Cannot connect to 127.0.0.1:7656
32022-12-29T12:45:45Z [i2p] Control socket error: not connected
4
5... after restarting i2pd (but not bitcoind) with brew services restart i2pd ...
6
72022-12-29T12:54:17Z [i2p] Creating persistent SAM session a82eb58150 with 127.0.0.1:7656
82022-12-29T12:55:17Z [i2p] Persistent SAM session a82eb58150 created, my address=****.b32.i2p:0
92022-12-29T12:55:17Z AddLocal(****.b32.i2p:0,4)

Vort commented at 5:30 pm on December 29, 2022: none

I’m now seeing i2pd failures (with version 2.44.0 (0.9.56))

Version built from latest commit https://github.com/PurpleI2P/i2pd/commit/126dc0ebe061f1d7fb1a5bd435a1a71ac7ba48b4 should have less crashes.

twofaktor referenced this in commit 99cae67a51 on Dec 29, 2022

zzzi2p commented at 10:30 pm on December 29, 2022: none

Thanks Jon.

Independent of the transient issue, at this early stage, it’s important to provide guidance to downstream devs that are bundling and configuring i2pd, or simply giving advice to their users. Especially if those devs don’t have much experience with I2P.

For the most part, defaults work fine, and the other network routers will rate the apparent capacity of the peer and route through them as appropriate. Guides or configs that (explicitly or implicitly) sharply limit the contributions of that peer may result in a whole generation of very-not-nice routers hogging capacity and contributing back.

zzzi2p commented at 10:39 pm on December 29, 2022: none

Here’s our guide for any downstreams that may be bundling i2pd with bitcoin.

https://geti2p.net/en/docs/applications/embedding

It also has some clues about the importance of configurations that may apply for non-bundlers. See the section “Participating Traffic Considerations”

Also, as promised, I have updated our SAM spec

https://geti2p.net/en/docs/api/samv3

to add a “General Guidance” section, note that i2pd and Java I2P have different tunnel quantity defaults, and explain how to override the defaults.

orignal commented at 10:55 pm on December 29, 2022: none

all I2P peer connections are lost, and I need to restart i2pd every few hours:

You are just out of file descriptors. Make sure you set to 8192

While I think I’m on the right track, others differ

Your theory doesn’t explain high traffic.

nor have they reproduced the issue

Because it’s not reproducible, e.g. many SAM sessions don’t cause any issues.

zzzi2p commented at 11:34 am on January 4, 2023: none

Status update:

Network stats bottomed out around 12/24-27 and have stabilized at that low level, we have not collapsed further
We identified a non-bitcoin project also creating ~150 tunnels and are working with them to reduce their resource usage
The relative contributions (if any) of Bitcoin, the other project, and various Java I2P and i2pd bugs and congestion control strategies to the current situation is still not clear
Embedding guide updated with additional advice https://geti2p.net/en/docs/applications/embedding
i2pd 2.45.0 released yesterday
Java I2P 2.1.0 release scheduled for Jan. 9
We’re hopeful the network will pull itself out of congestion by mid-January; will provide another update at that time

The Java I2P team

jonatack commented at 4:23 pm on January 4, 2023: contributor

Thanks for the info and updates, @zzzi2p, @vort and @orignal. Am testing and updating.

orignal commented at 5:33 pm on January 4, 2023: none

Also please test with new release 2.45.0

vasild referenced this in commit 74fa583fa0 on Jan 6, 2023

vasild referenced this in commit 65261b5c6e on Jan 6, 2023

vasild referenced this in commit 57fd27fd36 on Jan 6, 2023

jonatack commented at 8:13 pm on January 6, 2023: contributor

Proposed the requested I2P documentation updates in #26838.

You are just out of file descriptors. Make sure you set to 8192

Thank you. I tried the following in i2pd.conf, restarted i2pd and still saw the issue. Please let me know if that isn’t what you had in mind.

0## Limit number of open file descriptors (0 - use system limit)
1openfiles = 8192

Also please test with new release 2.45.0

Thanks for the release. I’ve been testing for a couple of days now, saw the issue once, then rebooted everything to be more sure and am continuing my testing.

Vort commented at 8:17 pm on January 6, 2023: none

Please let me know if that isn’t what you had in mind.

It should be changed somewhere in Linux. /lib/systemd/system/i2pd.service or something like that.

orignal commented at 8:22 pm on January 6, 2023: none

ulimit -n 8192 But with 2.45.0 is not longer an issue, 4096 is enough.

maflcko referenced this in commit dbca00ef76 on Jan 11, 2023

vasild referenced this in commit b906b64eb7 on Jan 11, 2023

vasild referenced this in commit 801b405f85 on Jan 11, 2023

vasild referenced this in commit 3c1de032de on Jan 11, 2023

jonatack commented at 9:42 pm on January 11, 2023: contributor

ulimit -n 8192 But with 2.45.0 is not longer an issue, 4096 is enough.

Testing with i2pd 2.45.0_1 seems to be stable so far (edit, the issue still occurs for me).

Vort commented at 9:45 pm on January 11, 2023: none

2.45.1 is released 1 hour ago by the way.

jonatack commented at 3:28 pm on January 13, 2023: contributor

2.45.1 is released 1 hour ago by the way.

Thanks! I upgraded immediately and have regularly hit the same issue that I reported above with 2.45.1. It generally happens after roughly a day.

kristapsk referenced this in commit 7dd8fe7914 on Jan 16, 2023

zzzi2p commented at 2:58 pm on January 24, 2023: none

As promised, a mid-January update, after consultation with the i2pd devs:

The Java I2P and i2pd releases the week of Jan. 9 did provide relief to the network, but only up to a point. About 45% of the network is on those releases, as is typical for two weeks after. The network storms of tunnel building continue, and while the newest releases are effectively handling it, the other 56% of the network is still getting hammered. We’re still optimistic that conditions will continue to improve as the network updates, but it’s taking longer than we expected.

It’s now clear, a month after I filed this issue, that Bitcoin is at most a very minor contributor and definitely not the main cause. Attacks, other bad applications, or poor congestion management in i2p routers are much higher on the list.

However, your fixes to reduce resource usage are still important, so you don’t become a bigger part of the problem later. The PR looks good to us, and we hope it gets merged in for your next release. Thanks again to the Bitcoin team.

The Java I2P team

zzzi2p commented at 3:30 pm on February 18, 2023: none

Update mid-February:

We are now confident that Dec. 19 marked the start of a DDoS attack that continues to this day. It was not caused by Bitcoin, other applications, or congestion management issues in I2P routers. Our investigations and mitigations continue.

I apologize for the OP theory that Bitcoin could be the cause, and the urgency stated there. My initial research was incorrect.

The PR to reduce resource usage is still important, and we hope it gets merged in for your next release once it is well-tested, as a part of your standard process.

Thanks again to the Bitcoin Core team, especially @vasild and @jonatack for supporting I2P.

fanquake referenced this in commit 30874a7cc9 on Feb 22, 2023

fanquake commented at 6:00 pm on February 22, 2023: member

Closing this post #26837.

fanquake closed this on Feb 22, 2023

fanquake referenced this in commit 5027e93b6a on Feb 27, 2023

fanquake referenced this in commit 29cdf42226 on Feb 27, 2023

fanquake referenced this in commit ab3bd457cd on Feb 27, 2023

janus referenced this in commit 562d0afa45 on Aug 27, 2023

janus referenced this in commit 61c3a4c58a on Aug 27, 2023

janus referenced this in commit 9d74aea7ae on Aug 27, 2023

bitcoin locked this on Feb 22, 2024

I2P: Limit transient addresses #26754