net: No more v2 onion support in upstream Tor master

laanwj commented at 1:54 pm on March 3, 2021: member

One of my nodes using onlynet=tor nodes is completely broken with current master as of 97a35f3ae5bff7f4a25d37d8961191fdbc8c5b67.

Tor is manually configured. Relevant bitcoin.conf:

0onlynet=onion
1proxy=127.0.0.1:9050
2onion=127.0.0.1:9050

It does not manage to make any outgoing connections. The log has an endless stream of:

 02021-03-03T13:42:13Z Socks5() connect to […].onion:8333 failed: general failure
 12021-03-03T13:42:13Z Socks5() connect to […].onion:8333 failed: general failure
 22021-03-03T13:42:14Z Socks5() connect to […].onion:8333 failed: general failure
 32021-03-03T13:42:14Z Socks5() connect to […].onion:8333 failed: general failure
 42021-03-03T13:42:15Z Socks5() connect to […].onion:8333 failed: general failure
 52021-03-03T13:42:15Z Socks5() connect to […].onion:8333 failed: general failure
 62021-03-03T13:42:16Z Socks5() connect to […].onion:8333 failed: general failure
 72021-03-03T13:42:16Z Socks5() connect to […].onion:8333 failed: general failure
 82021-03-03T13:42:17Z Socks5() connect to […].onion:8333 failed: general failure
 92021-03-03T13:42:17Z Socks5() connect to […].onion:8333 failed: general failure
102021-03-03T13:42:18Z Socks5() connect to […].onion:8333 failed: general failure
112021-03-03T13:42:18Z Socks5() connect to […].onion:8333 failed: general failure
122021-03-03T13:42:19Z Socks5() connect to […].onion:8333 failed: general failure
132021-03-03T13:42:19Z Socks5() connect to […].onion:8333 failed: general failure
142021-03-03T13:42:20Z Socks5() connect to […].onion:8333 failed: general failure
152021-03-03T13:42:20Z Socks5() connect to […].onion:8333 failed: general failure
16…

(multiple messages per second, all […] are short v2 onion addresses)

 0$ systemctl status tor
 1● tor.service - Anonymizing overlay network for TCP
 2     Loaded: loaded (/etc/systemd/system/tor.service; enabled; vendor preset: enabled)
 3     Active: active (running) since Wed 2021-03-03 14:45:24 CET; 3min 0s ago
 4    Process: 158931 ExecStartPre=/opt/tor/bin/tor -f /opt/tor/etc/tor/torrc --verify-config (code=exited, status=0/SUCCESS)
 5   Main PID: 158962 (tor)
 6      Tasks: 1 (limit: 9367)
 7     Memory: 38.4M
 8     CGroup: /system.slice/tor.service
 9             └─158962 /opt/tor/bin/tor -f /opt/tor/etc/tor/torrc
10
11Mar 03 14:48:20 tor[158962]: Mar 03 14:48:20.000 [warn] Invalid hostname [scrubbed]; rejecting
12Mar 03 14:48:20 tor[158962]: Mar 03 14:48:20.000 [warn] Invalid hostname [scrubbed]; rejecting
13Mar 03 14:48:21 tor[158962]: Mar 03 14:48:21.000 [warn] Invalid hostname [scrubbed]; rejecting
14Mar 03 14:48:21 tor[158962]: Mar 03 14:48:21.000 [warn] Invalid hostname [scrubbed]; rejecting
15Mar 03 14:48:22 tor[158962]: Mar 03 14:48:22.000 [warn] Invalid hostname [scrubbed]; rejecting
16Mar 03 14:48:22 tor[158962]: Mar 03 14:48:22.000 [warn] Invalid hostname [scrubbed]; rejecting
17Mar 03 14:48:23 tor[158962]: Mar 03 14:48:23.000 [warn] Invalid hostname [scrubbed]; rejecting
18Mar 03 14:48:23 tor[158962]: Mar 03 14:48:23.000 [warn] Invalid hostname [scrubbed]; rejecting
19Mar 03 14:48:24 tor[158962]: Mar 03 14:48:24.000 [warn] Invalid hostname [scrubbed]; rejecting
20Mar 03 14:48:24 tor[158962]: Mar 03 14:48:24.000 [warn] Invalid hostname [scrubbed]; rejecting

I’ll try to see if I can figure out what it’s sending. Tor (current master) is otherwise successfully able to connect to hidden services, at least v3 ones.

Did Tor completely remove v2 support?

0$ /opt/tor/bin/tor --version
1Tor version 0.4.6.0-alpha-dev (git-8020ae02abb105db).
2Tor is running on Linux with Libevent 2.1.11-stable, OpenSSL 1.1.1f, Zlib 1.2.11, Liblzma 5.2.4, Libzstd N/A and Glibc 2.31 as libc.
3Tor compiled with GCC version 9.3.0

laanwj added the label P2P on Mar 3, 2021

laanwj commented at 2:05 pm on March 3, 2021: member

Yes, it looks like Tor removed v2 support completely on the master branch:

https://github.com/torproject/tor/commit/c0a23303140fedce06e0b7d88ce475c39703717a https://github.com/torproject/tor/commit/32fc8a116a3a88ad6e7269d5e9afb751e5d39e50

“It’s all v3 now.”

Some information regarding Tor version planning:

02021-03-03 15:17:22     jonatack        0.4.6 FF is March 15, stable ready Jun 15 
12021-03-03 15:19:08     jonatack        0.4.5.7 release planned March 16

laanwj renamed this:
~~net: onlynet=tor node is failing completely on master~~
net: No more v2 onion support in upstream Tor master
on Mar 3, 2021

laanwj added the label Upstream on Mar 3, 2021

jonatack commented at 2:42 pm on March 3, 2021: member

Per https://gitlab.torproject.org/tpo/core/team/-/wikis/NetworkTeam/CoreTorReleases, tor 0.4.6 feature freeze is March 15, stable ready June 15.

Per https://blog.torproject.org/v2-deprecation-timeline:

“July 15th, 2021: 0.4.6.x: Tor will no longer support v2 and support will be removed from the code base. October 15th, 2021: We will release new Tor client stable versions for all supported series that will disable v2.”

Saibato commented at 8:14 am on March 4, 2021: contributor

RSA certificates (v2) is a broken lost case, the faster transition to v3 what ever that will hook in as final algo, since it’s now perfect modular, the better. imho.

laanwj commented at 9:48 am on March 4, 2021: member

Some further IRC discussion w.r.t. handling this

 02021-03-04 10:15:35     vasild  wumpus: wrt torv2 - if torv2 addresses are non-operational on the tor network, then I think we want to stop trying to connect to such ones and maybe even purge them from addrman.
 12021-03-04 10:29:48     wumpus  vasild: I'd agree--though the current situation seems more subtle: they still work on the tor network, but future Tor releases will no longer support them, I think with old tor versions they will still work for a while
 22021-03-04 10:30:52     vasild  hmm, right
 32021-03-04 10:31:11     wumpus  vasild: so we might want to make it an option first ! no strong preference though
 42021-03-04 10:31:15     vasild  nothing can prevent old tor nodes from exchaning torv2 between themselves
 52021-03-04 10:31:25     wumpus  if there is wider agreement to nuke v2 support completely for 0.22 that's ok with me
 62021-03-04 10:32:14     vasild  I am not sure either, when is 22.0 freeze?
 72021-03-04 10:32:42     vasild  "when is the 22.0 freeze"
 82021-03-04 10:32:47     fanquake        [#20851](/bitcoin-bitcoin/20851/)
 92021-03-04 10:32:48     wumpus  currently planned for 2021-06-15
102021-03-04 10:32:48     gribble [#20851](/bitcoin-bitcoin/20851/) | Release schedule for 22.0 · Issue [#20851](/bitcoin-bitcoin/20851/) · bitcoin/bitcoin · GitHub
112021-03-04 10:35:31     vasild  lets see how connecting to torv2 nodes works in e.g. beginning of June and decide then what to do: 1. do nothing, 2. introduce an option --ignore-torv2 with default of false, 3. --ignore-torv2 with default of true, 4. unconditionally ignore torv2 without any new options
122021-03-04 10:36:37     wumpus  right, good summary of the options
132021-03-04 10:38:07     vasild  a test may be something like: run (old) tor node that supports torv2, pick e.g. 100 torv2 addresses from addrman and try to connect each one of them

I agree with @vasild here. Our options are:

do nothing
introduce an option --ignore-torv2 with default of false
--ignore-torv2 with default of true
unconditionally ignore torv2 without any new options

(we might also backport [3] to 0.21.0, while doing [4] for 0.22)

Doing [2] soon makes sense imo to work around this issue for people who like to run bleeding-edge Tor, and to have something to suggest when Tor 0.4.6.x comes out. As I describe in the OP, that’s currently completely broken. I might have a go at implementing this.

Saibato commented at 10:19 am on March 4, 2021: contributor

option 2 seams reasonable.
bcs, Some might still run and prefer kind of v2 in there ports with short addr formats in the wild and have there own directory’s and infra, we can not be sure every port 9050/51 is a classic Tor socks5 proxy (i.e could be. Tor-stealth-x) we are not aware of, and in my view transitions should not be exclusive?

jonatack commented at 10:20 am on March 4, 2021: member

Agree with starting with 2 and then likely graduating to 3 and 4.

laanwj commented at 10:27 am on March 4, 2021: member

we are not aware of, and in my view transitions should not be exclusive?

Just keep in mind that the Tor project has been, both explicitly and reading between the lines, been very clear that v2 is broken.

As for transitions being exclusive, from a complexity point of view there’s something to be said to get rid of legacy stuff as soon as possible. Carrying around old cryptographic primitives for some vague idea of backwards compatibility can lead to a mess (like OpenSSL used to be, at least).

laanwj commented at 10:27 pm on March 30, 2021: member

It looks like the underlying problem here is not the presence of old v2 peers in the peer manager, but that even though it knows 4555 onion peers, somehow the node does not know any v3 addresses. There are also no hardcoded v3 onion addresses to add (#20239):

0>>> data=p.getnodeaddresses(0)
1>>> len(data)
24555
3>>> len([x for x in data if x['address'].endswith('.onion') and len(x['address']) <= 22])
44555
5>>> len([x for x in data if x['address'].endswith('.onion') and len(x['address']) > 22])
60

Compared to another, more busy node that is not -onlynet=onion:

0>>> data=p.getnodeaddresses(0)
1>>> len(data)
265357
3>>> len([x for x in data if x['address'].endswith('.onion') and len(x['address']) <= 22])
44634
5>>> len([x for x in data if x['address'].endswith('.onion') and len(x['address']) > 22])
617

Even 17 is surprisingly few.

jonatack commented at 0:33 am on March 31, 2021: member

Good find. I wouldn’t have thought the issue was the node not knowing v3 peers.

I reproduced the issue by upgrading to tor 0.4.6.1-alpha yesterday but have been able to run a node with onlynet=onion and onlynet=i2p and have it fill up quickly with tor v3 and i2p peers. I’m seeing the warnings for the tor v2 connections, but the frequency goes down quickly after an initial flurry.

0$ ./src/bitcoin-cli getnodeaddresses 0 | jq '.[] | .address' | wc -l
114573
2$ ./src/bitcoin-cli getnodeaddresses 0 | jq '.[] | (select(.address | contains(".onion")) | select(.address | length <= 22)) | .address' | wc -l
35886
4$ ./src/bitcoin-cli getnodeaddresses 0 | jq '.[] | (select(.address | contains(".onion")) | select(.address | length > 22)) | .address' | wc -l
5941
6$ grep "peers.dat" ~/.bitcoin/debug.log
72021-03-30T18:04:05Z Loaded 60672 addresses from peers.dat  3162ms

That node doesn’t have many peers; I restart it often for testing. It does have a couple dozen addnode peers in the conf file.

OTOH the peer rotation seems lower than usual (likely due to the tor v2 connection failures). After 7 hours the highest peer id is only 90 with 20 peers connected (2 inbound, and 18 outbound of which 8 are addnode and 2 are block relay) and seeing around 30 tor v2 warnings in the last hour.

jonatack commented at 8:06 am on March 31, 2021: member

Update here after 14 hours:

0$ ./src/bitcoin-cli getnodeaddresses 0 | jq '.[] | .address' | wc -l
113027
2$ ./src/bitcoin-cli getnodeaddresses 0 | jq '.[] | (select(.address | contains(".onion")) | select(.address | length <= 22)) | .address' | wc -l
35867
4$ ./src/bitcoin-cli getnodeaddresses 0 | jq '.[] | (select(.address | contains(".onion")) | select(.address | length > 22)) | .address' | wc -l
51028

Highest peer id is now 447. Same number of peers connected. ~40 tor v2 warnings in the past hour, but some hours before that only had 20-30.

jonatack commented at 8:22 am on March 31, 2021: member

It seems we could be thinking about:

v3 onion propagation improvements and seeding (#20239)
maybe an opt-in --ignore-torv2 config as discussed above, but indeed it won’t suffice if a node doesn’t know enough tor v3 addresses
an easy way (RPC) for people to see their address counts by network, with a tor v2/v3 breakdown

vasild commented at 11:52 am on March 31, 2021: member

Here: 52375 total addresses, of which: 4443 tor v2 14 tor v3

laanwj commented at 11:53 am on March 31, 2021: member

Agreed!

FWIW: #21560 solved my immediate problem (after deleting peers.dat).

maybe an opt-in –ignore-torv2 config as discussed above, but indeed it won’t suffice if a node doesn’t know enough tor v3 addresses

I’m no longer convinced that this is necessary. At least in my case it would not have improved anything, and once it did know TorV3 addresses it was resolved quickly. I think we can just drop v2 support wholesale as soon as it makes sense, no need for an intermediate option.

An easy way (RPC) for people to see their address counts by network, with a tor v2/v3 breakdown

Could be fully client-side, bitcoin-cli -getaddresstats maybe?

jonatack commented at 1:08 pm on March 31, 2021: member

Could be fully client-side, bitcoin-cli -getaddresstats maybe?

SGTM and right up my alley.

Latest stats on same node: 11405 total addresses, 5848 v2, 1084 v3. Total addresses and v2 are heading down, v3 heading up.

(Update 5 days later: tor v3 address count now up to 2432.)

laanwj referenced this in commit 9be7fe4849 on Apr 6, 2021

laanwj referenced this in commit 0180453471 on Apr 20, 2021

vasild commented at 7:49 am on May 21, 2021: member

Update: 55325 total addresses, of which: 3869 tor v2 242 tor v3

jonatack commented at 8:04 am on May 21, 2021: member

I had 3600 v3 addresses after a month (edit: six weeks) of running with tor master.

laanwj referenced this in commit 07ededa30c on Jun 3, 2021

laanwj commented at 7:02 pm on June 3, 2021: member

Tor v2 support has been dropped in #22050, and

will no longer be in 22.0.
0.21.x will continue to support both (or we could backport this?).
0.20.x and before do not support Tor v3 so users of that will be unable to use hidden services when v2 support is dropped upstream (but can still use Tor otherwise).

vasild commented at 7:58 am on June 4, 2021: member

0.20.x and before do not support Tor v3 so users of that will be unable to use hidden services when v2 support is dropped upstream (but can still use Tor otherwise).

Yeah, users of Bitcoin Core <=0.20.x can listen on Tor v3 service if it is configured in torrc with something like:

0HiddenServiceDir /var/db/tor/bitcoin/
1HiddenServiceVersion 3
2HiddenServicePort 8333 127.0.0.1:8333

jonatack commented at 6:22 pm on July 14, 2021: member

ISTM this issue can now be closed.

vasild commented at 12:29 pm on August 17, 2021: member

Ping. Close?

MarcoFalke commented at 12:55 pm on August 17, 2021: member

Closing for now, can be reopened if something was forgotten.

MarcoFalke closed this on Aug 17, 2021

felipelalli commented at 11:59 pm on March 7, 2022: none

If you are having issues on first connection, summing up:

Delete peers.dat and restart your node.

DrahtBot locked this on Mar 7, 2023

net: No more v2 onion support in upstream Tor master #21351