DNS seed node lookup is ambiguous in some (rare) scenarios #23193

practicalswift commented at 4:01 PM on October 5, 2021: contributor

DNS seed node lookup ambiguous in some (rare) scenarios.

To reproduce run:

$ RES_OPTIONS="options ndots:15" LOCALDOMAIN="localtest.me" src/bitcoind

Context from #22098 (comment):

It is outside of the scope of this PR but I think it would make sense to err on the safe side and specify all seed domain names with an ending dot to make them unambiguous.

In other words …
    vSeeds.emplace_back("seed.bitcoin.sipa.be."); // Pieter Wuille, only supports x1, x5, x9, and xd
… instead of the current …
    vSeeds.emplace_back("seed.bitcoin.sipa.be"); // Pieter Wuille, only supports x1, x5, x9, and xd
The former is guaranteed to be interpreted as seed.bitcoin.sipa.be. whereas the latter may be interpreted as sayseed.bitcoin.sipa.be.megacorp.com. depending on the content of the user's /etc/resolv.conf.

The case I'm thinking about is if search megacorp.com is specified in /etc/resolv.conf and say options ndots:5 is used.

Looking up seed.bitcoin.sipa.be would then result in the following DNS traffic:
IP <src> > <resolver>.53: A? seed.bitcoin.sipa.be.megacorp.com.
IP <resolver>.53 > <src>: NXDomain
IP <src> > <resolver>.53: A? seed.bitcoin.sipa.be.
IP <resolver>.53 > <src>: Some valid response.
In other words seed.bitcoin.sipa.be.megacorp.com. would be tried before seed.bitcoin.sipa.be. :(

I can think of three different approaches to fixing this:

Add a bool allow_non_fqdn argument to Lookup* functions, and automatically add a trailing . before passing the hostname string to getaddrinfo if !allow_non_fqdn. This approach would make the choice between FQDN vs "potentially non-FQDN" explicit. I think this is my preferred approach since the caller would explicitly signal his/her intent.
Specify seed node strings as FQDNs by appending . to the seed hostname strings in src/chainparams.cpp's vSeed.
Change from strprintf("x%x.%s", requiredServiceBits, seed) to strprintf("x%x.%s.", requiredServiceBits, seed) when building the seed hostname in CConnman::ThreadDNSAddressSeed().

Background: #22098 (comment) #22098 (comment) #22098 (comment)

practicalswift renamed this:
~~DNS seed node lookup is unambiguous in some (rare) scenarios~~
DNS seed node lookup is ambiguous in some (rare) scenarios
on Oct 5, 2021

ghost commented at 12:47 AM on October 6, 2021: none

Alternative solution:

Use IP addresses for DNS seeds instead of domains and method used by nodes behind a proxy.

Node -> IP address of seed node -> P2P message with address for several other nodes

https://bitcoin.stackexchange.com/a/109535/

practicalswift commented at 9:10 AM on October 6, 2021: contributor

@prayank23 Were you able to reproduce the issue described? :)

ghost commented at 10:08 AM on October 6, 2021: none

I was not able to reproduce this issue with below config in Fedora although learnt something new when I was reading your comments in that PR:

/etc/resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search localtest.me

And running command:

$ sudo RES_OPTIONS="options ndots:15" LOCALDOMAIN="localtest.me" bitcoind

2021-10-06T09:43:51Z dnsseed thread start
2021-10-06T09:43:51Z Loading addresses from DNS seed dnsseed.bitcoin.dashjr.org
2021-10-06T09:43:51Z msghand thread start
2021-10-06T09:43:51Z addcon thread start
2021-10-06T09:43:51Z opencon thread start
2021-10-06T09:43:51Z Loading addresses from DNS seed seed.bitcoin.sipa.be

[prayank@fedora ~]$ nslookup test
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	test.localtest.me
Address: 127.0.0.1

nslookup seed.bitcoin.sipa.be
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	seed.bitcoin.sipa.be
Address: 114.246.97.39

https://pastebin.com/raw/adgvS3JC

However I was assuming it happens in other networks or using different config and there are other issues with use of domains:

Use of DNS servers as explained in https://voluntarymind.com/bitcoin/2021/
One of the 9 domains getting hacked and pointing to malicious nodes for few hours
DNS leaks if onion is used instead of proxy
Few governments like to ask ISPs in their country to ban bitcoin related domains

So I was thinking if we can use IPs instead of domains for bootstrapping. Maybe try on testnet for few months and see how things work.

practicalswift commented at 10:28 AM on October 6, 2021: contributor

@prayank23 Did you get any valid addresses from those DNS seed lookups? Note that despite the log entry 2021-10-06T09:43:51Z Loading addresses from DNS seed seed.bitcoin.sipa.be the actual DNS lookup may have been made against seed.bitcoin.sipa.be.localtest.me instead of seed.bitcoin.sipa.be. You'll have to look at the actual DNS traffic using something like tcpdump -n 'port 53' to be certain.

ghost commented at 10:56 AM on October 6, 2021: none

Tried tcpdump -n 'port 53':

$ sudo tcpdump -n 'port 53'
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:15:37.086033 IP 192.168.170.134.39047 > 192.168.170.2.domain: 58619+ [1au] A? x9.seed.btc.petertodd.org. (54)
16:15:37.086303 IP 192.168.170.134.50888 > 192.168.170.2.domain: 42718+ [1au] AAAA? x9.seed.btc.petertodd.org. (54)
16:15:37.705798 IP 192.168.170.2.domain > 192.168.170.134.39047: 58619 24/0/1 CNAME x9.seed.btc.petertodd.net., A 147.219.148.106, A 46.8.179.222, A 193.198.34.24, A 3.112.7.115, A 34.139.129.19, A 128.199.212.225, A 78.21.167.8,

Checked one of the IP from results:

https://bitnodes.io/nodes/78.21.167.8-8333/

MarcoFalke added the label P2P on Oct 7, 2021

practicalswift commented at 11:49 AM on October 7, 2021: contributor

@prayank23 Oh, that's not the expected result! Could be that the environment variable approach is not working for some reason. Perhaps it would be easier to reproduce using the "traditional" /etc/resolv.conf approach :)

Try this:

$ cat /etc/resolv.conf
# Use CloudFlare public DNS
nameserver 1.1.1.1

# seed.bitcoin.sipa.be.localtest.me (and the other DNS seeds)
# will resolve to 127.0.0.1 since localtest.me is configured to
# respond with 127.0.0.1 for *.localtest.me.
search localtest.me

options ndots:15
$ git diff src/net.cpp
diff --git a/src/net.cpp b/src/net.cpp
index fca53a6f0..45276f5bc 100644
--- a/src/net.cpp
+++ b/src/net.cpp
@@ -1729,6 +1729,8 @@ void CConnman::ThreadDNSAddressSeed()
                 for (const CNetAddr& ip : vIPs) {
                     int nOneDay = 24*3600;
                     CAddress addr = CAddress(CService(ip, Params().GetDefaultPort()), requiredServiceBits);
+                    std::cout << "* DNS seed " << host << " responded with IP " << ip.ToString() << "\n";
+                    std::abort();
                     addr.nTime = GetTime() - 3*nOneDay - rng.randrange(4*nOneDay); // use a random age between 3 and 7 days old
                     vAdd.push_back(addr);
                     found++;
$ make -C src/ bitcoind
$ rm -f ~/.bitcoin/peers.dat
$ tcpdump -n 'port 53' &
$ src/bitcoind -forcednsseed=1
…
IP <src> > 1.1.1.1.53: A? x9.seed.bitcoin.sipa.be.localtest.me.
…
2021-10-01T01:23:45Z Loading addresses from DNS seed seed.bitcoin.sipa.be
* DNS seed x9.seed.bitcoin.sipa.be responded with IP 127.0.0.1
…

Are you able to reproduce now? :)

ghost commented at 12:50 PM on October 8, 2021: none

Are you able to reproduce now? :)

No. I followed everything you shared. Edited resolv.conf, made changes to src/net.cpp, compiled, deleted peers.dat and then run bitcoind

* DNS seed x9.seed.bitcoin.jonasschnelli.ch responded with IP 18.140.56.125
Aborted

practicalswift commented at 3:06 PM on October 8, 2021: contributor

@prayank23

That's super weird! :)

It seems like the instructions in your resolv.conf are disregarded for some reason.

If you have this (and only this) in your /etc/resolv.conf …

# Use CloudFlare public DNS
nameserver 1.1.1.1

# seed.bitcoin.sipa.be.localtest.me (and the other DNS seeds)
# will resolve to 127.0.0.1 since localtest.me is configured to
# respond with 127.0.0.1 for *.localtest.me.
search localtest.me

options ndots:15

… and then do host x9.seed.bitcoin.sipa.be -- what is the result? (It should be 127.0.0.1 given the above configuration since x9.seed.bitcoin.sipa.be.localtest.me. is the actual FQDN being looked up.)

Anyone else who is able to reproduce? :)

ghost commented at 10:30 AM on October 9, 2021: none

@practicalswift Its working. Sorry I had wrong resolv.conf earlier or maybe didn't save properly before exiting editor.

practicalswift commented at 11:05 AM on October 9, 2021: contributor

@prayank23 That's great!

What results do you get from bitcoind if you do the full reproduce routine below? :)

$ cat /etc/resolv.conf
# Use CloudFlare public DNS
nameserver 1.1.1.1

# seed.bitcoin.sipa.be.localtest.me (and the other DNS seeds)
# will resolve to 127.0.0.1 since localtest.me is configured to
# respond with 127.0.0.1 for *.localtest.me.
search localtest.me

options ndots:15
$ git diff src/net.cpp
diff --git a/src/net.cpp b/src/net.cpp
index fca53a6f0..45276f5bc 100644
--- a/src/net.cpp
+++ b/src/net.cpp
@@ -1729,6 +1729,8 @@ void CConnman::ThreadDNSAddressSeed()
                 for (const CNetAddr& ip : vIPs) {
                     int nOneDay = 24*3600;
                     CAddress addr = CAddress(CService(ip, Params().GetDefaultPort()), requiredServiceBits);
+                    std::cout << "* DNS seed " << host << " responded with IP " << ip.ToString() << "\n";
+                    std::abort();
                     addr.nTime = GetTime() - 3*nOneDay - rng.randrange(4*nOneDay); // use a random age between 3 and 7 days old
                     vAdd.push_back(addr);
                     found++;
$ make -C src/ bitcoind
$ rm -f ~/.bitcoin/peers.dat
$ tcpdump -n 'port 53' &
$ src/bitcoind

ghost commented at 11:34 AM on October 9, 2021: none

What results do you get from bitcoind if you do the full reproduce routine below? :)

resolv.conf

 # Use CloudFlare public DNS
 nameserver 1.1.1.1

# seed.bitcoin.sipa.be.localtest.me (and the other DNS seeds)
# will resolve to 127.0.0.1 since localtest.me is configured to
# respond with 127.0.0.1 for *.localtest.me.
search localtest.me

options ndots:15

src/net.cpp

[prayank@fedora ~]$ host x9.seed.bitcoin.wiz.biz
x9.seed.bitcoin.wiz.biz.localtest.me has address 127.0.0.1

rm -f /home/prayank/.bitcoin/peers.dat
tcpdump -n 'port 53'
cd /home/prayank/binaries/dns-seed/usr/local/ and ./bitcoind

2021-10-09T04:34:19Z dnsseed thread start
2021-10-09T04:34:19Z Loading addresses from DNS seed seed.bitcoin.wiz.biz
2021-10-09T04:34:19Z opencon thread start
2021-10-09T04:34:19Z msghand thread start
2021-10-09T04:34:19Z addcon thread start
2021-10-09T04:34:20Z New outbound peer connected: version: 70016, blocks=704207, peer=0 (block-relay-only)
* DNS seed x9.seed.bitcoin.wiz.biz responded with IP 69.64.225.5
Aborted (core dumped)

https://bitnodes.io/nodes/69.64.225.5-8333/

dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:00:17.082328 IP 192.168.170.134.35580 > 192.168.170.2.domain: 47835+ [1au] AAAA? fedoraproject.org. (46)
10:00:17.093759 IP 192.168.170.2.domain > 192.168.170.134.35580: 47835 6/0/1 AAAA 2620:52:3:1:dead:beef:cafe:fed6, AAAA 2605:bc80:3010:600:dead:beef:cafe:feda, AAAA 2620:52:3:1:dead:beef:cafe:fed7, AAAA 2610:28:3090:3001:dead:beef:cafe:fed3, AAAA 2604:1580:fe00:0:dead:beef:cafe:fed1, AAAA 2605:bc80:3010:600:dead:beef:cafe:fed9 (214)
10:01:47.052342 IP 192.168.170.134.58903 > 192.168.170.2.domain: 61722+ [1au] A? fedoraproject.org. (46)
10:01:47.064272 IP 192.168.170.2.domain > 192.168.170.134.58903: 61722 10/0/1 A 140.211.169.206, A 152.19.134.198, A 8.43.85.67, A 152.19.134.142, A 67.219.144.68, A 38.145.60.20, A 209.132.190.2, A 38.145.60.21, A 8.43.85.73, A 140.211.169.196 (206)
10:04:19.136429 IP 192.168.170.134.35432 > 192.168.170.2.domain: 32770+ [1au] A? x9.seed.bitcoin.wiz.biz. (52)
10:04:19.136540 IP 192.168.170.134.53555 > 192.168.170.2.domain: 14664+ [1au] AAAA? x9.seed.bitcoin.wiz.biz. (52)
10:04:21.203964 IP 192.168.170.2.domain > 192.168.170.134.35432: 32770 21/0/1 A 221.168.38.150, A 171.6.209.170, A 82.197.206.139, A 192.166.219.200, A 181.198.182.39, A 83.163.211.75, A 46.229.170.218, A 69.64.225.5, A 90.182.165.18, A 77.93.223.9, A 188.214.132.45, A 91.204.149.5, A 103.232.104.227, A 185.233.148.146, A 81.0.198.25, A 50.125.234.132, A 197.245.46.242, A 216.107.238.165, A 202.239.42.192, A 210.55.57.179, A 217.23.6.148 (388)
10:04:21.426191 IP 192.168.170.2.domain > 192.168.170.134.53555: 14664 10/0/1 AAAA 2605:ae00:203::203, AAAA 2001:f40:908:64ef:a933:9d18:93f5:be0, AAAA 2001:4454:539:d700:274a:5997:e231:ad58, AAAA 2003:cf:b71e:a400:4c1d:75d6:f396:c5a5, AAAA 2001:14bb:ae:5ea7:189:b34c:c58:1ccc, AAAA 2001:19f0:6801:6ec:2::1, AAAA 2a01:430:17:1::ffff:1153, AAAA 2a01:7c8:d001:1c1:5054:ff:feee:3e1a, AAAA 2a01:7e01::f03c:92ff:fe78:405f, AAAA 2620:6e:a000:1:42:42:42:42 (332)
10:05:01.324800 IP 192.168.170.134.44591 > 1.1.1.1.domain: 12666+ A? x9.seed.bitcoin.wiz.biz.localtest.me. (54)
10:05:02.009099 IP 1.1.1.1.domain > 192.168.170.134.44591: 12666 1/0/0 A 127.0.0.1 (70)
10:05:02.011849 IP 192.168.170.134.35867 > 1.1.1.1.domain: 57604+ AAAA? x9.seed.bitcoin.wiz.biz.localtest.me. (54)
10:05:02.550530 IP 1.1.1.1.domain > 192.168.170.134.35867: 57604 0/1/0 (112)
10:05:02.551849 IP 192.168.170.134.50518 > 1.1.1.1.domain: 23051+ MX? x9.seed.bitcoin.wiz.biz.localtest.me. (54)
10:05:02.752406 IP 1.1.1.1.domain > 192.168.170.134.50518: 23051 0/1/0 (112)

practicalswift commented at 11:48 AM on October 9, 2021: contributor

@prayank23 It seems like something is a bit off in the configuration: judging from the tcpdump output it appears that DNS lookups from the host command respect the /etc/resolv.conf settings whereas DNS lookups from bitcoind disregard these settings.

Are you able to test on another system? You don't happen to have a Debian or Ubuntu system to test under?

ghost commented at 12:01 PM on October 9, 2021: none

You don't happen to have a Debian or Ubuntu system to test under?

I have Pop!_OS and Ubuntu. Will try in one of them in sometime. I think there can be few changes. I remember hosts file being different in Fedora and few other things.

tylerchambers commented at 3:34 PM on October 9, 2021: contributor

Even though you have removed the systemd stub resolver form your /etc/resolv.conf, it's possible that getaddrinfo is still using systemd if /etc/nsswitch.conf contains "resolve" in the host directive. This is the default in fedora, but not ubuntu (not sure about debian).

https://www.freedesktop.org/software/systemd/man/nss-resolve.html

ghost commented at 5:10 PM on October 9, 2021: none

@practicalswift I have emailed you details and few questions after testing on Ubuntu

luke-jr commented at 1:38 AM on October 10, 2021: member

This is expected behaviour, though it probably wouldn't hurt to make the names used more absolute.

DNS seeds aren't supposed to be trusted, so IMO not an issue per se.

practicalswift commented at 5:40 AM on October 10, 2021: contributor

@luke-jr

This is expected behaviour, […]

I don't understand: what is expected behaviour?

Is it expected behaviour that the seed node string seed.bitcoin.sipa.be might end up being interpreted as seed.bitcoin.sipa.be.megacorp.com in practice?

What in the world would the use case be for that behaviour? :)

practicalswift commented at 5:45 AM on October 10, 2021: contributor

@prayank23

@practicalswift I have emailed you details and few questions after testing on Ubuntu

Please post here instead. I like all my Bitcoin Core communication to be public for transparency and documentation purposes. And often someone's dumb questions are someone else's dumb questions, too. I don't think there is anything security sensitive here.

ghost commented at 5:51 AM on October 10, 2021: none

I don't think there is anything security sensitive here

I wasn't sure so wanted to confirm before responding.

It's working on Ubuntu and DNS seed domain resolves to localhost. I am still trying to figure out if there is any way this can be exploited remotely.

ghost commented at 11:20 AM on October 10, 2021: none

I couldn't find anything interesting however either this should be fixed with one of the solutions suggested in issue description or we stop using domains to avoid DNS related issues

It can affect users who sometimes use public WiFi but that doesn't make it a vulnerability in Bitcoin Core: https://www.teiss.co.uk/dns-search-suffix-wi-fi-attacks/

Few CVEs also exist in which attacker needs access to victim's machine for few minutes: http://cve.mitre.org/cgi-bin/cvename.cgi?name=2018-12257

luke-jr commented at 3:48 PM on October 10, 2021: member

@practicalswift It's expected that any name resolution will fallback to the "search" domain reparenting. If you type google.com into your browser, for example, if google.com fails to resolve, it will try google.com.localtest.me

Anyone who can control your DNS search domains, can most likely also MITM your DNS lookups, or even your p2p connections. Furthermore, we do not treat DNS seeds as trusted, so resolving them incorrectly should not create any security issues.

practicalswift commented at 4:13 PM on October 11, 2021: contributor

@luke-jr

It's expected that any name resolution will fallback to the "search" domain reparenting.

The "search" domain will not will not be used if a FQDN is passed to getaddrinfo.

In other words:

Looking up seed.bitcoin.sipa.be. (a FQDN: with an ending dot) is guaranteed to give the results for seed.bitcoin.sipa.be. regardless of your "search" domain settings.
Looking up seed.bitcoin.sipa.be (a partially qualified domain name: without an ending dot) is not guaranteed to give the results for seed.bitcoin.sipa.be. as demonstrated by the PoC above. In the example above a lookup for seed.bitcoin.sipa.be resulted in a query for seed.bitcoin.sipa.be.localtest.me..

That's why I'm suggesting that we should err on the safe side and always pass FQDN to getaddrinfo for the seed nodes and in other similar contexts where the intention is to look up a FQDN (rather than a partially qualified domain name).

If you type google.com into your browser, for example, if google.com fails to resolve, it will try google.com.localtest.me

It seems like you've not tested the PoC above: seed.bitcoin.sipa.be.localtest.me. is tested before seed.bitcoin.sipa.be.

Anyone who can control your DNS search domains, can most likely also MITM your DNS lookups[…]

This is a correctness issue. I'm not claiming this is an exploitable security issue in any meaningful way.

Nit: Note that setting environment variables (RES_OPTIONS and LOCALDOMAIN) does not require the ability to MITM DNS lookups.

sipa commented at 6:51 PM on October 11, 2021: member

I think we should just use FQDNs for the DNS seeds.

practicalswift commented at 2:59 PM on October 12, 2021: contributor

@prayank23 Consider submitting that PR (https://github.com/bitcoinknots/bitcoin/pull/40) upstreams too :) I'd be glad to review.

fanquake closed this on Oct 16, 2021

DrahtBot locked this on Oct 30, 2022