net: support unix domain sockets for -proxy and -onion #27375

pull pinheadmz wants to merge 12 commits into bitcoin:master from pinheadmz:tor-unix-domain-socket changing 17 files +401 −141
  1. pinheadmz commented at 7:29 pm on March 30, 2023: member

    Closes #27252

    UNIX domain sockets are a mechanism for inter-process communication that are faster than local TCP ports (because there is no need for TCP overhead) and potentially more secure because access is managed by the filesystem instead of serving an open port on the system.

    There has been work on unix domain sockets before but for now I just wanted to start on this single use-case which is enabling unix sockets from the client side, specifically connecting to a local Tor proxy (Tor can listen on unix sockets and even enforces strict curent-user-only access permission before binding) configured by -onion= or -proxy=

    I copied the prefix unix: usage from Tor. With this patch built locally you can test with your own filesystem path (example):

    tor --SocksPort unix:/Users/matthewzipkin/torsocket/x

    bitcoind -proxy=unix:/Users/matthewzipkin/torsocket/x

    Prep work for this feature includes:

    • Moving where and how we create sockaddr and Sock to accommodate AF_UNIX without disturbing CService
    • Expanding Proxy class to represent either a CService or a UNIX socket (by its file path)

    Future work:

    • Enable UNIX sockets for ZMQ (https://github.com/bitcoin/bitcoin/pull/27679)
    • Enable UNIX sockets for I2P SAM proxy (some code is included in this PR but not tested or exposed to user options yet)
    • Enable UNIX sockets on windows where supported
    • Update Network Proxies dialog in GUI to support UNIX sockets
  2. DrahtBot commented at 7:29 pm on March 30, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK Sjors, vasild, tdb3, achow101
    Stale ACK willcl-ark

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #29641 (scripted-diff: Use LogInfo/LogDebug over LogPrintf/LogPrint by maflcko)
    • #29015 (kernel: Streamline util library by ryanofsky)
    • #28834 (net: attempts to connect to all resolved addresses when connecting to a node by sr-gi)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label P2P on Mar 30, 2023
  4. pinheadmz force-pushed on Mar 31, 2023
  5. in src/netbase.cpp:115 in 7785e4d179 outdated
    111@@ -112,6 +112,7 @@ std::string GetNetworkName(enum Network net)
    112     case NET_I2P: return "i2p";
    113     case NET_CJDNS: return "cjdns";
    114     case NET_INTERNAL: return "internal";
    115+    case NET_UNIX: return "unix_domain_socket";
    


    willcl-ark commented at 7:36 am on March 31, 2023:
    I think I’d go for unix here (but only because it’s shorter).

    pinheadmz commented at 5:43 pm on March 31, 2023:
    agreed, updated.
  6. in test/functional/feature_proxy.py:93 in 7785e4d179 outdated
    89@@ -89,13 +90,22 @@ def setup_nodes(self):
    90         else:
    91             self.log.warning("Testing without local IPv6 support")
    92 
    93+        socket_path = os.path.join(self.options.tmpdir, "socket_path")
    


    willcl-ark commented at 7:59 am on March 31, 2023:

    This needs to be shorter (< 92 chars?). From man 7 unix:

     0   Pathname sockets
     1       When binding a socket to a pathname, a few rules should be observed for maximum portability and ease of coding:
     2
     3       *  The pathname in sun_path should be null-terminated.
     4
     5       *  The length of the pathname, including the terminating null byte, should not exceed the size of sun_path.
     6
     7       *  The addrlen argument that describes the enclosing sockaddr_un structure should have a value of at least:
     8
     9              offsetof(struct sockaddr_un, sun_path)+strlen(addr.sun_path)+1
    10
    11          or, more simply, addrlen can be specified as sizeof(struct sockaddr_un).
    12
    13       There is some variation in how implementations handle UNIX domain socket addresses that do not follow the above rules.  For example, some (but not all) implementations append a null terminator if none is present in the supplied
    14       sun_path.
    15
    16       When coding portable applications, keep in mind that some implementations have sun_path as short as 92 bytes.
    

    It probably makes most sense to check this early on in init.cpp?


    willcl-ark commented at 8:00 am on March 31, 2023:
    On the test this is trying to use a path /private/var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build/ci/scratch/test_runner/test_runner_₿_🏃_20230331_010828/feature_proxy_156

    pinheadmz commented at 6:36 pm on March 31, 2023:
    Yeah this one may be tricky to work around. Some of the CI temp dir paths are crazy long like that. I’m going to add an init check & test to throw on socket paths longer than 92 characters. For the sake of passing the tests, I’ll try using a temp path that is in the parent of test_runner_... and we’ll see if that is short enough to pass on the CI VMs.
  7. in src/net.cpp:527 in 7785e4d179 outdated
    523@@ -524,8 +524,10 @@ CNode* CConnman::ConnectNode(CAddress addrConnect, const char *pszDest, bool fCo
    524         } else if (use_proxy) {
    525             sock = CreateSock(proxy.proxy);
    526             if (!sock) {
    527+                LogPrintLevel(BCLog::PROXY, BCLog::Level::Debug, "Failed to create socket to proxy: %s\n", proxy.proxy.ToStringAddr());
    


    willcl-ark commented at 8:08 am on March 31, 2023:
    I wonder if this should be a Warning or at least an Info level message?

    pinheadmz commented at 5:43 pm on March 31, 2023:
    I only added this for my own debugging, we could even just take it out. There is also a “proxy failed to initialize” error message in Socks5() which will get thrown by ConnectThroughProxy() below.

    vasild commented at 7:16 am on April 13, 2023:
    CreateSock() already logs a detailed message if it returns nullptr. This LogPrintLevel() call here is redundant.
  8. in src/init.cpp:1287 in 7785e4d179 outdated
    1282@@ -1283,7 +1283,10 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
    1283             std::string host_out;
    1284             uint16_t port_out{0};
    1285             if (!SplitHostPort(socket_addr, port_out, host_out)) {
    1286-                return InitError(InvalidPortErrMsg(port_option, socket_addr));
    1287+                // Allow unix domain sockets for proxy e.g. unix:/some/file/path
    1288+                if (port_option != "-proxy" || socket_addr.find(ADDR_PREFIX_UNIX) != 0) {
    


    willcl-ark commented at 8:48 am on March 31, 2023:

    Im curious why you didn’t enable it for -onion here, as it seems a compartiviely small addtion:

     0diff --git a/src/init.cpp b/src/init.cpp
     1index c0a320136..53b6b2a4b 100644
     2--- a/src/init.cpp
     3+++ b/src/init.cpp
     4@@ -1283,8 +1283,8 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
     5             std::string host_out;
     6             uint16_t port_out{0};
     7             if (!SplitHostPort(socket_addr, port_out, host_out)) {
     8-                // Allow unix domain sockets for proxy e.g. unix:/some/file/path
     9-                if (port_option != "-proxy" || socket_addr.find(ADDR_PREFIX_UNIX) != 0) {
    10+                // Allow unix domain sockets for -proxy and -onion e.g. unix:/some/file/path
    11+                if ((port_option != "-proxy" || port_option != "-onion") || socket_addr.find(ADDR_PREFIX_UNIX) != 0) {
    12                     return InitError(InvalidPortErrMsg(port_option, socket_addr));
    13                 }
    14             }
    

    pinheadmz commented at 5:41 pm on March 31, 2023:
    To literally answer your question: I was not clear on what -onion actually does! Now I do, adding this in next update.
  9. willcl-ark commented at 9:01 am on March 31, 2023: contributor

    Concept ACK

    With some light testing it seems to work pretty nicely:

     0will@ubuntu in ~/src/bitcoin on  tor-unix-domain-socket [$?] via 🐍 3.7.16 took 6s
     1₿ sudo exa -al /proc/(pidof bitcoind)/fd
     2lrwx------  64 will 31 Mar 09:21 0 -> /dev/null
     3lrwx------  64 will 31 Mar 09:21 1 -> /dev/null
     4lrwx------  64 will 31 Mar 09:21 2 -> /dev/null
     5lr-x------  64 will 31 Mar 09:21 3 -> pipe:[47084978]
     6l-wx------  64 will 31 Mar 09:21 4 -> pipe:[47084978]
     7lrwx------  64 will 31 Mar 09:21 5 -> /home/will/.bitcoin/signet/.lock
     8l-wx------  64 will 31 Mar 09:21 6 -> /home/will/.bitcoin/signet/debug.log
     9lrwx------  64 will 31 Mar 09:21 7 -> anon_inode:[eventpoll]
    10lr-x------  64 will 31 Mar 09:21 8 -> pipe:[47101090]
    11l-wx------  64 will 31 Mar 09:21 9 -> pipe:[47101090]
    12lrwx------  64 will 31 Mar 09:21 10 -> anon_inode:[eventfd]
    13lrwx------@ 64 will 31 Mar 09:21 11 -> socket:[47101093]
    14lrwx------@ 64 will 31 Mar 09:21 12 -> socket:[47101095]
    15lrwx------  64 will 31 Mar 09:21 13 -> /home/will/.bitcoin/signet/blocks/index/LOCK
    16l-wx------  64 will 31 Mar 09:21 14 -> /home/will/.bitcoin/signet/blocks/index/000387.log
    17l-wx------  64 will 31 Mar 09:21 15 -> /home/will/.bitcoin/signet/blocks/index/MANIFEST-000385
    18lrwx------  64 will 31 Mar 09:21 16 -> /home/will/.bitcoin/signet/chainstate/LOCK
    19l-wx------  64 will 31 Mar 09:21 17 -> /home/will/.bitcoin/signet/chainstate/002594.log
    20l-wx------  64 will 31 Mar 09:21 18 -> /home/will/.bitcoin/signet/chainstate/MANIFEST-002593
    21lrwx------  64 will 31 Mar 09:21 19 -> /home/will/.bitcoin/signet/wallets/legacy/.walletlock
    22l-wx------  64 will 31 Mar 09:21 20 -> /home/will/.bitcoin/signet/wallets/legacy/db.log
    23lrwx------  64 will 31 Mar 09:21 21 -> /home/will/.bitcoin/signet/wallets/legacy/wallet.dat
    24lrwx------  64 will 31 Mar 09:21 22 -> /home/will/.bitcoin/signet/wallets/legacy/database/log.0000000001
    25lrwx------  64 will 31 Mar 09:21 23 -> /home/will/.bitcoin/signet/wallets/test-descriptor/wallet.dat
    26lrwx------  64 will 31 Mar 09:21 24 -> /home/will/.bitcoin/signet/wallets/test-descriptor/wallet.dat-journal
    27lrwx------  64 will 31 Mar 09:21 25 -> anon_inode:[eventpoll]
    28lr-x------  64 will 31 Mar 09:21 26 -> pipe:[47084995]
    29l-wx------  64 will 31 Mar 09:21 27 -> pipe:[47084995]
    30lrwx------  64 will 31 Mar 09:21 28 -> anon_inode:[eventfd]
    31lrwx------@ 64 will 31 Mar 09:21 29 -> socket:[47099059]
    32lrwx------@ 64 will 31 Mar 09:21 30 -> socket:[47084996]
    33lrwx------@ 64 will 31 Mar 09:21 31 -> socket:[47084997]
    34lrwx------@ 64 will 31 Mar 09:21 32 -> socket:[47099065]
    35lrwx------@ 64 will 31 Mar 09:21 33 -> socket:[47084998]
    36lrwx------@ 64 will 31 Mar 09:21 34 -> socket:[47100197]
    37lrwx------@ 64 will 31 Mar 09:21 35 -> socket:[47099299]
    38lrwx------@ 64 will 31 Mar 09:21 36 -> socket:[47099533]
    39lrwx------@ 64 will 31 Mar 09:22 37 -> socket:[47099588]
    40lrwx------@ 64 will 31 Mar 09:22 38 -> socket:[47085043]
    41lrwx------@ 64 will 31 Mar 09:22 39 -> socket:[47085047]
    

    With new socket connections being opened for each connection.

    As I mentioned in one of my comments, I am curious why you didn’t go for the -onion case too, to reduce the scope of the change?

    Enabling -onion in the PortErrorMessage check loop did appear to also work nicely (although i did not check whether there were any other side-effects).

    The info shown by -netinfo 4 looked correct in both cases.

  10. pinheadmz force-pushed on Mar 31, 2023
  11. pinheadmz commented at 6:47 pm on March 31, 2023: member
    Thanks for the feedback @willcl-ark. force-push to 2436b00f2231c52e6bee6d7f79690c5e709cd6f9 should address your notes. I’m still expecting some CI failures from platforms that don’t support unix sockets, then I’ll add configure options like wumpus did here
  12. pinheadmz force-pushed on Mar 31, 2023
  13. pinheadmz force-pushed on Mar 31, 2023
  14. DrahtBot added the label Needs rebase on Apr 1, 2023
  15. pinheadmz force-pushed on Apr 2, 2023
  16. pinheadmz force-pushed on Apr 2, 2023
  17. DrahtBot removed the label Needs rebase on Apr 2, 2023
  18. pinheadmz force-pushed on Apr 3, 2023
  19. pinheadmz renamed this:
    net: support unix domain sockets for -proxy
    net: support unix domain sockets for -proxy and -onion
    on Apr 3, 2023
  20. pinheadmz force-pushed on Apr 3, 2023
  21. pinheadmz force-pushed on Apr 3, 2023
  22. pinheadmz force-pushed on Apr 3, 2023
  23. pinheadmz marked this as ready for review on Apr 3, 2023
  24. in src/netaddress.cpp:928 in 44dd4cb505 outdated
    923+    if (IsUnix()) {
    924+        if (*addrlen < (socklen_t)sizeof(struct sockaddr_un))
    925+            return false;
    926+        struct sockaddr_un* paddrun = (struct sockaddr_un*)paddr;
    927+        paddrun->sun_family = AF_UNIX;
    928+        strcpy(paddrun->sun_path, fs::PathToString(m_path).c_str());
    


    vasild commented at 8:28 am on April 13, 2023:

    This is dangerous. Better check the size before copying (untested):

    0        const std::string path{fs::PathToString(m_path)};
    1        if (sizeof(paddrun->sun_path) < path.length() + 1) {
    2            return false;
    3        }
    4        memcpy(paddrun->sun_path, path.c_str(), path.length() + 1);
    
  25. in src/netaddress.cpp:929 in 44dd4cb505 outdated
    924+        if (*addrlen < (socklen_t)sizeof(struct sockaddr_un))
    925+            return false;
    926+        struct sockaddr_un* paddrun = (struct sockaddr_un*)paddr;
    927+        paddrun->sun_family = AF_UNIX;
    928+        strcpy(paddrun->sun_path, fs::PathToString(m_path).c_str());
    929+        *addrlen = strlen(paddrun->sun_path) + sizeof(paddrun);
    


    vasild commented at 8:37 am on April 13, 2023:

    This looks wrong as paddrun is a pointer (sizeof(paddrun) will be always 8 bytes on 64 bit computers). According to https://www.man7.org/linux/man-pages/man7/unix.7.html it should be:

    0offsetof(struct sockaddr_un, sun_path)+strlen(addr.sun_path)+1
    

    FreeBSD has this, defined in sys/un.h:

    0#define SUN_LEN(su) \
    1        (sizeof(*(su)) - sizeof((su)->sun_path) + strlen((su)->sun_path))
    

    and Linux has:

    0/* Evaluate to actual length of the `sockaddr_un' structure.  */
    1# define SUN_LEN(ptr) ((size_t) (((struct sockaddr_un *) 0)->sun_path)        \
    2                      + strlen ((ptr)->sun_path))
    
  26. in src/netbase.cpp:233 in 44dd4cb505 outdated
    228+
    229+        // https://www.man7.org/linux/man-pages/man7/unix.7.html
    230+        //   "When coding portable applications, keep in mind that some
    231+        //   implementations have sun_path as short as 92 bytes."
    232+        if (str.size() > 92)
    233+            return false;
    


    vasild commented at 9:16 am on April 13, 2023:

    Better check the actual sun_path on the system, and also sun_path is supposed to contain the terminating '\0':

    0        if (str.size() + 1 > sizeof(sockaddr_un::sun_path)) {
    1            return false;
    2        }
    
  27. in src/netbase.cpp:237 in 44dd4cb505 outdated
    232+        if (str.size() > 92)
    233+            return false;
    234+
    235+        fs::path path = fs::PathFromString(str);
    236+        if (!fs::exists(path))
    237+            return false;
    


    vasild commented at 9:20 am on April 13, 2023:

    style: from doc/developer-notes.md:

    • If an if only has a single-statement then-clause, it can appear on the same line as the if, without braces. In every other case, braces are required, and the then and else clauses must appear correctly indented on a new line.
  28. in src/netbase.cpp:224 in 44dd4cb505 outdated
    219@@ -219,6 +220,26 @@ bool Lookup(const std::string& name, CService& addr, uint16_t portDefault, bool
    220     if (!ContainsNoNUL(name)) {
    221         return false;
    222     }
    223+
    224+    // unix domain socket
    


    vasild commented at 9:29 am on April 13, 2023:
    It is a bit odd to support unix:/path/to/socket in Lookup() because there is nothing to lookup in that and it is not like the other inputs supported by Lookup() begin with the socket type, e.g. ipv4://1.2.3.4.
  29. in src/netbase.cpp:415 in 44dd4cb505 outdated
    411@@ -391,7 +412,7 @@ bool Socks5(const std::string& strDest, uint16_t port, const ProxyCredentials* a
    412         return false;
    413     }
    414     if (pchRet1[0] != SOCKSVersion::SOCKS5) {
    415-        return error("Proxy failed to initialize");
    416+        return error("Proxy failed to initialize.");
    


    vasild commented at 9:30 am on April 13, 2023:
    Unrelated change.
  30. in src/netbase.cpp:518 in 44dd4cb505 outdated
    513@@ -493,8 +514,10 @@ std::unique_ptr<Sock> CreateSockTCP(const CService& address_family)
    514     }
    515 
    516     // Create a TCP socket in the address family of the specified service.
    517-    SOCKET hSocket = socket(((struct sockaddr*)&sockaddr)->sa_family, SOCK_STREAM, IPPROTO_TCP);
    518+    int protocol = address_family.IsUnix() ? 0 : IPPROTO_TCP;
    519+    SOCKET hSocket = socket(((struct sockaddr*)&sockaddr)->sa_family, SOCK_STREAM, protocol);
    


    vasild commented at 9:40 am on April 13, 2023:

    It is odd to have CreateSockTCP() create a non-TCP socket.

    Also, the code below sets TCP_NODELAY on the socket which does not apply to UNIX sockets.

  31. in test/functional/feature_proxy.py:67 in 44dd4cb505 outdated
    62@@ -60,14 +63,17 @@
    63 # Networks returned by RPC getnetworkinfo, defined in src/rpc/net.cpp::GetNetworksInfo()
    64 NETWORKS = frozenset({NET_IPV4, NET_IPV6, NET_ONION, NET_I2P, NET_CJDNS})
    65 
    66+# Use the shortest temp path possible since socket paths have a 92-char limit
    67+socket_path = os.path.join(tempfile.gettempdir(), "sockpath")
    


    vasild commented at 10:07 am on April 13, 2023:

    It is better to stay inside the individual test directory. This allows running multiple tests in parallel. With the above code multiple tests will collide on the common /tmp/sockpath.

    /tmp/bitcoin_func_test_99opr6sq/socks5_proxy is just 44 chars, should be below the limit on any platform, even if TMPDIR expands to something longer than /tmp.


    pinheadmz commented at 4:55 pm on April 13, 2023:

    See this ci failure: https://cirrus-ci.com/task/5039125594636288?logs=ci#L2638-L2640

    This is why I used a shorter path, sometimes TMPDIR is > 50 characters! You have a good point about getting max sun_path from the system though, maybe it will balance out

    0File "/private/var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build/ci/scratch/build/bitcoin-arm64-apple-darwin/test/functional/test_framework/socks5.py", line 131, in __init__
    1self.s.bind(conf.addr)
    2OSError: AF_UNIX path too long
    

    vasild commented at 11:19 am on April 14, 2023:

    I see. Then, to avoid multiple tests colliding on /tmp/socket, maybe this:

    0socket_path = os.path.join(tempfile.TemporaryDirectory(), "s")
    
  32. in test/functional/feature_proxy.py:443 in 44dd4cb505 outdated
    436@@ -383,6 +437,18 @@ def networks_dict(d):
    437         msg = "Error: Unknown network specified in -onlynet: 'abc'"
    438         self.nodes[1].assert_start_raises_init_error(expected_msg=msg)
    439 
    440+        self.log.info("Test passing too-long unix path to -proxy raises init error")
    441+        self.nodes[1].extra_args = [f"-proxy=unix:{'x' * 93}"]
    442+        if self.have_unix_sockets:
    443+            msg = f"Error: Invalid -proxy address or hostname: 'unix:{'x' * 93}'"
    


    vasild commented at 10:10 am on April 13, 2023:
    Given that the actual limit is platform-dependent, I would suggest to pass something really longer here, e.g. 1000 or 5000 chars.
  33. in src/netaddress.cpp:25 in 44dd4cb505 outdated
    19@@ -20,6 +20,10 @@
    20 #include <iterator>
    21 #include <tuple>
    22 
    23+#if HAVE_SOCKADDR_UN
    24+#include <sys/un.h>
    25+#endif
    


    vasild commented at 10:26 am on April 13, 2023:
    The include is introduced in the first commit and surrounded by #if HAVE_SOCKADDR_UN in the second commit. This means that the first commit is not self-contained (would not compile on some platforms). Better introduce the configure check first.
  34. in src/netaddress.h:149 in 44dd4cb505 outdated
    142@@ -136,9 +143,16 @@ class CNetAddr
    143      */
    144     uint32_t m_scope_id{0};
    145 
    146+    /**
    147+     * Filesystem path for unix domain socket (NET_UNIX)
    148+     */
    149+    fs::path m_path;
    


    vasild commented at 10:34 am on April 13, 2023:

    If this approach is chosen (to extend CNetAddr to support unix file addresses), then it would be better to store the path in the m_addr member instead of introducing a new one. m_addr is a vector of uint8_t - that is generic enough to store anything in it.

    sizeof(CNetAddr) on master is 32 and sizeof(fs::path) is 24 bytes, so CNetAddr would become almost twice as big.


    pinheadmz commented at 8:38 pm on April 13, 2023:
    I’m a bit intimidated by the syntax prevector<ADDR_IPV6_SIZE, uint8_t> m_addr{ADDR_IPV6_SIZE, 0x0}; – is that only 16 bytes?

    vasild commented at 11:33 am on April 14, 2023:

    It begins its life with 16 bytes pre-allocated (without using new or malloc()), but this is just an optimization which can be ignored. It is supposed to behave like std::vector.

    Something like this should work to store std::string in it (untested):

    0std::string path = ...;
    1m_addr.assign(path.begin(), path.end());
    
  35. vasild commented at 3:45 pm on April 13, 2023: contributor

    Concept ACK, this would be nice to have.

    I do not like that the current approach expands enum Network and CNetAddr with the UNIX type, making it first-class-citizen network that has to be handled all over the networking code, whereas it is only needed to connect to the proxy. It is not like this PR adds support for connecting to other Bitcoin nodes over UNIX sockets (using the P2P protocol). This creates meaningless situations like having a CService on a UNIX socket (it is meaningless because CService has port, whereas UNIX sockets don’t have port). Similarly CSubNet on a UNIX socket is meaningless.

    I will play a bit with this to see if it would be possible to constrain the logic in the Proxy class and hide from the rest of the code whether the proxy is at addr:port or unix:/path.

  36. pinheadmz force-pushed on Apr 13, 2023
  37. pinheadmz commented at 8:33 pm on April 13, 2023: member
    Thanks so much for the great review @vasild I think I addressed all your feedback up to the adding UNIX to CNetAddr. I understand your concerns there and we can chat about it, I’ll look at alternatives as well. What I did do in this last push was mostly separate unix sockets from regular sockets, and adding Proxy.GetSocket() to decide which socket factory to call.
  38. pinheadmz force-pushed on Apr 13, 2023
  39. pinheadmz force-pushed on Apr 13, 2023
  40. in configure.ac:1198 in 2c19dd7c1a outdated
    1259@@ -1260,10 +1260,23 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
    1260   ]], [[
    1261     getauxval(AT_HWCAP);
    1262   ]])],
    1263- [ AC_MSG_RESULT([yes]); HAVE_STRONG_GETAUXVAL=1; AC_DEFINE([HAVE_STRONG_GETAUXVAL], [1], [Define this symbol to build code that uses getauxval)]) ],
    1264+ [ AC_MSG_RESULT([yes]); HAVE_STRONG_GETAUXVAL=1; AC_DEFINE([HAVE_STRONG_GETAUXVAL], [1], [Define this symbol to build code that uses getauxval]) ],
    


    vasild commented at 12:59 pm on April 14, 2023:
    Unrelated change, I am not sure I fully understand. Is it the case that before the message was Define this symbol to build code that uses getauxval) and this change just fixes the message to not have a trailing )?

    pinheadmz commented at 10:28 am on May 16, 2023:
    Yeah
  41. pinheadmz force-pushed on Apr 14, 2023
  42. DrahtBot added the label CI failed on May 17, 2023
  43. DrahtBot removed the label CI failed on May 18, 2023
  44. pinheadmz force-pushed on May 26, 2023
  45. pinheadmz commented at 6:58 pm on May 26, 2023: member
    @vasild did an overhaul of this PR and implemented more how we discussed on IRC. Edited the description to explain. I ended up not using std::variant because I felt it added uneeded complexity. Current approach is Proxy just has a is_unix flag and abstracts away the CService methods like IsValid() etc to support TCP or UNIX sockets
  46. pinheadmz force-pushed on May 26, 2023
  47. DrahtBot added the label CI failed on May 26, 2023
  48. pinheadmz force-pushed on May 26, 2023
  49. pinheadmz force-pushed on May 26, 2023
  50. DrahtBot removed the label CI failed on May 26, 2023
  51. DrahtBot added the label Needs rebase on May 30, 2023
  52. pinheadmz force-pushed on May 30, 2023
  53. pinheadmz commented at 8:30 pm on May 30, 2023: member

    🐙 This pull request conflicts with the target branch and needs rebase.

    🕺

  54. DrahtBot added the label CI failed on May 30, 2023
  55. DrahtBot removed the label Needs rebase on May 30, 2023
  56. DrahtBot removed the label CI failed on May 31, 2023
  57. DrahtBot added the label CI failed on Jun 5, 2023
  58. in src/compat/compat.h:40 in 1a2095baf1 outdated
    36@@ -37,6 +37,12 @@
    37 #include <unistd.h>
    38 #endif
    39 
    40+// Linux defines sa_family_t as `unsigned short` but on Windows it's `u_short`
    41+// See https://learn.microsoft.com/en-us/windows/win32/api/winsock/ns-winsock-sockaddr#syntax
    42+#ifdef WIN32
    43+typedef u_short sa_family_t;
    44+#endif
    


    vasild commented at 4:55 am on June 5, 2023:

    The comment gives the impression that sa_family_t is defined on Windows (as u_short), but it is not defined at all (otherwise this typedef u_short sa_family_t; would not compile because it redefines it, right?). Maybe reword the comment to something like:

    0// Windows does not have `sa_family_t` - it defines `sockaddr::sa_family` as `u_short`.
    1// Thus define `sa_family_t` on Windows too so that the rest of the code can use `sa_family_t`.
    
  59. in src/netaddress.h:532 in 1a2095baf1 outdated
    528@@ -529,6 +529,7 @@ class CService : public CNetAddr
    529     uint16_t GetPort() const;
    530     bool GetSockAddr(struct sockaddr* paddr, socklen_t* addrlen) const;
    531     bool SetSockAddr(const struct sockaddr* paddr);
    532+    [[nodiscard]] sa_family_t GetTCPFamily() const;
    


    vasild commented at 8:05 am on June 5, 2023:
    TCP is a specific protocol (at layer 4 in the OSI model) while this refers to layer 3. Maybe rename it to something like GetProtocolFamily(), GetAddressFamily(), GetFamily() or GetSAFamily().
  60. in src/netaddress.cpp:854 in 1a2095baf1 outdated
    849+    case NET_IPV4:
    850+        return AF_INET;
    851+    case NET_IPV6:
    852+        return AF_INET6;
    853+    default:
    854+        return AF_UNSPEC; // Not TCP
    


    vasild commented at 8:07 am on June 5, 2023:
    0        return AF_UNSPEC;
    
  61. in src/netaddress.cpp:832 in 1a2095baf1 outdated
    852+        return AF_INET6;
    853+    default:
    854+        return AF_UNSPEC; // Not TCP
    855+    }
    856+
    857+}
    


    vasild commented at 8:08 am on June 5, 2023:
    Unnecessary empty line.
  62. in src/netaddress.cpp:828 in 1a2095baf1 outdated
    847+{
    848+    switch (m_net) {
    849+    case NET_IPV4:
    850+        return AF_INET;
    851+    case NET_IPV6:
    852+        return AF_INET6;
    


    vasild commented at 8:10 am on June 5, 2023:

    CService::GetSockAddr() treats CJDNS as IPv6. So:

    0    case NET_IPV6:
    1    case NET_CJDNS:
    2        return AF_INET6;
    
  63. in src/netbase.h:216 in 1a2095baf1 outdated
    213+ * Create a TCP or UNIX socket in the given address family.
    214  * @param[in] address_family The socket is created in the same address family as this address.
    215  * @return pointer to the created Sock object or unique_ptr that owns nothing in case of failure
    216  */
    217-std::unique_ptr<Sock> CreateSockTCP(const CService& address_family);
    218+std::unique_ptr<Sock> CreateSockOS(const sa_family_t& address_family);
    


    vasild commented at 8:17 am on June 5, 2023:

    sa_family_t is cheap to copy. No need to pass it as a const reference:

    0std::unique_ptr<Sock> CreateSockOS(sa_family_t address_family);
    
  64. in src/netbase.cpp:495 in 1a2095baf1 outdated
    497+#if HAVE_SOCKADDR_UN
    498+    // Create a UNIX socket.
    499+    if (address_family == AF_UNIX) {
    500+        SOCKET hSocket = socket(AF_UNIX, SOCK_STREAM, 0);
    501+        if (hSocket == INVALID_SOCKET) return nullptr;
    502+        return std::make_unique<Sock>(hSocket);
    


    vasild commented at 8:31 am on June 5, 2023:

    All tuning that happens to the socket below in this function seems relevant to UNIX sockets too, except TCP_NODELAY.

     0--- i/src/netbase.cpp
     1+++ w/src/netbase.cpp
     2@@ -484,23 +484,21 @@ bool Socks5(const std::string& strDest, uint16_t port, const ProxyCredentials* a
     3 
     4 std::unique_ptr<Sock> CreateSockOS(const sa_family_t& address_family)
     5 {
     6     // Not IPv4, IPv6 or UNIX
     7     if (address_family == AF_UNSPEC) return nullptr;
     8 
     9+    int protocol{IPPROTO_TCP};
    10 #if HAVE_SOCKADDR_UN
    11-    // Create a UNIX socket.
    12     if (address_family == AF_UNIX) {
    13-        SOCKET hSocket = socket(AF_UNIX, SOCK_STREAM, 0);
    14-        if (hSocket == INVALID_SOCKET) return nullptr;
    15-        return std::make_unique<Sock>(hSocket);
    16+        protocol = 0;
    17     }
    18 #endif
    19 
    20-    // Create a TCP socket in the address family of the specified service.
    21-    SOCKET hSocket = socket(address_family, SOCK_STREAM, IPPROTO_TCP);
    22+    // Create a socket in the specified address family.
    23+    SOCKET hSocket = socket(address_family, SOCK_STREAM, protocol);
    24     if (hSocket == INVALID_SOCKET) {
    25         return nullptr;
    26     }
    27 
    28     auto sock = std::make_unique<Sock>(hSocket);
    29 
    30@@ -518,23 +516,30 @@ std::unique_ptr<Sock> CreateSockOS(const sa_family_t& address_family)
    31     if (sock->SetSockOpt(SOL_SOCKET, SO_NOSIGPIPE, (void*)&set, sizeof(int)) == SOCKET_ERROR) {
    32         LogPrintf("Error setting SO_NOSIGPIPE on socket: %s, continuing anyway\n",
    33                   NetworkErrorString(WSAGetLastError()));
    34     }
    35 #endif
    36 
    37+    // Set the non-blocking option on the socket.
    38+    if (!sock->SetNonBlocking()) {
    39+        LogPrintf("Error setting socket to non-blocking: %s\n", NetworkErrorString(WSAGetLastError()));
    40+        return nullptr;
    41+    }
    42+
    43+#if HAVE_SOCKADDR_UN
    44+    if (address_family == AF_UNIX) {
    45+        return sock;
    46+    }
    47+#endif
    48+
    49     // Set the no-delay option (disable Nagle's algorithm) on the TCP socket.
    50     const int on{1};
    51     if (sock->SetSockOpt(IPPROTO_TCP, TCP_NODELAY, &on, sizeof(on)) == SOCKET_ERROR) {
    52         LogPrint(BCLog::NET, "Unable to set TCP_NODELAY on a newly created socket, continuing anyway\n");
    53     }
    54 
    55-    // Set the non-blocking option on the socket.
    56-    if (!sock->SetNonBlocking()) {
    57-        LogPrintf("Error setting socket to non-blocking: %s\n", NetworkErrorString(WSAGetLastError()));
    58-        return nullptr;
    59-    }
    60     return sock;
    61 }
    
  65. in src/netbase.h:65 in 1a2095baf1 outdated
    64-    explicit Proxy(const CService &_proxy, bool _randomize_credentials=false): proxy(_proxy), randomize_credentials(_randomize_credentials) {}
    65-
    66-    bool IsValid() const { return proxy.IsValid(); }
    67+    Proxy(): is_unix_socket(false), randomize_credentials(false) {}
    68+    explicit Proxy(const CService &_proxy, bool _randomize_credentials=false): proxy(_proxy), is_unix_socket(false), randomize_credentials(_randomize_credentials) {}
    69+    explicit Proxy(const std::string path, bool _randomize_credentials=false): unix_socket_path(path), is_unix_socket(true), randomize_credentials(_randomize_credentials) {}
    


    vasild commented at 8:48 am on June 5, 2023:

    Pass path by const reference, strings are expensive to copy.

    0    explicit Proxy(const std::string& path, bool _randomize_credentials=false): unix_socket_path(path), is_unix_socket(true), randomize_credentials(_randomize_credentials) {}
    
  66. in src/netbase.h:70 in 1a2095baf1 outdated
    69+    explicit Proxy(const std::string path, bool _randomize_credentials=false): unix_socket_path(path), is_unix_socket(true), randomize_credentials(_randomize_credentials) {}
    70 
    71     CService proxy;
    72+    std::string unix_socket_path;
    73+    bool is_unix_socket;
    74     bool randomize_credentials;
    


    vasild commented at 8:50 am on June 5, 2023:
    Members should start with m_. Maybe also rename the existent randomize_credentials to m_randomize_credentials (in a separate commit).

    vasild commented at 4:13 pm on June 29, 2023:
    The new member unix_socket_path is still without m_ prefix. Only is_unix_socket was renamed to m_is_unix_socket.
  67. in src/netbase.h:86 in 1a2095baf1 outdated
    87+
    88+    std::string ToStringAddrPort() const
    89+    {
    90+        if (is_unix_socket) return unix_socket_path;
    91+        return proxy.ToStringAddrPort();
    92+    }
    


    vasild commented at 8:53 am on June 5, 2023:
    Better omit AddrPort from the name of this method because it does not apply to UNIX sockets. Just ToString() seems fine (but still call proxy.ToStringAddrPort() for non-UNIX sockets).
  68. in src/netbase.cpp:235 in 1a2095baf1 outdated
    230+    // see https://manpages.ubuntu.com/manpages/xenial/en/man7/unix.7.html
    231+    if (str.size() + 1 > sizeof(((sockaddr_un*)nullptr)->sun_path)) return false;
    232+
    233+    // Path must exist
    234+    fs::path path = fs::PathFromString(str);
    235+    if (!fs::exists(path)) return false;
    


    vasild commented at 9:15 am on June 5, 2023:

    I wonder if this check is desirable. It makes IsUnixSocketPath() volatile - for example if the Tor server is not yet started it will be false and the moment it is started it will become true, or if the Tor server is restarted it will flip from true to false and back.

    With this check bitcoind will refuse to start if the Tor server is not started yet. This differs from the TCP case where the Tor server can start later.

    With TCP we only check if the given address is something we can try to connect to. Like “ok, it is 4 digits separated by 3 dots, maybe I can connect to this later when needed”. CService::IsValid() does not check whether somebody is listening on that address and port and that we can connect to it. An equivalent check for UNIX sockets would be “ok, if it starts with unix: and is short enough maybe I can connect to this later when needed”.

  69. in src/init.cpp:1311 in 1a2095baf1 outdated
    1285@@ -1286,7 +1286,14 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
    1286             std::string host_out;
    1287             uint16_t port_out{0};
    1288             if (!SplitHostPort(socket_addr, port_out, host_out)) {
    1289+#if HAVE_SOCKADDR_UN
    1290+                // Allow unix domain sockets for -proxy and -onion e.g. unix:/some/file/path
    1291+                if ((port_option != "-proxy" && port_option != "-onion") || socket_addr.find(ADDR_PREFIX_UNIX) != 0) {
    


    vasild commented at 9:21 am on June 5, 2023:

    This should use IsUnixSocketPath()?

    0                if ((port_option != "-proxy" && port_option != "-onion") || !IsUnixSocketPath(socket_addr)) {
    

    pinheadmz commented at 5:32 pm on June 5, 2023:
    This change would make feature_proxy.py test fail because the error would be "Invalid port..." which is misleading if the real problem is (for example) a too-long file path, which will get caught by IsUnixSocketPath() on L1370 and described as "Invalid -proxy address or hostname...". So I think it makes more sense to keep this line as-is: if we think the user is trying to set a UNIX socket path, we just ignore the invalid port check.
  70. in test/functional/feature_proxy.py:66 in 1a2095baf1 outdated
    62@@ -60,14 +63,17 @@
    63 # Networks returned by RPC getnetworkinfo, defined in src/rpc/net.cpp::GetNetworksInfo()
    64 NETWORKS = frozenset({NET_IPV4, NET_IPV6, NET_ONION, NET_I2P, NET_CJDNS})
    65 
    66+# Use the shortest temp path possible since paths may have as little as 92-char limit
    


    vasild commented at 9:28 am on June 5, 2023:

    nit:

    0# Use the shortest temp path possible since UNIX sockets may have as little as 92-char limit
    
  71. vasild commented at 9:48 am on June 5, 2023: contributor
    Approach ACK 1a2095baf12b8f8995bed513a23d5faf16917fba
  72. DrahtBot removed the label CI failed on Jun 5, 2023
  73. pinheadmz force-pushed on Jun 5, 2023
  74. pinheadmz commented at 5:35 pm on June 5, 2023: member
    @vasild thanks for the excellent detailed review! All comments addressed except where noted.
  75. pinheadmz force-pushed on Jun 22, 2023
  76. pinheadmz commented at 2:43 pm on June 22, 2023: member
    Rebased on master to pass CI (✅ ). @vasild and @willcl-ark would really love re-review from you handsome chaps when convenient!
  77. in test/functional/test_framework/netutil.py:165 in 64d0f234e9 outdated
    152@@ -153,3 +153,12 @@ def test_ipv6_local():
    153     except socket.error:
    154         have_ipv6 = False
    155     return have_ipv6
    156+
    157+def test_unix_socket():
    158+    '''Return True if UNIX sockets are available on this platform.'''
    159+    try:
    160+        socket.AF_UNIX
    


    luke-jr commented at 10:14 pm on June 22, 2023:
    It’s possible this could pass, but the necessary configure check fail, in theory. But maybe we want the test(s) to fail in that scenario anyway.
  78. DrahtBot added the label Needs rebase on Jun 29, 2023
  79. in src/netbase.h:216 in 64d0f234e9 outdated
    213+ * Create a TCP or UNIX socket in the given address family.
    214  * @param[in] address_family The socket is created in the same address family as this address.
    215  * @return pointer to the created Sock object or unique_ptr that owns nothing in case of failure
    216  */
    217-std::unique_ptr<Sock> CreateSockTCP(const CService& address_family);
    218+std::unique_ptr<Sock> CreateSockOS(const sa_family_t address_family);
    


    vasild commented at 3:10 pm on June 29, 2023:

    nit: it does not have to be const when passing by value - there is no way to for the function to modify the variable that is being passed (and have effects visible outside of the function).

    0std::unique_ptr<Sock> CreateSockOS(sa_family_t address_family);
    

    See f3() in http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rf-in


    pinheadmz commented at 7:14 pm on July 3, 2023:
    ah great tip thanks ;-)
  80. vasild commented at 4:12 pm on June 29, 2023: contributor

    64d0f234e9 looks good, except there is some messup in the contents of these two commits:

    87064712899dacfac7f66f60c8ed1c0f4e65c24f netbase: refactor CreateSock() to accept sa_family_t bbff39d3619428df3745d884ea36d7febff337b1 netbase: extend Proxy class to wrap UNIX socket as well as TCP

    The first one introduces one more IsValid() method, in addition to the existent one. The new method uses a variable that does not exist: is_unix_socket, thus it will not compile. Then the other commit removes one of the IsValid() methods, introduces m_is_unix_socket and changes usage from is_unix_socket to m_is_unix_socket.

    To resolve this:

     0diff --git a/src/net.cpp b/src/net.cpp
     1index a50807fb64..556c8a84e7 100644
     2--- a/src/net.cpp
     3+++ b/src/net.cpp
     4@@ -515,13 +515,13 @@ CNode* CConnman::ConnectNode(CAddress addrConnect, const char *pszDest, bool fCo
     5 
     6             if (connected) {
     7                 sock = std::move(conn.sock);
     8                 addr_bind = CAddress{conn.me, NODE_NONE};
     9             }
    10         } else if (use_proxy) {
    11-            sock = CreateSock(proxy.proxy.GetTCPFamily());
    12+            sock = CreateSock(proxy.proxy.GetSAFamily());
    13             if (!sock) {
    14                 return nullptr;
    15             }
    16             LogPrintLevel(BCLog::PROXY, BCLog::Level::Debug, "Using proxy: %s to connect to %s:%s\n", proxy.proxy.ToStringAddrPort(), addrConnect.ToStringAddr(), addrConnect.GetPort());
    17             connected = ConnectThroughProxy(proxy, addrConnect.ToStringAddr(), addrConnect.GetPort(),
    18                                             *sock, nConnectTimeout, proxyConnectionFailed);
    19@@ -537,13 +537,13 @@ CNode* CConnman::ConnectNode(CAddress addrConnect, const char *pszDest, bool fCo
    20         if (!proxyConnectionFailed) {
    21             // If a connection to the node was attempted, and failure (if any) is not caused by a problem connecting to
    22             // the proxy, mark this as an attempt.
    23             addrman.Attempt(addrConnect, fCountFailure);
    24         }
    25     } else if (pszDest && GetNameProxy(proxy)) {
    26-        sock = CreateSock(proxy.proxy.GetTCPFamily());
    27+        sock = CreateSock(proxy.proxy.GetSAFamily());
    28         if (!sock) {
    29             return nullptr;
    30         }
    31         std::string host;
    32         uint16_t port{default_port};
    33         SplitHostPort(std::string(pszDest), port, host);
    34diff --git a/src/netbase.h b/src/netbase.h
    35index 320496f73b..a9e1ea4c62 100644
    36--- a/src/netbase.h
    37+++ b/src/netbase.h
    38@@ -52,30 +52,12 @@ public:
    39     explicit Proxy(const CService &_proxy, bool _randomize_credentials=false): proxy(_proxy), randomize_credentials(_randomize_credentials) {}
    40 
    41     bool IsValid() const { return proxy.IsValid(); }
    42 
    43     CService proxy;
    44     bool randomize_credentials;
    45-
    46-    bool IsValid() const
    47-    {
    48-        if (is_unix_socket) return IsUnixSocketPath(unix_socket_path);
    49-        return proxy.IsValid();
    50-    }
    51-
    52-    sa_family_t GetFamily() const
    53-    {
    54-        if (is_unix_socket) return AF_UNIX;
    55-        return proxy.GetSAFamily();
    56-    }
    57-
    58-    std::string ToStringAddrPort() const
    59-    {
    60-        if (is_unix_socket) return unix_socket_path;
    61-        return proxy.ToStringAddrPort();
    62-    }
    63 };
    64 
    65 /** Credentials for proxy authentication */
    66 struct ProxyCredentials
    67 {
    68     std::string username;
    

    and make sure that the state of the tree after commit netbase: extend Proxy class to wrap UNIX socket as well as TCP is the same as it is now.

    It is also here: https://github.com/vasild/bitcoin/tree/tor-unix-domain-socket

  81. pinheadmz force-pushed on Jul 3, 2023
  82. pinheadmz force-pushed on Jul 3, 2023
  83. pinheadmz commented at 7:17 pm on July 3, 2023: member

    64d0f23 looks good, except there is some messup in the contents of these two commits:

    Thanks @vasild sorry about the messy rebase there. I cleaned up the commits and fixed the const and m_... nits. Rebased on master for merge conflicts.

  84. pinheadmz force-pushed on Jul 3, 2023
  85. pinheadmz force-pushed on Jul 3, 2023
  86. DrahtBot added the label CI failed on Jul 3, 2023
  87. DrahtBot removed the label Needs rebase on Jul 3, 2023
  88. DrahtBot removed the label CI failed on Jul 3, 2023
  89. vasild approved
  90. vasild commented at 9:09 am on July 6, 2023: contributor

    ACK c3b0f137a9ccd894f2f1d287ccdab26310b3aaf2

    Would be an excellent addition, thank you!

  91. willcl-ark commented at 11:37 am on July 11, 2023: contributor

    It’s a tNACK from me currently @ c3b0f137a9 as I can no longer get this functionality to work in practice on Ubuntu 23.04 with Tor 0.4.7.13.

    The send of the data in vSocks5Init seems to be failing, meaning the proxy cannot be used: https://github.com/pinheadmz/bitcoin/blob/c3b0f137a9ccd894f2f1d287ccdab26310b3aaf2/src/netbase.cpp#L372-L375

    I haven’t yet totally ruled out issues with Tor, but I do get responses from the unix socket about it not being an HTTP proxy if I try to send an HTTP request over it, so I conclude that the Tor daemon is listening on the socket correctly waiting for SOCKS requests, and the issue may lie with Bitcoin Core.

    Will continue investigating and report back.

     02023-07-11T11:12:35Z Config file: /home/will/.bitcoin/bitcoin.conf
     12023-07-11T11:12:35Z Config file arg: [main] addnode="xx.xx.xx.xx:8333"
     22023-07-11T11:12:35Z Config file arg: [main] blockfilterindex="1"
     32023-07-11T11:12:35Z Config file arg: [main] blocksdir="/media/will/crucial1/bitcoin"
     42023-07-11T11:12:35Z Config file arg: [main] daemon="1"
     52023-07-11T11:12:35Z Config file arg: [main] debug="net"
     62023-07-11T11:12:35Z Config file arg: [main] debug="proxy"
     72023-07-11T11:12:35Z Config file arg: [main] debug="tor"
     82023-07-11T11:12:35Z Config file arg: [main] discover="1"
     92023-07-11T11:12:35Z Config file arg: [main] externalip="xx.xx.xx.xx"
    102023-07-11T11:12:35Z Config file arg: [main] i2pacceptincoming="1"
    112023-07-11T11:12:35Z Config file arg: [main] i2psam="127.0.0.1:7656"
    122023-07-11T11:12:35Z Config file arg: [main] listen="1"
    132023-07-11T11:12:35Z Config file arg: [main] listenonion="1"
    142023-07-11T11:12:35Z Config file arg: [main] maxconnections="125"
    152023-07-11T11:12:35Z Config file arg: [main] mempoolfullrbf="1"
    162023-07-11T11:12:35Z Config file arg: [main] onion="unix:/run/tor/socks"
    172023-07-11T11:12:35Z Config file arg: [main] onlynet="onion"
    182023-07-11T11:12:35Z Config file arg: [main] port="8833"
    192023-07-11T11:12:35Z Config file arg: [main] rpcauth=****
    202023-07-11T11:12:35Z Config file arg: [main] rpcthreads="16"
    212023-07-11T11:12:35Z Config file arg: [main] rpcworkqueue="64"
    222023-07-11T11:12:35Z Config file arg: [main] server="1"
    232023-07-11T11:12:35Z Config file arg: [main] txindex="1"
    
    0SocksPort unix:/run/tor/socks WorldWritable
    1ControlPort 9051
    2CookieAuthentication 1
    3CookieAuthFileGroupReadable 1
    4DataDirectoryGroupReadable 1
    
  92. vasild referenced this in commit af4eb52ff6 on Jul 12, 2023
  93. pinheadmz force-pushed on Jul 12, 2023
  94. pinheadmz commented at 8:51 pm on July 12, 2023: member

    @willcl-ark wow thank you so much for catching that. After all the refactoring, we had forgotten to actually connect to the unix socket! I added a functional test to ensure that the proxy works now and I have also tested in production with Tor locally.

    This required a fairly intense refactor that included inserting 2 new commits and rebasing several others:

    git range-diff 32a5a20225 c3b0f137a9 b1199bce0f

  95. pinheadmz commented at 8:52 pm on July 12, 2023: member
    @vasild Thanks for your work on this :-) I reorganized the extra commit you sent me. For now, still going without variant for Proxy just to keep the refator more simple, so leaving the i2p stuff mostly alone as well.
  96. DrahtBot added the label CI failed on Jul 12, 2023
  97. in src/netbase.cpp:607 in b1199bce0f outdated
    606+    struct sockaddr_storage sockaddr;
    607+    socklen_t len = sizeof(sockaddr);
    608+    if (sock->Get() == INVALID_SOCKET) {
    609+        LogPrintf("Cannot connect to %s: invalid socket\n", dest.ToStringAddrPort());
    610+        return {};
    611+    }
    


    vasild commented at 8:05 am on July 13, 2023:
    This check can be omitted from commit dc2361401c net: move CreateSock() calls from ConnectNode() to netbase methods since the code checks if (!sock) earlier. CreateSock() will never return a non-empty unique_ptr that contains a Sock object with a bogus internal file descriptor (INVALID_SOCKED).

    pinheadmz commented at 2:00 pm on July 13, 2023:
    thanks
  98. in src/netbase.cpp:539 in b1199bce0f outdated
    535@@ -509,22 +536,10 @@ static void LogConnectFailure(bool manual_connection, const char* fmt, const Arg
    536     }
    537 }
    538 
    539-bool ConnectSocketDirectly(const CService &addrConnect, const Sock& sock, int nTimeout, bool manual_connection)
    540+std::unique_ptr<Sock> ConnectToSocket(std::unique_ptr<Sock> sock, struct sockaddr* sockaddr, socklen_t len, std::string dest_str, bool manual_connection)
    


    vasild commented at 9:54 am on July 13, 2023:

    The interface of taking unique_ptr as an argument and returning the same is a bit weird. Usually functions take unique_ptr when the ownership of the object is transferred from the caller to the function. But in this case it is transferred to the function which then returns it back to the caller. It would be better to have an interface like:

     0bool ConnectToSocket(const Sock& sock, ...);
     1
     2// and call it like:
     3
     4sock = ... // is unique_ptr<Sock>
     5if (!ConnectToSocket(*sock, ...)) {
     6    handle the error
     7    return {}; // empty unique_ptr
     8}
     9
    10return sock;
    

    vasild commented at 9:59 am on July 13, 2023:

    Make the string argument const reference and the function static since it is used only in netbase.cpp:

    0static std::unique_ptr<Sock> ConnectToSocket(std::unique_ptr<Sock> sock, struct sockaddr* sockaddr, socklen_t len, const std::string& dest_str, bool manual_connection)
    

    pinheadmz commented at 2:13 pm on July 13, 2023:
    ah great thanks

    pinheadmz commented at 2:13 pm on July 13, 2023:
    :+1:
  99. in src/netbase.cpp:624 in b1199bce0f outdated
    617+    return ConnectToSocket(std::move(sock), (struct sockaddr*)&sockaddr, len, dest.ToStringAddrPort(), manual_connection);
    618+}
    619+
    620+std::unique_ptr<Sock> Proxy::Connect() const
    621+{
    622+    if (!m_is_unix_socket) return ConnectDirectly(proxy, /*manual_connection=*/true);
    


    vasild commented at 10:30 am on July 13, 2023:
    Calling this on an invalid Proxy (e.g. default-constructed) will cause weird results. Better check if (!IsValid()) { log; return {}; } at the start of this function.

    pinheadmz commented at 2:18 pm on July 13, 2023:
    good catch thanks
  100. in src/netbase.cpp:630 in b1199bce0f outdated
    628+    }
    629+
    630+    const std::string path{m_unix_socket_path.substr(ADDR_PREFIX_UNIX.length())};
    631+
    632+    struct sockaddr_storage sockaddr;
    633+    struct sockaddr_un* paddrun = (struct sockaddr_un*)&sockaddr;
    


    vasild commented at 10:35 am on July 13, 2023:

    There is no need to use the generic sockaddr_storage since we know it is going to be sockaddr_un. Also, clear it for compatibility, see the example in https://www.man7.org/linux/man-pages/man7/unix.7.html

    0    struct sockaddr_un addrun;
    1    memset(&addrun, 0, sizeof(addrun));
    

    pinheadmz commented at 2:27 pm on July 13, 2023:
    ah, thanks. I had a hard time with this one for some reason because casting to sockaddr kept truncating the unix socket path to 14 bytes before it got to Connect(). Anyway this works and I don’t know why I didn’t try it like this at first.
  101. in src/netbase.cpp:627 in b1199bce0f outdated
    619+
    620+std::unique_ptr<Sock> Proxy::Connect() const
    621+{
    622+    if (!m_is_unix_socket) return ConnectDirectly(proxy, /*manual_connection=*/true);
    623+
    624+    auto sock = CreateSock(AF_UNIX);
    


    vasild commented at 10:37 am on July 13, 2023:

    This and below I guess have to be inside #if HAVE_SOCKADDR_UN, otherwise it does not compile on Windows (CI has choked on it).

    https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ is relevant. Would it compile on Windows if we just #include <afunix.h>?


    pinheadmz commented at 2:33 pm on July 13, 2023:

    Ok thanks I added the preprocessor logic to the function.

    I’ll look into adding afunix.h for windows, maybe in compat.h? also add it to the first commit to test for AF_UNIX in configure.ac. Might be better in a follow-up PR since not all windows platforms will have afunix.h and well need to test for it

  102. in src/netbase.cpp:632 in b1199bce0f outdated
    630+    const std::string path{m_unix_socket_path.substr(ADDR_PREFIX_UNIX.length())};
    631+
    632+    struct sockaddr_storage sockaddr;
    633+    struct sockaddr_un* paddrun = (struct sockaddr_un*)&sockaddr;
    634+    paddrun->sun_family = AF_UNIX;
    635+    strcpy(paddrun->sun_path, path.c_str());
    


    vasild commented at 10:39 am on July 13, 2023:

    strcpy() is a dangerous function because it can be easily misused in a way that leads to security flaws. There is not a single usage of it in Bitcoin Core. In this particular case if path is longer it will buffer overflow the fixed size array sun_path[].

    I would suggest to do something like this:

    0// leave the last char in addrun.sun_path[] to be always '\0'
    1memcpy(addrun.sun_path, path.c_str(), std::min(sizeof(addrun.sun_path) - 1, path.length()));
    

    pinheadmz commented at 2:41 pm on July 13, 2023:
    great catch thanks yeah I saw that in your commit but I was confused. makes sense now
  103. in src/netbase.cpp:635 in b1199bce0f outdated
    633+    struct sockaddr_un* paddrun = (struct sockaddr_un*)&sockaddr;
    634+    paddrun->sun_family = AF_UNIX;
    635+    strcpy(paddrun->sun_path, path.c_str());
    636+    socklen_t len = strlen(paddrun->sun_path) + sizeof(paddrun);
    637+
    638+    return ConnectToSocket(std::move(sock), (struct sockaddr*)&sockaddr, len, path, /*manual_connection=*/true);
    


    vasild commented at 11:21 am on July 13, 2023:

    The len calculation seems to be wrong since sizeof(paddrun) is just 8 bytes (it is a pointer). Anyway, you can use just sizeof(sockaddr) (not the pointer) or sizeof(addrun) if you change it to sockaddr_un addrun;

    0    return ConnectToSocket(std::move(sock), (struct sockaddr*)&addrun, sizeof(addrun), path, /*manual_connection=*/true);
    

    pinheadmz commented at 2:43 pm on July 13, 2023:
    got it thanks, fixed this when changing sockaddr_storage to sockaddr_un and it’s now addrun

    vasild commented at 1:49 pm on July 19, 2023:

    In the latest code it is:

    0      socklen_t len = sizeof(addrun);
    1  
    2      if(!ConnectToSocket(*sock, (struct sockaddr*)&addrun, len, path, /*manual_connection=*/true)) {
    

    feel free to ditch the len variable and use sizeof(addrun) directly when calling ConnectToSocket().

  104. in test/functional/feature_proxy.py:256 in b1199bce0f outdated
    252@@ -230,6 +253,13 @@ def run_test(self):
    253             proxies=[self.serv1, self.serv1, self.serv1, self.serv1],
    254             auth=False, test_onion=True, test_cjdns=True)
    255 
    256+        # -proxyrandomize=0
    


    vasild commented at 11:26 am on July 13, 2023:
    What is the meaning of this comment here?

    pinheadmz commented at 3:47 pm on July 13, 2023:
    ah sorry
  105. in test/functional/feature_proxy.py:138 in b1199bce0f outdated
    134         ]
    135         if self.have_ipv6:
    136             args[3] = ['-listen', f'-proxy=[{self.conf3.addr[0]}]:{self.conf3.addr[1]}','-proxyrandomize=0', '-noonion']
    137+        if self.have_unix_sockets:
    138+            args[5] = ['-listen', f'-proxy=unix:{socket_path}', '-proxyrandomize=0']
    139+            args[6] = ['-listen', f'-onion=unix:{socket_path}', '-proxyrandomize=0']
    


    vasild commented at 11:26 am on July 13, 2023:
    Why use -proxyrandomize=0 is it not irrelevant for this test?

    pinheadmz commented at 2:59 pm on July 13, 2023:
    ok fixed this, just using default (with auth) for the test
  106. in src/netbase.h:65 in b1199bce0f outdated
    66-    bool IsValid() const { return proxy.IsValid(); }
    67+    Proxy() : m_is_unix_socket(false), m_randomize_credentials(false) {}
    68+    explicit Proxy(const CService& _proxy, bool _randomize_credentials = false) : proxy(_proxy), m_is_unix_socket(false), m_randomize_credentials(_randomize_credentials) {}
    69+    explicit Proxy(const std::string path, bool _randomize_credentials = false) : m_unix_socket_path(path), m_is_unix_socket(true), m_randomize_credentials(_randomize_credentials) {}
    70 
    71     CService proxy;
    


    vasild commented at 11:41 am on July 13, 2023:

    The Proxy::proxy member is still accessed directly from some high level code:

    0src/net.cpp:494:                            std::make_unique<i2p::sam::Session>(proxy.proxy, &interruptNet);
    1src/qt/optionsdialog.cpp:416:    ui->proxyReachIPv4->setChecked(has_proxy && proxy.proxy == ui_proxy);
    2src/qt/optionsdialog.cpp:419:    ui->proxyReachIPv6->setChecked(has_proxy && proxy.proxy == ui_proxy);
    3src/qt/optionsdialog.cpp:422:    ui->proxyReachTor->setChecked(has_proxy && proxy.proxy == ui_proxy);
    

    even though it may not be set if it is a UNIX socket proxy.

    For the I2P case we can kind of assume it is a TCP proxy, but then better add assert(!proxy.m_is_unix_socket); to ensure that (eh, that’s ugly). Or you can pass the Proxy object to the Session() constructor, change m_control_host from CService to Proxy and use m_control_host.Connect() in Session::Hello() to create the socket. This will transparently add support for UNIX sockets also for the I2P proxy.

    For the GUI case, this function has to be changed:

    https://github.com/bitcoin/bitcoin/blob/b4794740f82e1a08294c8a5f5d586bc1925412f3/src/qt/optionsdialog.cpp#L407-L423

    First it has to check if ui->proxyIp->text().toStdString() starts with unix: and if yes, avoid LookupHost(). Then create Proxy ui_proxy instead of CService ui_proxy.

    When this is done, then Proxy::proxy can be made private inside the Proxy class, together with the m_unix_socket_path and m_is_unix_socket members.


    pinheadmz commented at 6:33 pm on July 13, 2023:

    Ok adding two commits to handle these two affected areas. For the GUI I’ve written logic that accommodates the new Proxy features but I’ll have to submit a follow-up PR to enable unix socket path configuration from the GUI as a separate element from address:port proxy configuration.

    So with this only PR, the user can only set unix sockets outside the GUI, and in the network tab they will see "Options set in this dialog are overridden by the command line..." anyway.

  107. in src/netbase.cpp:512 in b1199bce0f outdated
    517 
    518     // Set the non-blocking option on the socket.
    519     if (!sock->SetNonBlocking()) {
    520         LogPrintf("Error setting socket to non-blocking: %s\n", NetworkErrorString(WSAGetLastError()));
    521         return nullptr;
    522     }
    


    vasild commented at 12:27 pm on July 13, 2023:

    This will leave the socket in a blocking mode. I think the rest of the P2P code expects and assumes non-blocking sockets. Better set this also to non-blocking and make the Socks5() more robust:

      0commit 4c8d08ca2b495da609d5586247dbd2b306e90e61 (HEAD -> tor-unix-domain-socket)
      1Parent: af4eb52ff611342dbd285f218474649f16badbaf
      2Author:     Vasil Dimov <vd@FreeBSD.org>
      3AuthorDate: Thu Jul 13 14:17:30 2023 +0200
      4Commit:     Vasil Dimov <vd@FreeBSD.org>
      5CommitDate: Thu Jul 13 14:17:30 2023 +0200
      6gpg: Signature made Thu Jul 13 14:23:03 2023 CEST
      7gpg:                using RSA key E64D8D45614DB07545D9CCC154DF06F64B55CBBF
      8gpg: Good signature from "Vasil Dimov <vd@myforest.net>" [ultimate]
      9gpg:                 aka "Vasil Dimov <vd@FreeBSD.org>" [ultimate]
     10gpg:                 aka "Vasil Dimov <vasild@gmail.com>" [ultimate]
     11
     12
     13    netbase: use reliable send() during SOCKS5 handshake
     14    
     15    `send(2)` can be interrupted or for another reason it may not fully
     16    complete sending all the bytes. We should be ready to retry the send
     17    with the remaining bytes. This is what `Sock::SendComplete()` does,
     18    thus use it in `Socks5()`.
     19    
     20    Since `Sock::SendComplete()` takes a `CThreadInterrupt` argument,
     21    change also the recv part of `Socks5()` to use `CThreadInterrupt`
     22    instead of a boolean.
     23    
     24    Easier reviewed with `git show -b` (ignore white-space changes).
     25
     26diff --git a/src/net.cpp b/src/net.cpp
     27index a46cd25e90..4eb96f950d 100644
     28--- a/src/net.cpp
     29+++ b/src/net.cpp
     30@@ -2336,13 +2336,12 @@ bool CConnman::Start(CScheduler& scheduler, const Options& connOptions)
     31     }
     32 
     33     //
     34     // Start threads
     35     //
     36     assert(m_msgproc);
     37-    InterruptSocks5(false);
     38     interruptNet.reset();
     39     flagInterruptMsgProc = false;
     40 
     41     {
     42         LOCK(mutexMsgProc);
     43         fMsgProcWake = false;
     44@@ -2408,13 +2407,13 @@ void CConnman::Interrupt()
     45         LOCK(mutexMsgProc);
     46         flagInterruptMsgProc = true;
     47     }
     48     condMsgProc.notify_all();
     49 
     50     interruptNet();
     51-    InterruptSocks5(true);
     52+    g_socks5_interrupt();
     53 
     54     if (semOutbound) {
     55         for (int i=0; i<m_max_outbound; i++) {
     56             semOutbound->post();
     57         }
     58     }
     59diff --git a/src/netbase.cpp b/src/netbase.cpp
     60index 548e1483b8..cee50e67bb 100644
     61--- a/src/netbase.cpp
     62+++ b/src/netbase.cpp
     63@@ -32,13 +32,13 @@ static Proxy proxyInfo[NET_MAX] GUARDED_BY(g_proxyinfo_mutex);
     64 static Proxy nameProxy GUARDED_BY(g_proxyinfo_mutex);
     65 int nConnectTimeout = DEFAULT_CONNECT_TIMEOUT;
     66 bool fNameLookup = DEFAULT_NAME_LOOKUP;
     67 
     68 // Need ample time for negotiation for very slow proxies such as Tor
     69 std::chrono::milliseconds g_socks5_recv_timeout = 20s;
     70-static std::atomic<bool> interruptSocks5Recv(false);
     71+CThreadInterrupt g_socks5_interrupt;
     72 
     73 std::vector<CNetAddr> WrappedGetAddrInfo(const std::string& name, bool allow_lookup)
     74 {
     75     addrinfo ai_hint{};
     76     // We want a TCP port, which is a streaming socket type
     77     ai_hint.ai_socktype = SOCK_STREAM;
     78@@ -289,13 +289,13 @@ enum class IntrRecvError {
     79  * [@param](/bitcoin-bitcoin/contributor/param/) sock The socket (has to be in non-blocking mode) from which to read bytes.
     80  *
     81  * [@returns](/bitcoin-bitcoin/contributor/returns/) An IntrRecvError indicating the resulting status of this read.
     82  *          IntrRecvError::OK only if all of the specified number of bytes were
     83  *          read.
     84  *
     85- * [@see](/bitcoin-bitcoin/contributor/see/) This function can be interrupted by calling InterruptSocks5(bool).
     86+ * [@see](/bitcoin-bitcoin/contributor/see/) This function can be interrupted by calling g_socks5_interrupt().
     87  *      Sockets can be made non-blocking with Sock::SetNonBlocking().
     88  */
     89 static IntrRecvError InterruptibleRecv(uint8_t* data, size_t len, std::chrono::milliseconds timeout, const Sock& sock)
     90 {
     91     auto curTime{Now<SteadyMilliseconds>()};
     92     const auto endTime{curTime + timeout};
     93@@ -317,14 +317,15 @@ static IntrRecvError InterruptibleRecv(uint8_t* data, size_t len, std::chrono::m
     94                     return IntrRecvError::NetworkError;
     95                 }
     96             } else {
     97                 return IntrRecvError::NetworkError;
     98             }
     99         }
    100-        if (interruptSocks5Recv)
    101+        if (g_socks5_interrupt) {
    102             return IntrRecvError::Interrupted;
    103+        }
    104         curTime = Now<SteadyMilliseconds>();
    105     }
    106     return len == 0 ? IntrRecvError::OK : IntrRecvError::Timeout;
    107 }
    108 
    109 /** Convert SOCKS5 reply to an error message */
    110@@ -351,127 +352,120 @@ static std::string Socks5ErrorString(uint8_t err)
    111             return "unknown";
    112     }
    113 }
    114 
    115 bool Socks5(const std::string& strDest, uint16_t port, const ProxyCredentials* auth, const Sock& sock)
    116 {
    117-    IntrRecvError recvr;
    118-    LogPrint(BCLog::NET, "SOCKS5 connecting %s\n", strDest);
    119-    if (strDest.size() > 255) {
    120-        return error("Hostname too long");
    121-    }
    122-    // Construct the version identifier/method selection message
    123-    std::vector<uint8_t> vSocks5Init;
    124-    vSocks5Init.push_back(SOCKSVersion::SOCKS5); // We want the SOCK5 protocol
    125-    if (auth) {
    126-        vSocks5Init.push_back(0x02); // 2 method identifiers follow...
    127-        vSocks5Init.push_back(SOCKS5Method::NOAUTH);
    128-        vSocks5Init.push_back(SOCKS5Method::USER_PASS);
    129-    } else {
    130-        vSocks5Init.push_back(0x01); // 1 method identifier follows...
    131-        vSocks5Init.push_back(SOCKS5Method::NOAUTH);
    132-    }
    133-    ssize_t ret = sock.Send(vSocks5Init.data(), vSocks5Init.size(), MSG_NOSIGNAL);
    134-    if (ret != (ssize_t)vSocks5Init.size()) {
    135-        return error("Error sending to proxy");
    136-    }
    137-    uint8_t pchRet1[2];
    138-    if (InterruptibleRecv(pchRet1, 2, g_socks5_recv_timeout, sock) != IntrRecvError::OK) {
    139-        LogPrintf("Socks5() connect to %s:%d failed: InterruptibleRecv() timeout or other failure\n", strDest, port);
    140-        return false;
    141-    }
    142-    if (pchRet1[0] != SOCKSVersion::SOCKS5) {
    143-        return error("Proxy failed to initialize");
    144-    }
    145-    if (pchRet1[1] == SOCKS5Method::USER_PASS && auth) {
    146-        // Perform username/password authentication (as described in RFC1929)
    147-        std::vector<uint8_t> vAuth;
    148-        vAuth.push_back(0x01); // Current (and only) version of user/pass subnegotiation
    149-        if (auth->username.size() > 255 || auth->password.size() > 255)
    150-            return error("Proxy username or password too long");
    151-        vAuth.push_back(auth->username.size());
    152-        vAuth.insert(vAuth.end(), auth->username.begin(), auth->username.end());
    153-        vAuth.push_back(auth->password.size());
    154-        vAuth.insert(vAuth.end(), auth->password.begin(), auth->password.end());
    155-        ret = sock.Send(vAuth.data(), vAuth.size(), MSG_NOSIGNAL);
    156-        if (ret != (ssize_t)vAuth.size()) {
    157-            return error("Error sending authentication to proxy");
    158-        }
    159-        LogPrint(BCLog::PROXY, "SOCKS5 sending proxy authentication %s:%s\n", auth->username, auth->password);
    160-        uint8_t pchRetA[2];
    161-        if (InterruptibleRecv(pchRetA, 2, g_socks5_recv_timeout, sock) != IntrRecvError::OK) {
    162-            return error("Error reading proxy authentication response");
    163+    try {
    164+        IntrRecvError recvr;
    165+        LogPrint(BCLog::NET, "SOCKS5 connecting %s\n", strDest);
    166+        if (strDest.size() > 255) {
    167+            return error("Hostname too long");
    168         }
    169-        if (pchRetA[0] != 0x01 || pchRetA[1] != 0x00) {
    170-            return error("Proxy authentication unsuccessful");
    171+        // Construct the version identifier/method selection message
    172+        std::string vSocks5Init;
    173+        vSocks5Init.push_back(SOCKSVersion::SOCKS5); // We want the SOCK5 protocol
    174+        if (auth) {
    175+            vSocks5Init.push_back(0x02); // 2 method identifiers follow...
    176+            vSocks5Init.push_back(SOCKS5Method::NOAUTH);
    177+            vSocks5Init.push_back(SOCKS5Method::USER_PASS);
    178+        } else {
    179+            vSocks5Init.push_back(0x01); // 1 method identifier follows...
    180+            vSocks5Init.push_back(SOCKS5Method::NOAUTH);
    181         }
    182-    } else if (pchRet1[1] == SOCKS5Method::NOAUTH) {
    183-        // Perform no authentication
    184-    } else {
    185-        return error("Proxy requested wrong authentication method %02x", pchRet1[1]);
    186-    }
    187-    std::vector<uint8_t> vSocks5;
    188-    vSocks5.push_back(SOCKSVersion::SOCKS5); // VER protocol version
    189-    vSocks5.push_back(SOCKS5Command::CONNECT); // CMD CONNECT
    190-    vSocks5.push_back(0x00); // RSV Reserved must be 0
    191-    vSocks5.push_back(SOCKS5Atyp::DOMAINNAME); // ATYP DOMAINNAME
    192-    vSocks5.push_back(strDest.size()); // Length<=255 is checked at beginning of function
    193-    vSocks5.insert(vSocks5.end(), strDest.begin(), strDest.end());
    194-    vSocks5.push_back((port >> 8) & 0xFF);
    195-    vSocks5.push_back((port >> 0) & 0xFF);
    196-    ret = sock.Send(vSocks5.data(), vSocks5.size(), MSG_NOSIGNAL);
    197-    if (ret != (ssize_t)vSocks5.size()) {
    198-        return error("Error sending to proxy");
    199-    }
    200-    uint8_t pchRet2[4];
    201-    if ((recvr = InterruptibleRecv(pchRet2, 4, g_socks5_recv_timeout, sock)) != IntrRecvError::OK) {
    202-        if (recvr == IntrRecvError::Timeout) {
    203-            /* If a timeout happens here, this effectively means we timed out while connecting
    204-             * to the remote node. This is very common for Tor, so do not print an
    205-             * error message. */
    206+        sock.SendComplete(vSocks5Init, g_socks5_recv_timeout, g_socks5_interrupt);
    207+        uint8_t pchRet1[2];
    208+        if (InterruptibleRecv(pchRet1, 2, g_socks5_recv_timeout, sock) != IntrRecvError::OK) {
    209+            LogPrintf("Socks5() connect to %s:%d failed: InterruptibleRecv() timeout or other failure\n", strDest, port);
    210             return false;
    211+        }
    212+        if (pchRet1[0] != SOCKSVersion::SOCKS5) {
    213+            return error("Proxy failed to initialize");
    214+        }
    215+        if (pchRet1[1] == SOCKS5Method::USER_PASS && auth) {
    216+            // Perform username/password authentication (as described in RFC1929)
    217+            std::string vAuth;
    218+            vAuth.push_back(0x01); // Current (and only) version of user/pass subnegotiation
    219+            if (auth->username.size() > 255 || auth->password.size() > 255)
    220+                return error("Proxy username or password too long");
    221+            vAuth.push_back(auth->username.size());
    222+            vAuth.insert(vAuth.end(), auth->username.begin(), auth->username.end());
    223+            vAuth.push_back(auth->password.size());
    224+            vAuth.insert(vAuth.end(), auth->password.begin(), auth->password.end());
    225+            sock.SendComplete(vAuth, g_socks5_recv_timeout, g_socks5_interrupt);
    226+            LogPrint(BCLog::PROXY, "SOCKS5 sending proxy authentication %s:%s\n", auth->username, auth->password);
    227+            uint8_t pchRetA[2];
    228+            if (InterruptibleRecv(pchRetA, 2, g_socks5_recv_timeout, sock) != IntrRecvError::OK) {
    229+                return error("Error reading proxy authentication response");
    230+            }
    231+            if (pchRetA[0] != 0x01 || pchRetA[1] != 0x00) {
    232+                return error("Proxy authentication unsuccessful");
    233+            }
    234+        } else if (pchRet1[1] == SOCKS5Method::NOAUTH) {
    235+            // Perform no authentication
    236         } else {
    237-            return error("Error while reading proxy response");
    238+            return error("Proxy requested wrong authentication method %02x", pchRet1[1]);
    239         }
    240-    }
    241-    if (pchRet2[0] != SOCKSVersion::SOCKS5) {
    242-        return error("Proxy failed to accept request");
    243-    }
    244-    if (pchRet2[1] != SOCKS5Reply::SUCCEEDED) {
    245-        // Failures to connect to a peer that are not proxy errors
    246-        LogPrintf("Socks5() connect to %s:%d failed: %s\n", strDest, port, Socks5ErrorString(pchRet2[1]));
    247-        return false;
    248-    }
    249-    if (pchRet2[2] != 0x00) { // Reserved field must be 0
    250-        return error("Error: malformed proxy response");
    251-    }
    252-    uint8_t pchRet3[256];
    253-    switch (pchRet2[3])
    254-    {
    255+        std::string vSocks5;
    256+        vSocks5.push_back(SOCKSVersion::SOCKS5);   // VER protocol version
    257+        vSocks5.push_back(SOCKS5Command::CONNECT); // CMD CONNECT
    258+        vSocks5.push_back(0x00);                   // RSV Reserved must be 0
    259+        vSocks5.push_back(SOCKS5Atyp::DOMAINNAME); // ATYP DOMAINNAME
    260+        vSocks5.push_back(strDest.size());         // Length<=255 is checked at beginning of function
    261+        vSocks5.insert(vSocks5.end(), strDest.begin(), strDest.end());
    262+        vSocks5.push_back((port >> 8) & 0xFF);
    263+        vSocks5.push_back((port >> 0) & 0xFF);
    264+        sock.SendComplete(vSocks5, g_socks5_recv_timeout, g_socks5_interrupt);
    265+        uint8_t pchRet2[4];
    266+        if ((recvr = InterruptibleRecv(pchRet2, 4, g_socks5_recv_timeout, sock)) != IntrRecvError::OK) {
    267+            if (recvr == IntrRecvError::Timeout) {
    268+                /* If a timeout happens here, this effectively means we timed out while connecting
    269+                 * to the remote node. This is very common for Tor, so do not print an
    270+                 * error message. */
    271+                return false;
    272+            } else {
    273+                return error("Error while reading proxy response");
    274+            }
    275+        }
    276+        if (pchRet2[0] != SOCKSVersion::SOCKS5) {
    277+            return error("Proxy failed to accept request");
    278+        }
    279+        if (pchRet2[1] != SOCKS5Reply::SUCCEEDED) {
    280+            // Failures to connect to a peer that are not proxy errors
    281+            LogPrintf("Socks5() connect to %s:%d failed: %s\n", strDest, port, Socks5ErrorString(pchRet2[1]));
    282+            return false;
    283+        }
    284+        if (pchRet2[2] != 0x00) { // Reserved field must be 0
    285+            return error("Error: malformed proxy response");
    286+        }
    287+        uint8_t pchRet3[256];
    288+        switch (pchRet2[3]) {
    289         case SOCKS5Atyp::IPV4: recvr = InterruptibleRecv(pchRet3, 4, g_socks5_recv_timeout, sock); break;
    290         case SOCKS5Atyp::IPV6: recvr = InterruptibleRecv(pchRet3, 16, g_socks5_recv_timeout, sock); break;
    291-        case SOCKS5Atyp::DOMAINNAME:
    292-        {
    293+        case SOCKS5Atyp::DOMAINNAME: {
    294             recvr = InterruptibleRecv(pchRet3, 1, g_socks5_recv_timeout, sock);
    295             if (recvr != IntrRecvError::OK) {
    296                 return error("Error reading from proxy");
    297             }
    298             int nRecv = pchRet3[0];
    299             recvr = InterruptibleRecv(pchRet3, nRecv, g_socks5_recv_timeout, sock);
    300             break;
    301         }
    302         default: return error("Error: malformed proxy response");
    303+        }
    304+        if (recvr != IntrRecvError::OK) {
    305+            return error("Error reading from proxy");
    306+        }
    307+        if (InterruptibleRecv(pchRet3, 2, g_socks5_recv_timeout, sock) != IntrRecvError::OK) {
    308+            return error("Error reading from proxy");
    309+        }
    310+        LogPrint(BCLog::NET, "SOCKS5 connected %s\n", strDest);
    311+        return true;
    312+    } catch (const std::runtime_error& e) {
    313+        return error("Error during SOCKS5 proxy handshake: %s", e.what());
    314     }
    315-    if (recvr != IntrRecvError::OK) {
    316-        return error("Error reading from proxy");
    317-    }
    318-    if (InterruptibleRecv(pchRet3, 2, g_socks5_recv_timeout, sock) != IntrRecvError::OK) {
    319-        return error("Error reading from proxy");
    320-    }
    321-    LogPrint(BCLog::NET, "SOCKS5 connected %s\n", strDest);
    322-    return true;
    323 }
    324 
    325 std::unique_ptr<Sock> CreateSockOS(sa_family_t address_family)
    326 {
    327     // Not IPv4, IPv6 or UNIX
    328     if (address_family == AF_UNSPEC) return nullptr;
    329@@ -743,17 +737,12 @@ bool LookupSubNet(const std::string& subnet_str, CSubNet& subnet_out)
    330             return subnet_out.IsValid();
    331         }
    332     }
    333     return false;
    334 }
    335 
    336-void InterruptSocks5(bool interrupt)
    337-{
    338-    interruptSocks5Recv = interrupt;
    339-}
    340-
    341 bool IsBadPort(uint16_t port)
    342 {
    343     /* Don't forget to update doc/p2p-bad-ports.md if you change this list. */
    344 
    345     switch (port) {
    346     case 1:     // tcpmux
    347diff --git a/src/netbase.h b/src/netbase.h
    348index 997bccf30d..3c41ba3834 100644
    349--- a/src/netbase.h
    350+++ b/src/netbase.h
    351@@ -10,12 +10,13 @@
    352 #endif
    353 
    354 #include <compat/compat.h>
    355 #include <netaddress.h>
    356 #include <serialize.h>
    357 #include <util/sock.h>
    358+#include <util/threadinterrupt.h>
    359 
    360 #include <functional>
    361 #include <memory>
    362 #include <stdint.h>
    363 #include <string>
    364 #include <type_traits>
    365@@ -266,13 +267,16 @@ std::unique_ptr<Sock> ConnectDirectly(const CService& dest, bool manual_connecti
    366  */
    367 std::unique_ptr<Sock> ConnectThroughProxy(const Proxy& proxy,
    368                                           const std::string& dest,
    369                                           uint16_t port,
    370                                           bool& proxy_connection_failed);
    371 
    372-void InterruptSocks5(bool interrupt);
    373+/**
    374+ * Interrupt SOCKS5 reads or writes.
    375+ */
    376+extern CThreadInterrupt g_socks5_interrupt;
    377 
    378 /**
    379  * Connect to a specified destination service through an already connected
    380  * SOCKS5 proxy.
    381  *
    382  * [@param](/bitcoin-bitcoin/contributor/param/) strDest The destination fully-qualified domain name.
    383diff --git a/src/test/fuzz/socks5.cpp b/src/test/fuzz/socks5.cpp
    384index 73235b7ced..7378cec7b8 100644
    385--- a/src/test/fuzz/socks5.cpp
    386+++ b/src/test/fuzz/socks5.cpp
    387@@ -29,13 +29,15 @@ void initialize_socks5()
    388 FUZZ_TARGET_INIT(socks5, initialize_socks5)
    389 {
    390     FuzzedDataProvider fuzzed_data_provider{buffer.data(), buffer.size()};
    391     ProxyCredentials proxy_credentials;
    392     proxy_credentials.username = fuzzed_data_provider.ConsumeRandomLengthString(512);
    393     proxy_credentials.password = fuzzed_data_provider.ConsumeRandomLengthString(512);
    394-    InterruptSocks5(fuzzed_data_provider.ConsumeBool());
    395+    if (fuzzed_data_provider.ConsumeBool()) {
    396+        g_socks5_interrupt();
    397+    }
    398     // Set FUZZED_SOCKET_FAKE_LATENCY=1 to exercise recv timeout code paths. This
    399     // will slow down fuzzing.
    400     g_socks5_recv_timeout = (fuzzed_data_provider.ConsumeBool() && std::getenv("FUZZED_SOCKET_FAKE_LATENCY") != nullptr) ? 1ms : default_socks5_recv_timeout;
    401     FuzzedSock fuzzed_sock = ConsumeSock(fuzzed_data_provider);
    402     // This Socks5(...) fuzzing harness would have caught CVE-2017-18350 within
    403     // a few seconds of fuzzing.
    

    pinheadmz commented at 6:36 pm on July 13, 2023:
    Ok I’ll put this statement after SetNonBlocking() but I think the Socks5 stuff should live in a follow-up PR?

    vasild commented at 4:52 pm on July 14, 2023:

    This resulted in netbase.cpp:415:27: runtime error: implicit conversion from type 'int' of value 210 (32-bit, signed) to type 'char' changed the value to -46 (8-bit, signed). This is ok in practice because the data is transferred as bytes anyway over the socket (not as integers). Anyway, to silence the sanitizer:

      0commit 8467080a08327aaf0d853a73412456c44558a9b3
      1Parent: af4eb52ff611342dbd285f218474649f16badbaf
      2Author:     Vasil Dimov <vd@FreeBSD.org>
      3AuthorDate: Fri Jul 14 18:36:34 2023 +0200
      4Commit:     Vasil Dimov <vd@FreeBSD.org>
      5CommitDate: Fri Jul 14 18:36:34 2023 +0200
      6gpg: Signature made Fri Jul 14 18:38:55 2023 CEST
      7gpg:                using RSA key E64D8D45614DB07545D9CCC154DF06F64B55CBBF
      8gpg: Good signature from "Vasil Dimov <vd@myforest.net>" [ultimate]
      9gpg:                 aka "Vasil Dimov <vd@FreeBSD.org>" [ultimate]
     10gpg:                 aka "Vasil Dimov <vasild@gmail.com>" [ultimate]
     11
     12
     13    sock: introduce a generic Sock::SendComplete(const void*, size_t, ...)
     14    
     15    Use that in `Sock::SendComplete(const std::string&, ...)` and introduce
     16    a new `Sock::SendComplete(const std::vector<uint8_t>&, ...)` to be used
     17    in the `Socks5()` function.
     18
     19diff --git a/src/util/sock.cpp b/src/util/sock.cpp
     20index c83869bc77..6cf1b84f18 100644
     21--- a/src/util/sock.cpp
     22+++ b/src/util/sock.cpp
     23@@ -248,25 +248,26 @@ bool Sock::WaitMany(std::chrono::milliseconds timeout, EventsPerSock& events_per
     24     }
     25 
     26     return true;
     27 #endif /* USE_POLL */
     28 }
     29 
     30-void Sock::SendComplete(const std::string& data,
     31+void Sock::SendComplete(const void* data,
     32+                        size_t len,
     33                         std::chrono::milliseconds timeout,
     34                         CThreadInterrupt& interrupt) const
     35 {
     36     const auto deadline = GetTime<std::chrono::milliseconds>() + timeout;
     37     size_t sent{0};
     38 
     39     for (;;) {
     40-        const ssize_t ret{Send(data.data() + sent, data.size() - sent, MSG_NOSIGNAL)};
     41+        const ssize_t ret{Send(static_cast<const uint8_t*>(data) + sent, len - sent, MSG_NOSIGNAL)};
     42 
     43         if (ret > 0) {
     44             sent += static_cast<size_t>(ret);
     45-            if (sent == data.size()) {
     46+            if (sent == len) {
     47                 break;
     48             }
     49         } else {
     50             const int err{WSAGetLastError()};
     51             if (IOErrorIsPermanent(err)) {
     52                 throw std::runtime_error(strprintf("send(): %s", NetworkErrorString(err)));
     53@@ -274,27 +275,41 @@ void Sock::SendComplete(const std::string& data,
     54         }
     55 
     56         const auto now = GetTime<std::chrono::milliseconds>();
     57 
     58         if (now >= deadline) {
     59             throw std::runtime_error(strprintf(
     60-                "Send timeout (sent only %u of %u bytes before that)", sent, data.size()));
     61+                "Send timeout (sent only %u of %u bytes before that)", sent, len));
     62         }
     63 
     64         if (interrupt) {
     65             throw std::runtime_error(strprintf(
     66-                "Send interrupted (sent only %u of %u bytes before that)", sent, data.size()));
     67+                "Send interrupted (sent only %u of %u bytes before that)", sent, len));
     68         }
     69 
     70         // Wait for a short while (or the socket to become ready for sending) before retrying
     71         // if nothing was sent.
     72         const auto wait_time = std::min(deadline - now, std::chrono::milliseconds{MAX_WAIT_FOR_IO});
     73         (void)Wait(wait_time, SEND);
     74     }
     75 }
     76 
     77+void Sock::SendComplete(const std::string& data,
     78+                        std::chrono::milliseconds timeout,
     79+                        CThreadInterrupt& interrupt) const
     80+{
     81+    SendComplete(data.data(), data.size(), timeout, interrupt);
     82+}
     83+
     84+void Sock::SendComplete(const std::vector<uint8_t>& data,
     85+                        std::chrono::milliseconds timeout,
     86+                        CThreadInterrupt& interrupt) const
     87+{
     88+    SendComplete(data.data(), data.size(), timeout, interrupt);
     89+}
     90+
     91 std::string Sock::RecvUntilTerminator(uint8_t terminator,
     92                                       std::chrono::milliseconds timeout,
     93                                       CThreadInterrupt& interrupt,
     94                                       size_t max_data) const
     95 {
     96     const auto deadline = GetTime<std::chrono::milliseconds>() + timeout;
     97diff --git a/src/util/sock.h b/src/util/sock.h
     98index 6bac2dfd34..7f7b426665 100644
     99--- a/src/util/sock.h
    100+++ b/src/util/sock.h
    101@@ -230,21 +230,37 @@ public:
    102 
    103     /* Higher level, convenience, methods. These may throw. */
    104 
    105     /**
    106      * Send the given data, retrying on transient errors.
    107      * [@param](/bitcoin-bitcoin/contributor/param/)[in] data Data to send.
    108+     * [@param](/bitcoin-bitcoin/contributor/param/)[in] len Length of the data in bytes.
    109      * [@param](/bitcoin-bitcoin/contributor/param/)[in] timeout Timeout for the entire operation.
    110      * [@param](/bitcoin-bitcoin/contributor/param/)[in] interrupt If this is signaled then the operation is canceled.
    111      * [@throws](/bitcoin-bitcoin/contributor/throws/) std::runtime_error if the operation cannot be completed. In this case only some of
    112      * the data will be written to the socket.
    113      */
    114+    virtual void SendComplete(const void* data,
    115+                              size_t len,
    116+                              std::chrono::milliseconds timeout,
    117+                              CThreadInterrupt& interrupt) const;
    118+
    119+    /**
    120+     * Convenience method, equivalent to `SendComplete(data.data(), data.size(), timeout, interrupt)`.
    121+     */
    122     virtual void SendComplete(const std::string& data,
    123                               std::chrono::milliseconds timeout,
    124                               CThreadInterrupt& interrupt) const;
    125 
    126+    /**
    127+     * Convenience method, equivalent to `SendComplete(data.data(), data.size(), timeout, interrupt)`.
    128+     */
    129+    virtual void SendComplete(const std::vector<uint8_t>& data,
    130+                              std::chrono::milliseconds timeout,
    131+                              CThreadInterrupt& interrupt) const;
    132+
    133     /**
    134      * Read from socket until a terminator character is encountered. Will never consume bytes past
    135      * the terminator from the socket.
    136      * [@param](/bitcoin-bitcoin/contributor/param/)[in] terminator Character up to which to read from the socket.
    137      * [@param](/bitcoin-bitcoin/contributor/param/)[in] timeout Timeout for the entire operation.
    138      * [@param](/bitcoin-bitcoin/contributor/param/)[in] interrupt If this is signaled then the operation is canceled.
    

    Also in this branch: https://github.com/vasild/bitcoin/tree/tor-unix-domain-socket (the second commit)

    and adapt Socks5() to use std::vector<uint8_t> for the arguments it passes to SendComplete() (done in the last commit of https://github.com/vasild/bitcoin/tree/tor-unix-domain-socket).

    That is, I think the two topmost commits from https://github.com/vasild/bitcoin/tree/tor-unix-domain-socket should be the first commits in this PR.


    pinheadmz commented at 6:33 pm on July 14, 2023:
    Now its ERROR: Error during SOCKS5 proxy handshake: send(): Transport endpoint is not connected (107) I’m still digging in to this…

    vasild commented at 12:13 pm on July 17, 2023:

    Transport endpoint is not connected

    Is this still a problem or it got resolved?


    pinheadmz commented at 12:18 pm on July 17, 2023:
    Resolved!
  108. pinheadmz force-pushed on Jul 13, 2023
  109. pinheadmz force-pushed on Jul 14, 2023
  110. pinheadmz force-pushed on Jul 16, 2023
  111. DrahtBot removed the label CI failed on Jul 16, 2023
  112. pinheadmz commented at 4:13 pm on July 16, 2023: member

    @vasild OK I got this branch passing all CI. ✅ I ran tests locally on Ubuntu and MacOS and also tested the feature in production on both platforms with tor --SocksPort unix:/... Everything is working! yay.

    The real fix was here actually (diff below) Embarrassingly simple. This was a bug on Linux but NOT on macOS.

    I did not need the SendComplete changes or Socks5 changes which is great because that keeps review simple and the branch focused. I will be happy to review those changes if you want to open a new PR for them (or I can open it if you like)

    The only other new change to the branch besides this is adding a release note and explanation in help text for -onion and -proxy

     0diff --git a/src/netbase.cpp b/src/netbase.cpp
     1index a49f882130..044b0ba6c3 100644
     2--- a/src/netbase.cpp
     3+++ b/src/netbase.cpp
     4@@ -638,7 +638,7 @@ std::unique_ptr<Sock> Proxy::Connect() const
     5     addrun.sun_family = AF_UNIX;
     6     // leave the last char in addrun.sun_path[] to be always '\0'
     7     memcpy(addrun.sun_path, path.c_str(), std::min(sizeof(addrun.sun_path) - 1, path.length()));
     8-    socklen_t len = strlen(addrun.sun_path) + sizeof(addrun);
     9+    socklen_t len = sizeof(addrun);
    10 
    11     if(!ConnectToSocket(*sock, (struct sockaddr*)&addrun, len, path, /*manual_connection=*/true)) {
    12         LogPrintf("Cannot connect to socket for %s\n", path);
    
  113. in src/init.cpp:495 in f74da5eba9 outdated
    491@@ -492,7 +492,7 @@ void SetupServerArgs(ArgsManager& argsman)
    492     argsman.AddArg("-maxsendbuffer=<n>", strprintf("Maximum per-connection send buffer, <n>*1000 bytes (default: %u)", DEFAULT_MAXSENDBUFFER), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
    493     argsman.AddArg("-maxtimeadjustment", strprintf("Maximum allowed median peer time offset adjustment. Local perspective of time may be influenced by outbound peers forward or backward by this amount (default: %u seconds).", DEFAULT_MAX_TIME_ADJUSTMENT), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
    494     argsman.AddArg("-maxuploadtarget=<n>", strprintf("Tries to keep outbound traffic under the given target per 24h. Limit does not apply to peers with 'download' permission or blocks created within past week. 0 = no limit (default: %s). Optional suffix units [k|K|m|M|g|G|t|T] (default: M). Lowercase is 1000 base while uppercase is 1024 base", DEFAULT_MAX_UPLOAD_TARGET), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
    495-    argsman.AddArg("-onion=<ip:port>", "Use separate SOCKS5 proxy to reach peers via Tor onion services, set -noonion to disable (default: -proxy)", ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
    496+    argsman.AddArg("-onion=<ip:port|path>", "Use separate SOCKS5 proxy to reach peers via Tor onion services, set -noonion to disable (default: -proxy). May be a local file path prefixed with 'unix:' if UNIX domain sockets are supported by the platform.", ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
    


    vasild commented at 2:19 pm on July 19, 2023:

    We have already checked if UNIX sockets are supported.

    0#if HAVE_SOCKADDR_UN
    1    argsman.AddArg("-onion=<ip:port|path>", "Use separate SOCKS5 proxy to reach peers via Tor onion services, set -noonion to disable (default: -proxy). May be a local file path prefixed with 'unix:'.", ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
    2#else
    3    argsman.AddArg("-onion=<ip:port>", "Use separate SOCKS5 proxy to reach peers via Tor onion services, set -noonion to disable (default: -proxy)", ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
    4#endif
    

    pinheadmz commented at 4:32 pm on July 19, 2023:
    ah of course thanks, fixed and pushed
  114. pinheadmz force-pushed on Jul 19, 2023
  115. achow101 requested review from willcl-ark on Sep 20, 2023
  116. achow101 requested review from vasild on Sep 20, 2023
  117. DrahtBot added the label Needs rebase on Oct 3, 2023
  118. in src/netbase.cpp:541 in d8f993b77d outdated
    551-        return false;
    552-    }
    553-
    554-    // Connect to the addrConnect service on the hSocket socket.
    555-    if (sock.Connect(reinterpret_cast<struct sockaddr*>(&sockaddr), len) == SOCKET_ERROR) {
    556+    // Connect to the dest service on the hSocket socket.
    


    vasild commented at 12:44 pm on October 13, 2023:

    nit:

    0    // Connect to `sockaddr` using `sock`.
    

    or drop the comment altogether.

  119. in src/init.cpp:510 in d8f993b77d outdated
    505@@ -502,7 +506,11 @@ void SetupServerArgs(ArgsManager& argsman)
    506     // TODO: remove the sentence "Nodes not using ... incoming connections." once the changes from
    507     // https://github.com/bitcoin/bitcoin/pull/23542 have become widespread.
    508     argsman.AddArg("-port=<port>", strprintf("Listen for connections on <port>. Nodes not using the default ports (default: %u, testnet: %u, signet: %u, regtest: %u) are unlikely to get incoming connections. Not relevant for I2P (see doc/i2p.md).", defaultChainParams->GetDefaultPort(), testnetChainParams->GetDefaultPort(), signetChainParams->GetDefaultPort(), regtestChainParams->GetDefaultPort()), ArgsManager::ALLOW_ANY | ArgsManager::NETWORK_ONLY, OptionsCategory::CONNECTION);
    509+#if HAVE_SOCKADDR_UN
    510+    argsman.AddArg("-proxy=<ip:port|path>", "Connect through SOCKS5 proxy, set -noproxy to disable (default: disabled). May be a local file path prefixed with 'unix:' if UNIX domain sockets are supported by the platform.", ArgsManager::ALLOW_ANY | ArgsManager::DISALLOW_ELISION, OptionsCategory::CONNECTION);
    


    vasild commented at 1:08 pm on October 13, 2023:

    nit:

    0    argsman.AddArg("-proxy=<ip:port|path>", "Connect through SOCKS5 proxy, set -noproxy to disable (default: disabled). May be a local file path prefixed with 'unix:'.", ArgsManager::ALLOW_ANY | ArgsManager::DISALLOW_ELISION, OptionsCategory::CONNECTION);
    

    (this is inside #if HAVE_SOCKADDR_UN)

  120. vasild approved
  121. vasild commented at 2:44 pm on October 13, 2023: contributor

    ACK d8f993b77d2f0bad0e7a5659885ffbbd9f75dbec (needs rebase)

    I tested this and it works. Got one unexpected failure to connect (it stalled for more than a minute after the first 4 lines):

     02023-10-13T13:36:50Z [net:debug] trying connection <strip>.onion:8333 lastseen=0.0hrs
     12023-10-13T13:36:50Z [proxy:debug] Using proxy: unix:/var/run/tor/socks to connect to <strip>.onion:8333
     22023-10-13T13:36:50Z [net] SOCKS5 connecting <strip>.onion
     32023-10-13T13:36:50Z [proxy] SOCKS5 sending proxy authentication 0:0
     42023-10-13T13:38:11Z [net:debug] trying connection <strip>.onion:8333 lastseen=0.0hrs
     52023-10-13T13:38:11Z [proxy:debug] Using proxy: unix:/var/run/tor/socks to connect to <strip>.onion:8333
     62023-10-13T13:38:11Z [net] SOCKS5 connecting <strip>.onion
     72023-10-13T13:38:11Z [proxy] SOCKS5 sending proxy authentication 1:1
     82023-10-13T13:38:11Z [net] SOCKS5 connected <strip>.onion
     92023-10-13T13:38:11Z [net] Added connection to <strip>.onion:8333 peer=0
    102023-10-13T13:38:11Z [net] sending version (103 bytes) peer=0
    

    It could be some intermediate problem from the Tor network. The address it managed to connect later is the same.

    I have submitted the discussed Sock5() improvement here: https://github.com/bitcoin/bitcoin/pull/28649

  122. pinheadmz force-pushed on Oct 16, 2023
  123. pinheadmz commented at 2:36 pm on October 16, 2023: member
    @vasild thanks again for your review! Nits addressed and rebased on master. I will help you review https://github.com/bitcoin/bitcoin/pull/28649
  124. DrahtBot removed the label Needs rebase on Oct 16, 2023
  125. vasild approved
  126. vasild commented at 8:26 am on October 17, 2023: contributor
    ACK e60df927bffa844d08fa908128b10303b352cfc5
  127. in src/netbase.cpp:644 in 7239ec264a outdated
    643 {
    644     // first connect to proxy server
    645-    if (!ConnectSocketDirectly(proxy.proxy, sock, nTimeout, true)) {
    646-        outProxyConnectionFailed = true;
    647-        return false;
    648+    auto sock = ConnectDirectly(proxy.proxy, /*manual_connection=*/true);
    


    willcl-ark commented at 9:57 am on October 17, 2023:

    In 7239ec264a1def8e8aeff8c1ca57ece2ea90f65c

    Why do we always set manual_connection = true here (I appreciate it was existing code)? If we come via ConnectThroughProxy I don’t think that’s always the case, even though the docstring implies it should be.

    edit: it doesn’t seem to have much effect anyway, besides logging a different error message on failure.


    Sjors commented at 2:32 pm on February 28, 2024:
    Both call sites of ConnectThroughProxy have access to conn_type so might as well pass in manual_connection.
  128. in src/netbase.cpp:230 in 892e0f4fae outdated
    219+    std::string str{name.substr(ADDR_PREFIX_UNIX.length())};
    220+
    221+    // Path size limit is platform-dependent
    222+    // see https://manpages.ubuntu.com/manpages/xenial/en/man7/unix.7.html
    223+    if (str.size() + 1 > sizeof(((sockaddr_un*)nullptr)->sun_path)) return false;
    224+
    


    willcl-ark commented at 10:26 am on October 17, 2023:

    nit: if you touch again we could also check here whether the path exists and return an appropriate error if it doesn’t?

    I tried some (wrong) unix onion options, like onion=unix:%2Frun%2Ftor%2Fsocks and onion=127.0.0.1:9050 and these are accepted by init, but (obviously) fail to connect:

    02023-10-17T10:25:01Z connect() to 127.0.0.1:9050 failed: No such file or directory (2)
    12023-10-17T10:25:01Z Cannot connect to socket for 127.0.0.1:9050
    22023-10-17T10:25:02Z New block-relay-only v1 peer connected: version: 70016, blocks=812593, peer=2
    32023-10-17T10:25:07Z Cannot connect to socket for 117.86.73.230:8333
    42023-10-17T10:25:08Z Cannot connect to socket for 203.91.85.237:8333
    52023-10-17T10:25:08Z connect() to 127.0.0.1:9050 failed: No such file or directory (2)
    62023-10-17T10:25:08Z Cannot connect to socket for 127.0.0.1:9050
    72023-10-17T10:25:14Z Cannot connect to socket for 178.66.156.78:8333
    

    The logs are kind of clear about this, but could be improved.


    pinheadmz commented at 5:34 pm on October 17, 2023:

    This was discussed but we don’t normally check this kind of option that deeply. For example, you can set -onion=127.0.0.1:9050 and if Tor isn’t running, all you get is a log message. The user should be able to start Tor after Bitcoin.

    #27375 (review)

  129. willcl-ark approved
  130. willcl-ark commented at 11:02 am on October 17, 2023: contributor

    ACK e60df927bffa844d08fa908128b10303b352cfc5

    Reviewed the changes since last time, and tested both options again pretty thoroughly. Left a few nits which don’t need to be addressed unless re-touching.

    I wonder if we should update doc/tor.md in a followup? We currently describe options needed for ControlPort in torrc, and could add the directive for unix sockets (and explain permissions needed to access the socket, or include the WorldWritable arg):

    0SocksPort /run/tor/socks
    1# or
    2SocksPort /run/tor/socks WorldWritable
    

    …along with benefits/differences of using unix sockets for onion/proxy too? (easier access control to socket via filesystem vs network namespace, marginal speed gain, etc.) Other reasons discussed in https://riseup.net/en/security/network-security/tor/onionservices-best-practices#protecting-your-services

  131. in configure.ac:1281 in e60df927bf outdated
    1277+    #include <sys/un.h>
    1278+  ]], [[
    1279+    struct sockaddr_un addr;
    1280+    addr.sun_family = AF_UNIX;
    1281+  ]])],
    1282+ [ AC_MSG_RESULT([yes]); HAVE_SOCKADDR_UN=1; AC_DEFINE([HAVE_SOCKADDR_UN], [1], [Define this symbol if platform supports unix domain sockets]) ],
    


    fanquake commented at 2:27 pm on October 19, 2023:
    There shouldn’t be a need for the HAVE_SOCKADDR_UN=1 or HAVE_SOCKADDR_UN=0 in the AC_MSG_RESULT’s?
  132. pinheadmz force-pushed on Oct 19, 2023
  133. DrahtBot added the label Needs rebase on Oct 19, 2023
  134. vasild approved
  135. vasild commented at 11:05 am on October 23, 2023: contributor
    ACK b46e24ccc19d8a7dc40d0b5e993f245ede3c7086 (needs rebase)
  136. DrahtBot requested review from willcl-ark on Oct 23, 2023
  137. pinheadmz force-pushed on Oct 24, 2023
  138. pinheadmz commented at 5:46 pm on October 24, 2023: member
    Thanks again @vasild rebased on master - I see some new I2P tests in there now! Had to update those with the changes to CreateSock()
  139. DrahtBot removed the label Needs rebase on Oct 24, 2023
  140. pinheadmz force-pushed on Oct 24, 2023
  141. DrahtBot added the label CI failed on Oct 25, 2023
  142. DrahtBot removed the label CI failed on Oct 25, 2023
  143. vasild approved
  144. vasild commented at 3:16 pm on October 30, 2023: contributor
    ACK b6da3897f36547139b83abde486bddbf50dab54e
  145. DrahtBot added the label CI failed on Nov 2, 2023
  146. maflcko commented at 7:35 am on November 4, 2023: member
    0test/i2p_tests.cpp:162:79: error: no matching function for call to i2p::sam::Session::Session(const fs::path&, CService, CThreadInterrupt*)
    
  147. pinheadmz force-pushed on Nov 5, 2023
  148. pinheadmz commented at 7:36 pm on November 5, 2023: member
    Rebased on master to fix silent conflict with new I2P tests. @maflcko thanks. Can you help me understand why this only broke the previous builds task?
  149. DrahtBot removed the label CI failed on Nov 5, 2023
  150. maflcko commented at 12:33 pm on November 6, 2023: member
    To preserve CI resources, only one task is re-run to detect silent merge conflicts.
  151. in src/net.cpp:486 in 0c5f8c36c9 outdated
    482-            if (!sock) {
    483-                return nullptr;
    484-            }
    485-            connected = ConnectThroughProxy(proxy, addrConnect.ToStringAddr(), addrConnect.GetPort(),
    486-                                            *sock, nConnectTimeout, proxyConnectionFailed);
    487+            LogPrintLevel(BCLog::PROXY, BCLog::Level::Debug, "Using proxy: %s to connect to %s:%s\n", proxy.ToString(), addrConnect.ToStringAddr(), addrConnect.GetPort());
    


    willcl-ark commented at 10:06 pm on November 6, 2023:

    nit: in c079e29c266a413de1859c0cbc8ebb55ea6a3203

    I get a compilation failure:

     0net.cpp:481:109: error: no member named 'ToString' in 'Proxy'
     1            LogPrintLevel(BCLog::PROXY, BCLog::Level::Debug, "Using proxy: %s to connect to %s:%s\n", proxy.ToString(), addrConnect.ToStringAddr(), addrConnect.GetPort());
     2                                                                                                      ~~~~~ ^
     3./logging.h:257:45: note: expanded from macro 'LogPrintLevel'
     4            LogPrintLevel_(category, level, __VA_ARGS__); \
     5                                            ^~~~~~~~~~~
     6./logging.h:234:104: note: expanded from macro 'LogPrintLevel_'
     7#define LogPrintLevel_(category, level, ...) LogPrintf_(__func__, __FILE__, __LINE__, category, level, __VA_ARGS__)
     8                                                                                                       ^~~~~~~~~~~
     91 error generated.
    10make: *** [Makefile:10385: libbitcoin_node_a-net.o] Error 1
    11make: Leaving directory '/home/will/src/bitcoin/src'
    

    Looks like ToString() is added to Proxy in e7d29d5


    pinheadmz commented at 5:00 pm on November 8, 2023:
    great catch thanks, will fix.
  152. in src/netbase.h:284 in 0c5f8c36c9 outdated
    295  *
    296- * @returns Whether or not a connection was successfully made.
    297+ * @returns the connected socket if the operation succeeded, empty unique_ptr otherwise
    298  */
    299-bool ConnectSocketDirectly(const CService &addrConnect, const Sock& sock, int nTimeout, bool manual_connection);
    300+std::unique_ptr<Sock> ConnectDirectly(const CService& dest, bool manual_connection);
    


    willcl-ark commented at 12:43 pm on November 7, 2023:

    In c079e29

    ConnectDirectly and ConnectThroughProxy now differ in their primary args, with Directly taking a CService and Proxy remaining with std::string& (for host and port) which results in:

    src/net.cpp

    0        } else if (use_proxy) {
    1            LogPrintLevel(BCLog::PROXY, BCLog::Level::Debug, "Using proxy: %s to connect to %s:%s\n", proxy.proxy.ToStringAddrPort(), addrConnect.ToStringAddr(), addrConnect.GetPort());
    2            sock = ConnectThroughProxy(proxy, addrConnect.ToStringAddr(), addrConnect.GetPort(), proxyConnectionFailed);
    3        } else {
    4            // no proxy needed (none set for target network)
    5            sock = ConnectDirectly(addrConnect, conn_type == ConnectionType::MANUAL);
    6        }
    

    If you touch again we could take the chance to synchronise the interfaces for these two similar functions?


    pinheadmz commented at 5:00 pm on November 8, 2023:
    Good observation. This is how ConnectSocketDirectly() and ConnectThroughProxy() are implemented on master currently as well. I think the reason is that we actually send the SOCKS5 proxy the destination host as a string whereas for the direct connection we actually get a socket from the OS. So even if throughProxy accepted a CService, we’d have to split into host string and port right away, anyway. I think I’ll leave this alone for now unless other reviewers want to weigh in.
  153. willcl-ark approved
  154. willcl-ark commented at 2:23 pm on November 7, 2023: contributor

    tACK 0c5f8c36c90fab4fb39ce39d1b4c204a2edd25c4

    Left a few nits which IMO don’t need addressing. Tested out new socket type for proxy and onion and found it working as expected.

  155. DrahtBot requested review from vasild on Nov 7, 2023
  156. achow101 referenced this in commit 0528cfd307 on Nov 7, 2023
  157. pinheadmz force-pushed on Nov 8, 2023
  158. pinheadmz commented at 5:56 pm on November 8, 2023: member
    force pushed to clean up git commit history. diff is null
  159. vasild approved
  160. vasild commented at 9:49 am on November 14, 2023: contributor
    ACK 6dddc1c2eda514d35219cce03c229bd6f822be55
  161. DrahtBot requested review from willcl-ark on Nov 14, 2023
  162. DrahtBot added the label CI failed on Jan 13, 2024
  163. tdb3 commented at 8:48 pm on February 12, 2024: contributor

    ACK for 6dddc1c2eda514d35219cce03c229bd6f822be55. Great work on a great feature. Seems to work well. Reviewed the code changes, but didn’t see anything noteworthy over what vasild, luke-jr, and willcl-ark already saw.

    Ran the following test actions:

    • Cloned and checked out PR branch, compiled, and ran all functional tests (all passed)
    • Configured Tor to listen on only the unix domain socket /run/tor/socks (SocksPort 9050 and ControlPort 9051 disabled)
    • Started bitcoind (signet), proxy=/run/tor/socks set, onion and onlynet not set. Connections were established with multiple peers and successful block download was observed. Sniffed the unix socket (/run/tor/socks) with https://github.com/mechpen/sockdump, and observed bitcoin traffic in Wireshark with a pcap dump from sockdump. getpeerinfo command to the node showed ipv4, ipv6, and onion addr peers.
    • Started bitcoind (signet), onion=/run/tor/socks set, proxy not set, but onlynet=onion set. Manually added seed v7ajjeirttkbnt32wpy3c6w3emwnfr3fkla7hpxcfokr3ysd3kqtzmqd.onion:38333 with addnode command. Connections were established with multiple peers and successful block download was observed. Sniffed the unix socket and again observed bitcoin traffic over the socket. getpeerinfo showed multiple onion addr peers.
  164. Sjors commented at 11:41 am on February 28, 2024: member

    Would be nice to have this for RPC too, but can wait for a followup.

    Ditto for direct peer connections.

  165. in src/netaddress.h:547 in 4ca9b322f0 outdated
    543@@ -544,6 +544,7 @@ class CService : public CNetAddr
    544     uint16_t GetPort() const;
    545     bool GetSockAddr(struct sockaddr* paddr, socklen_t* addrlen) const;
    546     bool SetSockAddr(const struct sockaddr* paddr);
    547+    [[nodiscard]] sa_family_t GetSAFamily() const;
    


    Sjors commented at 12:07 pm on February 28, 2024:

    4ca9b322f00db56a390301f08d3de8a13ff66694: suggested documentation:

    0    /**
    1     * Get the address family
    2     * [@returns](/bitcoin-bitcoin/contributor/returns/) AF_UNSPEC if unspecified
    3     */
    
  166. in src/netbase.h:269 in 4ca9b322f0 outdated
    235@@ -236,12 +236,12 @@ CSubNet LookupSubNet(const std::string& subnet_str);
    236  * @param[in] address_family The socket is created in the same address family as this address.
    237  * @return pointer to the created Sock object or unique_ptr that owns nothing in case of failure
    238  */
    239-std::unique_ptr<Sock> CreateSockTCP(const CService& address_family);
    240+std::unique_ptr<Sock> CreateSockOS(sa_family_t address_family);
    


    Sjors commented at 12:12 pm on February 28, 2024:

    4ca9b322f00db56a390301f08d3de8a13ff66694: “Create a TCP socket” should be changed to create “Create a socket”?

    And:

    0* [@param](/bitcoin-bitcoin/contributor/param/)[in] address_family to use for the socket.
    
  167. in src/i2p.cpp:329 in 4ca9b322f0 outdated
    325@@ -326,7 +326,7 @@ Session::Reply Session::SendRequestAndGetReply(const Sock& sock,
    326 
    327 std::unique_ptr<Sock> Session::Hello() const
    328 {
    329-    auto sock = CreateSock(m_control_host);
    330+    auto sock = CreateSock(m_control_host.GetSAFamily());
    


    Sjors commented at 12:24 pm on February 28, 2024:
    4ca9b322f00db56a390301f08d3de8a13ff66694: this is a nice readability improvement; makes it clear CreateSock only needs the socket address family and the real work happens in ConnectSocketDirectly.
  168. in src/netbase.cpp:464 in d513ac7bf6 outdated
    462 
    463-    // Create a TCP socket in the address family of the specified service.
    464-    SOCKET hSocket = socket(address_family, SOCK_STREAM, IPPROTO_TCP);
    465+    int protocol{IPPROTO_TCP};
    466+#if HAVE_SOCKADDR_UN
    467+    if (address_family == AF_UNIX)
    


    Sjors commented at 12:38 pm on February 28, 2024:
    d513ac7bf615239053762a4f9d64db9f4c70d12a nit: brackets or on the same line
  169. in src/netbase.cpp:484 in d513ac7bf6 outdated
    467+    if (address_family == AF_UNIX)
    468+        protocol = 0;
    469+#endif
    470+
    471+    // Create a socket in the specified address family.
    472+    SOCKET hSocket = socket(address_family, SOCK_STREAM, protocol);
    


    Sjors commented at 12:46 pm on February 28, 2024:

    d513ac7bf615239053762a4f9d64db9f4c70d12a: On macOS man 2 socket lists PF_LOCAL (alias for PF_UNIX), PF_INET and PF_INET6 for the domain parameter of socket (first argument).

    They map to the same integer values as AF_UNIX, AF_INET and AF_INET6, so it’s just a cosmetic thing. And apparently nobody cares: https://stackoverflow.com/a/6737450/313633


    pinheadmz commented at 9:40 pm on February 29, 2024:
    That’s… weird 🤷 Do you think I need to change anything here?

    vasild commented at 7:56 am on March 1, 2024:
    On FreeBSD all of AF_UNIX, AF_LOCAL, PF_UNIX, PF_LOCAL have the same value: https://cgit.freebsd.org/src/tree/sys/sys/socket.h

    Sjors commented at 11:25 am on March 1, 2024:
    I think if someone cares, they can make a followup PR.
  170. in src/init.cpp:513 in 6dddc1c2ed outdated
    508@@ -505,7 +509,11 @@ void SetupServerArgs(ArgsManager& argsman)
    509     // TODO: remove the sentence "Nodes not using ... incoming connections." once the changes from
    510     // https://github.com/bitcoin/bitcoin/pull/23542 have become widespread.
    511     argsman.AddArg("-port=<port>", strprintf("Listen for connections on <port>. Nodes not using the default ports (default: %u, testnet: %u, signet: %u, regtest: %u) are unlikely to get incoming connections. Not relevant for I2P (see doc/i2p.md).", defaultChainParams->GetDefaultPort(), testnetChainParams->GetDefaultPort(), signetChainParams->GetDefaultPort(), regtestChainParams->GetDefaultPort()), ArgsManager::ALLOW_ANY | ArgsManager::NETWORK_ONLY, OptionsCategory::CONNECTION);
    512+#if HAVE_SOCKADDR_UN
    513+    argsman.AddArg("-proxy=<ip:port|path>", "Connect through SOCKS5 proxy, set -noproxy to disable (default: disabled). May be a local file path prefixed with 'unix:'.", ArgsManager::ALLOW_ANY | ArgsManager::DISALLOW_ELISION, OptionsCategory::CONNECTION);
    


    Sjors commented at 1:54 pm on February 28, 2024:

    6dddc1c2eda514d35219cce03c229bd6f822be55 the SOCKS5 standard seems to assume that clients are connecting via either TCP or UDP. Other than Tor, are there other SOCKS5 proxies out there that support connecting via socket? https://datatracker.ietf.org/doc/html/rfc1928

    (it’s probably fine to allow it even if not)


    pinheadmz commented at 10:02 pm on February 29, 2024:
    Hm good question, I found some vague docs that imply NordVPN client has a unix socket option… Would you suggest specifying Tor in the help description of the proxy arg?

    Sjors commented at 11:24 am on March 1, 2024:
    No I think that would be confusing. Perhaps add something something like “unix: if the proxy supports it” (assuming it’s not standard)
  171. in src/netbase.cpp:232 in 8d06446fc7 outdated
    223+    // Path size limit is platform-dependent
    224+    // see https://manpages.ubuntu.com/manpages/xenial/en/man7/unix.7.html
    225+    if (str.size() + 1 > sizeof(((sockaddr_un*)nullptr)->sun_path)) return false;
    226+
    227+    return true;
    228+#else
    


    Sjors commented at 2:43 pm on February 28, 2024:
    8d06446fc7a17b593466fccd0f638eb2527a493b: maybe add Assume(false); here? Since the answer of this function is unreliable for systems that don’t support unix sockets.

    pinheadmz commented at 9:45 pm on February 29, 2024:
    Hm not sure what you mean by unreliable. Wont the pre-processor directives here guaruntee this function always returns false on systems that don’t support unix sockets?

    Sjors commented at 11:21 am on March 1, 2024:
    It would, but “unix:/somepath” is valid unix socket path, so IsUnixSocketPath should return true. So it seems better to ensure nobody accidentally calls this function on a system that doesn’t support unix sockets.

    pinheadmz commented at 5:46 pm on March 1, 2024:

    is valid unix socket path,

    This can actually only be determined if sockaddr_un is defined, because different platforms have different length limits for the path (https://manpages.ubuntu.com/manpages/xenial/en/man7/unix.7.html)

    So if sockaddr_un is not defined, nothing can be a valid unix socket path.

    Also I think Assume(false) is overkill because this function will be called from AppInitMain() regardless of platform. If a user without unix socket support still tries to set -onion=unix:/whatever they’ll get an init error

  172. in src/netbase.cpp:639 in 9eb40e6f39 outdated
    639+
    640+    struct sockaddr_un addrun;
    641+    memset(&addrun, 0, sizeof(addrun));
    642+    addrun.sun_family = AF_UNIX;
    643+    // leave the last char in addrun.sun_path[] to be always '\0'
    644+    memcpy(addrun.sun_path, path.c_str(), std::min(sizeof(addrun.sun_path) - 1, path.length()));
    


    Sjors commented at 4:28 pm on February 28, 2024:
    9eb40e6f399169ade8e96c60ba633e8a0e4ddadc: Given the IsValid() check above, you could also do Assume(path.length() < sizeof(addrun.sun_path) - 1) and leave out the std::min stuff, but maybe this is more robust.

    pinheadmz commented at 10:03 pm on February 29, 2024:
    Yeah I think this was recommended by vasild in an earlier interation of the branch.
  173. Sjors approved
  174. Sjors commented at 4:55 pm on February 28, 2024: member

    utACK 6dddc1c2eda514d35219cce03c229bd6f822be55

    Mostly cosmetic feedback.

  175. pinheadmz commented at 10:13 pm on February 29, 2024: member

    @Sjors #27375 (review)

    Both call sites of ConnectThroughProxy have access to conn_type so might as well pass in manual_connection.

    Do you mean to pass it ConnectThroughProxy() -> proxy.Connect() -> ConnectDirectly()?

    That would also mean we need manual_connection in i2p.cpp where we call proxy.Connect() ?

  176. Sjors commented at 11:28 am on March 1, 2024: member

    That would also mean we need manual_connection in i2p.cpp where we call proxy.Connect() ? @vasild does the i2p code know if a connection is manual? We can always set manual_connection to true if we don’t know (or it adds too much complexity), but we should set it to the correct value if we do know.

  177. pinheadmz commented at 6:04 pm on March 1, 2024: member

    @Sjors If I’m following this correctly, the only benefit of passing manual_connection to ConnectDirectly() is that it is then passed to ConnectToSocket() which only uses the boolean when logging with LogConnectFailure().

    On current master, ConnectThroughProxy() calls ConnectSocketDirectly() with a hard-coded true for manual_connection. I believe the motivation is in #12569

    So I’m inclined to leave this hard-coded as true because IIUC the connection being described by the boolean in this context isn’t “manual” in the “p2p addrman vs addnode” sense, but is used to refer to the connection between bitcoind and the input to the proxy.

  178. configure: test for unix domain sockets
    Copied from https://github.com/bitcoin/bitcoin/pull/9979
    
    Co-authored-by: laanwj <126646+laanwj@users.noreply.github.com>
    adb3a3e51d
  179. netbase: refactor CreateSock() to accept sa_family_t
    Also implement CService::GetSAFamily() to provide sa_family_t
    bae86c8d31
  180. netbase: allow CreateSock() to create UNIX sockets if supported 74f568cb6f
  181. net: move CreateSock() calls from ConnectNode() to netbase methods 3a7d6548ef
  182. netbase: extend Proxy class to wrap UNIX socket as well as TCP a89c3f59dc
  183. proxy: rename randomize_credentials to m_randomize_credentials ac2ecf3182
  184. net: split ConnectToSocket() from ConnectDirectly() for unix sockets d9318a37ec
  185. i2p: construct Session with Proxy instead of CService a88bf9dedd
  186. gui: accomodate unix socket Proxy in updateDefaultProxyNets()
    This will require a follow-up to add unix socket options to the GUI
    c3bd43142e
  187. init: allow UNIX socket path for -proxy and -onion c65c0d0163
  188. test: cover UNIX sockets in feature_proxy.py bfe5192891
  189. doc: add release notes and help text for unix sockets 567cec9a05
  190. pinheadmz force-pushed on Mar 1, 2024
  191. pinheadmz commented at 7:51 pm on March 1, 2024: member

    git range-diff master 6dddc1c2eda514d35219cce03c229bd6f822be55 567cec9a05e1261e955535f734826b12341684b6

    Rebased on master to fix conflicts and ( 🤞 ) pass CI. @Sjors thank you so much for the review, I addressed all your nits except the one or two where indicated in discussion.

  192. DrahtBot removed the label CI failed on Mar 1, 2024
  193. Sjors commented at 11:09 am on March 4, 2024: member

    but is used to refer to the connection between bitcoind and the input to the proxy

    I agree that shouldn’t be called “manual”.

    re-ACK 567cec9a05e1261e955535f734826b12341684b6

  194. DrahtBot requested review from tdb3 on Mar 4, 2024
  195. DrahtBot requested review from vasild on Mar 4, 2024
  196. vasild approved
  197. vasild commented at 8:38 am on March 7, 2024: contributor

    ACK 567cec9a05e1261e955535f734826b12341684b6

    I agree it is better to keep the hardcoded /*manual=*/true because in this case that is about connecting to the proxy itself, not to the bitcoin peer. In this case it is better to log with higher severity.

    In the future maybe it would be better to rename the manual_connection argument of ConnectToSocket() to log_severity or error_severity or is_imporant_please_log_errorswithhigherseverity.

  198. DrahtBot removed review request from tdb3 on Mar 7, 2024
  199. DrahtBot requested review from tdb3 on Mar 7, 2024
  200. tdb3 commented at 8:43 pm on March 8, 2024: contributor

    re ACK for 567cec9a05e1261e955535f734826b12341684b6.

    Pulled, built, ran all unit and functional tests (all passed).

    Re-ran the test actions (quote below) from #27375#pullrequestreview-1876248880

    Overall, similar results were observed. Each test action was performed with a fresh signet directory (with the exception of blocks and chainstate, which were not cleared, to prevent IBD). For the first action (i.e. running with proxy=unix:/run/tor/socks set, but onion and onlynet not set), peer connection was observed to grow over the span of a half our (with ipv4, ipv6, and onion peers connected). debug log showed general socks connection failures (although these also occur in v26.0 when using localhost:9050 for socks):

    0<date> Socks5() connect to <IPv4 ADDR>:38333 failed: general failure
    

    For the second action (i.e. running with proxy and onlynet=onion with manual seed added), onion peer connections were observed.

    For both actions, UNIX socket traffic was sniffed and contained bitcoin protocol traffic.

    ACK for 6dddc1c. Great work on a great feature. Seems to work well. Reviewed the code changes, but didn’t see anything noteworthy over what vasild, luke-jr, and willcl-ark already saw.

    Ran the following test actions:

    * Cloned and checked out PR branch, compiled, and ran all functional tests (all passed)
    
    * Configured Tor to listen on only the unix domain socket /run/tor/socks (SocksPort 9050 and ControlPort 9051 disabled)
    
    * Started bitcoind (signet), `proxy=/run/tor/socks` set, `onion` and `onlynet` not set.  Connections were established with multiple peers and successful block download was observed.  Sniffed the unix socket (`/run/tor/socks`) with https://github.com/mechpen/sockdump, and observed bitcoin traffic in Wireshark with a pcap dump from sockdump.  `getpeerinfo` command to the node showed ipv4, ipv6, and onion addr peers.
    
    * Started bitcoind (signet), `onion=/run/tor/socks` set, `proxy` not set, but `onlynet=onion` set.  Manually added seed `v7ajjeirttkbnt32wpy3c6w3emwnfr3fkla7hpxcfokr3ysd3kqtzmqd.onion:38333` with `addnode` command.  Connections were established with multiple peers and successful block download was observed.  Sniffed the unix socket and again observed bitcoin traffic over the socket.  `getpeerinfo` showed multiple onion addr peers.
    
  201. DrahtBot removed review request from tdb3 on Mar 8, 2024
  202. achow101 commented at 5:55 pm on March 12, 2024: member
    ACK 567cec9a05e1261e955535f734826b12341684b6
  203. achow101 commented at 6:13 pm on March 12, 2024: member

    Seems like there’s a silent merge conflict somewhere, getting a functional test failure when merged with master:

    02024-03-12T18:12:19.695000Z TestFramework (ERROR): Assertion failed
    1Traceback (most recent call last):
    2  File "/home/ava/bitcoin/bitcoin/master/test/functional/test_framework/test_framework.py", line 565, in start_nodes
    3    node.wait_for_rpc_connection()
    4  File "/home/ava/bitcoin/bitcoin/master/test/functional/test_framework/test_node.py", line 255, in wait_for_rpc_connection
    5    raise FailedToStartError(self._node_msg(
    6test_framework.test_node.FailedToStartError: [node 5] bitcoind exited with status 1 during initialization. Error: Invalid -proxy address or hostname: 'unix:/tmp/tmp9eh_pwt0'
    
  204. in src/netbase.cpp:8 in 567cec9a05
    2@@ -3,6 +3,10 @@
    3 // Distributed under the MIT software license, see the accompanying
    4 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
    5 
    6+#if defined(HAVE_CONFIG_H)
    7+#include <config/bitcoin-config.h>
    8+#endif
    


    pinheadmz commented at 3:50 am on March 13, 2024:

    @achow101 Its very mysterious, when I rebase this branch on master (git rebase master) these lines disappear. I can’t figure out why!

    However when I do interactive git rebase master -i and stop on this commit with an edit command, DO NOTHING, and just git rebase --continue… the lines remain intact. I am force-pushing this interactive rebase branch now. :face_with_spiral_eyes:


    maflcko commented at 7:41 am on March 13, 2024:

    when I rebase this branch on master (git rebase master) these lines disappear. I can’t figure out why!

    What are the exact steps to reproduce?

    I did

    0git checkout  567cec9a05e1261e955535f734826b12341684b6
    1git rebase bitcoin-core/master
    2git range-diff bitcoin-core/master 6df896fc37f5b2023b891d5300345e2a4352e521
    

    And got an empty diff (only the commit ids are different)


    maflcko commented at 7:44 am on March 13, 2024:
    Maybe you two are using a different git --version (with a bug)?
  205. pinheadmz force-pushed on Mar 13, 2024
  206. maflcko commented at 7:39 am on March 13, 2024: member

    Seems like there’s a silent merge conflict somewhere, getting a functional test failure when merged with master:

    I re-ran the CI and it didn’t fail. Are you sure this is a silent conflict, because if it was, the CI should be detecting it on a re-run?

  207. achow101 commented at 10:33 am on March 13, 2024: member
    Oh! The test command I use doesn’t rerun autogen.sh every time, I think that’s what caused the test failure. @pinheadmz You can reset the branch back to 567cec9a05e and this can be merged. Sorry about the confusion.
  208. pinheadmz force-pushed on Mar 13, 2024
  209. pinheadmz commented at 10:35 am on March 13, 2024: member

    Oh! The test command I use doesn’t rerun autogen.sh every time, I think that’s what caused the test failure. @pinheadmz You can reset the branch back to 567cec9 and this can be merged. Sorry about the confusion.

    Ugh and when I was trying to reproduce I ALSO didn’t rerun autogen or config :facepalm: wow THANKS @maflcko for the sanity check

  210. achow101 merged this on Mar 13, 2024
  211. achow101 closed this on Mar 13, 2024

  212. vasild commented at 12:19 pm on March 13, 2024: contributor

    autogen

    wen cmake?

  213. fanquake commented at 12:27 pm on March 13, 2024: member
    Is the expectation here that we should see a lot more Cannot connect to socket for xxxxx in debug.logs? (when not using any of these new features).
  214. pinheadmz commented at 2:11 pm on March 13, 2024: member

    Is the expectation here that we should see a lot more Cannot connect to socket for xxxxx in debug.logs? (when not using any of these new features).

    Yes but that log message can be removed or set to debug in a follow-up. With -debug=net you can see what the original messages were for that code path, with the extra message following:

    02024-03-13T14:08:49Z [net] connection attempt to 95.165.161.198:8333 timed out
    12024-03-13T14:08:49Z Cannot connect to socket for 95.165.161.198:8333
    2...
    32024-03-13T14:08:50Z [net] connect() to 31.150.240.73:8333 failed after wait: Connection refused (61)
    42024-03-13T14:08:50Z Cannot connect to socket for 31.150.240.73:8333
    

    I have a few unix sockets follow ups planned: RPC & bitcoin-cli (maybe speed up the tests!) and ZMQ. I’ll clean up this log message in those

  215. fanquake commented at 2:13 pm on March 13, 2024: member

    Ok, it needs to be cleaned up because it’s just spammy/pointless. i.e:

     02024-03-13T13:26:00Z Cannot connect to socket for 151.213.177.128:8333
     12024-03-13T13:26:24Z Cannot connect to socket for 75.88.65.59:8333
     22024-03-13T13:26:48Z Cannot connect to socket for 76.114.61.112:8333
     32024-03-13T13:28:55Z Cannot connect to socket for [2600:8805:5c26:eb00:61b0:c6a7:451:1b39]:8333
     42024-03-13T13:30:45Z New block-relay-only v1 peer connected: version: 70016, blocks=834513, peer=39
     52024-03-13T13:32:01Z Cannot connect to socket for [2a04:6ec0:20d:d050:819a:ceb8:4416:2bc0]:8333
     62024-03-13T13:32:14Z New block-relay-only v1 peer connected: version: 70016, blocks=834513, peer=40
     72024-03-13T13:34:47Z New block-relay-only v1 peer connected: version: 70016, blocks=834513, peer=41
     82024-03-13T13:34:58Z Cannot connect to socket for 122.206.190.60:8333
     92024-03-13T13:36:23Z New block-relay-only v1 peer connected: version: 70016, blocks=834513, peer=42
    102024-03-13T13:37:38Z Cannot connect to socket for 138.199.60.26:8333
    112024-03-13T13:38:34Z Cannot connect to socket for 120.236.171.239:8333
    122024-03-13T13:39:00Z Cannot connect to socket for 35.72.149.104:8333
    132024-03-13T13:41:22Z Cannot connect to socket for 176.199.211.63:8333
    142024-03-13T13:41:25Z Cannot connect to socket for [2804:7f0:7c80:196f:b0b2:1642:e4a9:b794]:8333
    152024-03-13T13:41:37Z Cannot connect to socket for [2a00:b700::3:379]:8333
    162024-03-13T13:42:55Z Cannot connect to socket for [2001:871:24b:ab3e:d1f2:1f7e:b950:692f]:8333
    172024-03-13T13:43:01Z New block-relay-only v1 peer connected: version: 70016, blocks=834513, peer=43
    182024-03-13T13:45:30Z Cannot connect to socket for 121.237.138.96:8333
    192024-03-13T13:46:56Z Cannot connect to socket for 110.153.86.87:8333
    202024-03-13T13:48:32Z Saw new header hash=0000000000000000000071923ef5a18dc17d47717fae386296dfe8902414ea71 height=834514
    212024-03-13T13:48:32Z Saw new cmpctblock header hash=0000000000000000000071923ef5a18dc17d47717fae386296dfe8902414ea71 peer=16
    222024-03-13T13:48:32Z UpdateTip: new best=0000000000000000000071923ef5a18dc17d47717fae386296dfe8902414ea71 height=834514 version=0x2fffe000 log2_work=94.792163 tx=976064192 date='2024-03-13T13:48:27Z' progress=1.000000 cache=232.4MiB(1767736txo)
    232024-03-13T13:50:28Z Cannot connect to socket for 58.219.96.208:8333
    242024-03-13T13:50:41Z Cannot connect to socket for 189.19.183.194:8333
    252024-03-13T13:52:56Z New block-relay-only v1 peer connected: version: 70016, blocks=834514, peer=45
    262024-03-13T13:53:12Z Cannot connect to socket for [2600:381:6a90:cfa5:b0a4:5c43:fdb8:9298]:8333
    272024-03-13T13:56:25Z Cannot connect to socket for [2003:d7:df24:a900:53a3:8916:d525:9958]:8333
    282024-03-13T14:00:04Z Cannot connect to socket for 97.96.100.48:8333
    292024-03-13T14:03:56Z Cannot connect to socket for 195.181.167.197:8333
    302024-03-13T14:04:02Z Saw new header hash=00000000000000000001ee20083bb8df96e256bb46dd856c48f686b692dbc29b height=834515
    312024-03-13T14:04:02Z Saw new cmpctblock header hash=00000000000000000001ee20083bb8df96e256bb46dd856c48f686b692dbc29b peer=16
    322024-03-13T14:04:02Z UpdateTip: new best=00000000000000000001ee20083bb8df96e256bb46dd856c48f686b692dbc29b height=834515 version=0x294c8000 log2_work=94.792178 tx=976067423 date='2024-03-13T14:03:37Z' progress=1.000000 cache=234.9MiB(1785815txo)
    332024-03-13T14:04:31Z Cannot connect to socket for [2606:6080:1001:30:5f94:f242:c829:1017]:8333
    342024-03-13T14:04:32Z Saw new header hash=00000000000000000001cee842398455cae402ab13037a450fe4829db7efd2e2 height=834516
    352024-03-13T14:04:32Z Saw new cmpctblock header hash=00000000000000000001cee842398455cae402ab13037a450fe4829db7efd2e2 peer=16
    362024-03-13T14:04:33Z UpdateTip: new best=00000000000000000001cee842398455cae402ab13037a450fe4829db7efd2e2 height=834516 version=0x27fd4000 log2_work=94.792192 tx=976069773 date='2024-03-13T14:04:13Z' progress=1.000000 cache=236.4MiB(1799558txo)
    372024-03-13T14:04:59Z Cannot connect to socket for 178.70.102.186:8333
    382024-03-13T14:05:43Z Cannot connect to socket for [2409:8a55:6ad:7101:49d4:4737:ccad:17c]:8333
    392024-03-13T14:06:21Z Saw new header hash=000000000000000000030739bcd83bfdfc6a03a1a58e21e4c61d6b59ac047569 height=834517
    402024-03-13T14:06:21Z Saw new cmpctblock header hash=000000000000000000030739bcd83bfdfc6a03a1a58e21e4c61d6b59ac047569 peer=16
    412024-03-13T14:06:21Z UpdateTip: new best=000000000000000000030739bcd83bfdfc6a03a1a58e21e4c61d6b59ac047569 height=834517 version=0x2cbea000 log2_work=94.792206 tx=976071288 date='2024-03-13T14:05:33Z' progress=1.000000 cache=236.4MiB(1802022txo)
    422024-03-13T14:06:45Z Cannot connect to socket for 86.168.50.224:8333
    432024-03-13T14:08:29Z Cannot connect to socket for 194.163.131.91:39388
    442024-03-13T14:11:12Z Cannot connect to socket for [2603:6080:7a0e:59ff:9da7:4932:3d76:58ec]:8333
    452024-03-13T14:11:41Z Cannot connect to socket for 108.214.8.13:60501
    462024-03-13T14:12:06Z Cannot connect to socket for 84.178.138.51:8333
    
  216. pinheadmz referenced this in commit 95fda8a339 on Mar 13, 2024
  217. pinheadmz referenced this in commit bdd38fc6e3 on Mar 13, 2024
  218. pinheadmz referenced this in commit c70e4fc9a3 on Mar 13, 2024
  219. fanquake referenced this in commit 55c6323434 on Mar 14, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-07-03 10:13 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me