RFC: Introducing AltNet, a pluggable framework for alternative transports

ariard commented at 10:50 am on May 16, 2020: member

As of 14/03/2022 (chain tip hash : 00000000000000000000628002621832be9281faf72c394e66a8bc8446f80c75), this project is under implementation in separate process relying on the multiprocess framework. See https://github.com/chaincodelabs/libmultiprocess/issues/56 for more information on current state.

This PR proposes to introduce a new subsystem aiming towards providing a generic framework for integrating alternative transport communication channels inside Bitcoin Core. This PR is a functional proof-of-concept using LN transport protocol as a new link layer for header fetching. See #18989 for high-level discussion and proposal merits.

The Design

Previous work has been attempted to increase network security and robustness by increasing communication redundancy. While being optional and fault-tolerant, these attempt don’t scale well to integrate a wide-range of alternatives such as domain-fronting, orbital communications, amateur radio, Tor pluggable transports, unlicensed UHF, Bluetooth, I2P, Lightning Noise transport, … Each of these transports has its own availability, latency, don’t bind to the Berkeley sockets model, are unidirectional, require custom peer discovery, are not suited for block traffic, …

This proposal tries to overcome these issues by laying out a generic “driver” framework with its own new processing stack, thereby encapsulating per-transport oddities. It introduces:

Per-transport capabilities (TransportCapabilities). Originally declared by each driver, they abstract driver specifities to general flags, like directionality, protocol messages scope. They could be extended with further attributes, like max bandwidth for adjusted packing or “privacy”/“trustness” niceness of the link.
Abstract driver model (CDriver). Basically, it dissociate driver warmup, message preprocessing, sending and reception. Ideally you would want to load them dynamically.
“Anycast” header-fetching (AltLogicValidation::BlockHeaderAnomalie). After notification from the watchdog module, it sends a GETHEADERS message to any peers supporting bidirectionality and headers processing.
Alternative processing stack (AltLogicValidation). Relying on its own threads, it manages drivers and may take action based on static logical capabilites instead of peer state, reducing likelihood of bugs. In the future, one could implement a more robust headers-syncing strategy like reverse-header or compressed headers. It may also leverage a future asynchronous validation interface.

There are many advantages of this kind of design –, it is easy to integrate new drivers binding to an abstract class, it is easy to reason about due to logical capabilities and at the same time provides granular tuning due to the fine-integration with rpc. Peer discovery and policy or bandwidth/memory DoS-protection are all possible driver add-ons as well. One could envision an rpc like sendemergencytx, broadcasting a transaction on any transport protocol available, like a LN’s justice transaction after detection of being eclipsed.

This module could be made entirely optional at compile-time with --enable-altnet even if such integration with the build system hasn’t been made.

Obviously this is an exploratory PoC, I don’t expect code here to be merged. Beyond worthiness of doing this in Rust there are a lot of open questions, including the threading/memory model, interface with existing codebase, build system, resources consumption and fault-tolerance. As next steps, I would favor building on the multiprocess work and hosting this new network stack in its own to avoid any bug contamination from introducing complex driver. I encourage those with opinions on the higher level motivation and design discussion to leave their ideas on the linked issue.

High-level design:

          ___________            ______________________
         |           |          |                      |
         | CWatchdog |----------|  AltLogicValidation  |
         |___________|          |______________________|
                                           |
                                     ______|______
                                    |             |
                                    |  CAltstack  |                                                         
                                    |_____________| 
                                           |
                                       ____|____
                                      |         |
                                      | CDriver |
                                      |_________|
                                           |
                                   ________|__________
                                  |                  |
                                  | ClightningDriver |
                                  |__________________|

The Demo

This PoC features a header-fetching round-trip, leveraging LN connection protocol. Each LN node connects to its peers through an encrypted and authenticated channel. Since your LN peer must have chain access, you could leverage this privileged link to create a redundant chain access.

Using C-lightning custom LN message support and plugin API, we could embed directly inside HEADERS/GETHEADERS and realize a round-trip between 2 disconnected bitcoin nodes through their respective C-lightning nodes.

For this to work, you need to compile and run 2 C-lightning nodes, each with its own data directory and launched with this custom bridge plugin. Once those LN nodes are running, activate bridges by lightning-cli alice_on and lightning-cli bob_on. The plugin doesn’t seem to accept CLI argument so I had to hardcode some node addresses.

Then compile 2 bitcoin nodes with this patchset. For one of them you must change the hardcoded port in C-lightning driver and deactivate the watchdog logic. Also, it would be best if you changed SCAN_INTERVAL to something reasonable, for instance 1 min.

High-level setup:

                                             -->

                __________          LN_msg(`GETHEADERS`)                   ________
               |          |                                               |        |
               | CL-Alice |-----------------------------------------------| CL-Bob |
               |__________|                                               |________|           
                    |               LN_msg(`HEADERS`)                         |
                ____|_____                                                 ___|_____
               |          |                 <--                           |         |
               |  bridge  |                                               | bridge  |
               |__________|                                               |_________|
                    |                                                         |
                ____|_____                                                 ___|____
               |          |                                               |        |
               | BC-Alice |                                               | BC-Bob |
               |__________|                                               |________|

                  HEIGHT                                                   HEIGH + 1

Running both nodes, they should automatically connect to the bridge, and after anomaly detection realize a header exchange, if one of them is at least one block forward (run generatetoaddress 1 $ADDR).

Otherwise you may observe past logs:

Alice:

 0Ê0-05-16T00:55:11Z Boostrapping Altstack
 12020-05-16T00:55:11Z init message: Done loading
 22020-05-16T00:55:11Z msghand thread start
 32020-05-16T00:55:11Z drivers-warmup thread start
 42020-05-16T00:55:11Z addcon thread start
 52020-05-16T00:55:11Z Registered driver with 0
 62020-05-16T00:55:11Z net thread start
 72020-05-16T00:55:11Z altstack-processing thread start
 82020-05-16T00:55:11Z Clightning - Warmup
 92020-05-16T00:55:11Z dnsseed thread start
102020-05-16T00:55:11Z altstack-handle thread start
112020-05-16T00:55:11Z 0 addresses found from DNS seeds
122020-05-16T00:55:11Z dnsseed thread exit
132020-05-16T00:55:11Z Clightning - Bridge connection
142020-05-16T00:55:11Z Registering node_id 0 with driver_id 0
152020-05-16T00:55:11Z drivers-warmup thread exit
162020-05-16T00:55:16Z Block header anomalie detected - notifying subscribers
172020-05-16T00:55:16Z Block header anomalie detected - Anycasting header fetching from 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206
182020-05-16T00:55:16Z Clightning - Preprocessing getheader
192020-05-16T00:55:16Z Cligthning - Bridge send success 71
202020-05-16T00:55:19Z Cligtning - Bridge receive success
212020-05-16T00:55:19Z Receive HEADERS from 0
222020-05-16T00:55:19Z Receiver header 65084efeaffe9929de0a0bb747cf48ac7f9b4edb214b2d1ad2ea468977be8e1e vs tip 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206

Bob:

 02020-05-16T00:55:09Z Boostrapping Altstack
 12020-05-16T00:55:09Z addcon thread start
 22020-05-16T00:55:09Z opencon thread start
 32020-05-16T00:55:09Z msghand thread start
 42020-05-16T00:55:09Z drivers-warmup thread start
 52020-05-16T00:55:09Z Registered driver with 0
 62020-05-16T00:55:09Z Clightning - Warmup
 72020-05-16T00:55:09Z init message: Done loading
 82020-05-16T00:55:09Z dnsseed thread exit
 92020-05-16T00:55:09Z altstack-processing thread start
102020-05-16T00:55:09Z altstack-handle thread start
112020-05-16T00:55:09Z Clightning - Bridge connection
122020-05-16T00:55:09Z Registering node_id 0 with driver_id 0
132020-05-16T00:55:09Z drivers-warmup thread exit
142020-05-16T00:55:10Z Leaving InitialBlockDownload (latching to false)
152020-05-16T00:55:17Z Cligtning - Bridge receive success
162020-05-16T00:55:17Z Receive GETHEADERS from 0
172020-05-16T00:55:17Z Fetch header starting from 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206, tip 65084efeaffe9929de0a0bb747cf48ac7f9b4edb214b2d1ad2ea468977be8e1e
182020-05-16T00:55:17Z Sending back 1 headers
192020-05-16T00:55:17Z Clightning - Preprocessing header
202020-05-16T00:55:17Z Cligthning - Bridge send success 84
212020-05-16T00:56:10Z Adding fixed seed nodes as DNS doesn't seem to be available.

[watchdog] Introduce CWatchdog class

CWatchdog is a cross-layer anomalies detection module aims to
cover block issuance, peer management, local clock or net level.

If any anomalie is detected an event is triggered to inform an
application layer or any internal consumer of a future watchdog
interface.

This code is only integrated in next commit.

816f50f4b1

[watchdoginterface] Introduce CWatchdogInterface class

CWatchdogInterface is consumed by another module or application
willingly to take corrective actions based on anomalies detected.

CWatchdogInterface only callback is BlockHeaderAnomalies, meaning
block header aren't received at a normal rate modulo block variance.

14c392ae08

[init] Integrate CWatchdog and its interface in init sequence

CWatchdog is integrated in NodeContext to make it accessible to any
futute rpc calls. Some application may fine-tune watchdog according
to their requirements.

This code is not scheduled yet.

fcafe5787f

[scheduler] Schedule ScanAnomalies every 60min

Run ScanAnomalies to process any heuristics and trigger watchdog
interface callback.

As of present commit, there is no heuristic implemented, callback
is blankly called to exerce watchdog interface.

7dd1b22b6f

[net] Integrate CWatchdog in CConman

Insert watchdog LogHeader after a successful ProcessNewBlockHeaders,
therefore updating last time watchdog have seen a step forward
on headers tree construction.

Watchdog loggers should be designed on being invasive-minimal at they
may encumber hot paths.

7d0e7c9884

[altnet] Introduce CAltstack class

CAltstack is a secondary network stack for alternative transports.
It aims to provide a pluggable framework to host diverse transports,
like obfuscated traffic, radio-based, protocols piggy-backing, ...

This class introduces a foundation, followed-up in next commits.

e8d825f6fa

[altnet] Add CAltMsg class

This new class helps to communicate identifier payload between low-level
alternative stack drivers and message processing logic.

Add vRecvMsg to CAltstack to serve as a buffer between drivers processing
and messages processing.

7485180066

[init] Integrate CAltstack and its interface in init sequence

CAltstack is integrated in NodeContext to make it accessible to any
future rpc calls. Some application may load alternative drivers
according to their requirements.

e727ba6fc1

[altnet] Add CDriver class

CDriver is an abstract interface which must be implemented by each
transport protocol driver to interface with CAltstack.

It contains basic methods to separate warmup, preprocessing, flusing
and receiving.

This is used in next commit.

d599505e21

[altnet] Add ThreadWarmupDrivers

Add a thread iterating over all drivers and warmup, associating to
each a driver id.

e9a676d459

[altnet] Add ThreadHandleDrivers

ThreadHandleDrivers is the logical equivalent of CConnman::ThreadSocketHandler,
listen for incoming connections, serve and receive messages per-transport
protocol drivers.

8dc65ad811

[altnet] Add ThreadAltProcessing

ThreadAltProcessing is the logical equivalent of Cconnman::ThreadAltProcessing,
processing messages to and from drivers.

7262d0c498

[altnet] Add TransportCapabilities class

This new class represents all capabilities supported by driver and must
be inherited to peer processing state.

Capabilities may be relatives to link bidirectionality, protocol
messages supported, privacy niceness, priority.

c784a40a1c

[alt_processing] Add AltLogicValidation class

This new class aims to process protocol messages from alternatives
transports. As those may not be bidirectional, prohibitive, privacy-
leaking or size-scoped, processing logic needs to do different
assumptions based on capabilities.

This is used in next commit.

59e33cade4

[altnet] Integrate AltLogicValidation

AltLogicValidation is integrated in NodeContext to make it accessible
to any future rpc calls. Some application may favor protocol messages
differently according to their requirement.

374074523f

[alt_processing] Add CAltNodeState class

This new class encapsulate a alternative validation-specific state.
It makes available peer capabilites to processing layer.

This design is exploratory, per-peer state may be dropped completely
and processing logic may only rely on drivers capabilities to make
decision.

ae1e3f0fc0

[altnet] Add mapNodesDriver, mapping node_id back to their driver 9335da7e25

[altnet] Add PushMessage to send message from processing

PushMessage dispatch message to the requested driver.

b8f43834bf

[alt_processing] Support GETHEADERS processing

At GETHEADERS reception if peer capabilities support sending we
reply back with headers.

ebdea790d4

[alt_processing] Implement CWatchdogInterface for AltLogicValidation

This is used in next commit.

d6e4b3d61b

[alt_processing] Implement "anycast" block headers fetching

"Anycast" is a header-fetching strategy where any abstract drivers
supporting header and bidirectionality is send a GETHEADERS, aiming
to receive back a longuer-valid headers chain.

10794716b0

[alt_processing] Integrate some basic logging ab6faa553b

[bugfix] Fix locking logic

As this whole patch is a PoC, threading and locking model is
non-robust.

e7ff7a5796

[alt_processing] Support HEADERS processing

At HEADERS reception we print headers received against our
on tip.

This processing may be easily plug to some validation interface
to call ProcessNewBlockHeaders.

06c456a7cf

[drivers] Add ClighntingDriver support

ClightningDriver connects through a bridge to a C-lightning node
to exchange headers on the LN transport layer through custom LN
messages.

Plugin uses C-lightning's `sendcustommessage` rpc aand "custommsg" hook.

3667d18a76

DrahtBot added the label Build system on May 16, 2020

DrahtBot added the label P2P on May 16, 2020

icota commented at 1:16 pm on May 16, 2020: contributor

Exciting stuff but I question the logic of there being a framework just for “alternatives”. Shouldn’t ipv4/6 and onion be one of many drivers with their own capabilities and trustworthiness?

More work initially but probably easier maintenance down the road and a guarantee that any type of smoke signal protocol driver could be a first-class citizen like ip.

Concept NACK for not being ambitious enough :wink:

DrahtBot commented at 5:52 pm on May 16, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#18354 (Protect wallet by using shared pointers by bvbfan)
#16365 (Log RPC parameters (arguments) if -debug=rpcparams by LarryRuane)
#15367 (feature: Added ability for users to add a startup command by benthecarman)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

in src/altnet.h:12 in 3667d18a76

 7+#define BITCOIN_ALTNET_H
 8+
 9+#include <alt_processing.h>
10+#include <drivers.h>
11+#include <net.h>
12+#include <netmessagemaker.h>

TheBlueMatt commented at 8:27 pm on May 16, 2020:

I think this should be way more disconnected from the existing P2P stack. Accessing anything but validation.h and some utils seems bad.

TheBlueMatt commented at 8:32 pm on May 16, 2020: member

This is really cool! Its certainly the direction I was playing around with with the attempts at adding Rust stuff. That said, I’m really not sure the API is right, though - it seems much too tied to the existing serialization/network framework, which both misses a good chunk of the redundancy possibility and prevents things like blocks-over-HTTP(s), DNS, radio, etc, etc.

ariard commented at 0:08 am on May 17, 2020: member

@TheBlueMatt what’s about a thread pool for every blocking alternative and a CDriver::IsReady when we try to read from it ? Have you in mind other alternative which needs a really specific threading model ? That’s a bit more complexity but you may allocate priority to avoid starvation.

I agree that the mapNodeState isn’t great and ideally you do want to defer every peer policy to the driver. Reducing state is less error-prone. @icota Do you have in mind a different ipv4/ipv6 flow that the one implemented by main net ? Path-aware routing could be cool and definitely something belonging there.

willcl-ark commented at 8:20 am on May 18, 2020: member

Hello @ariard pluggable alternative transport systems for Bitcoin Core gets a concept ACK from me.

I wonder if in the future this framework could include some transport prioritisation which might, for example, relay blocks over an internet/Tor connection but attempt to broadcast and relay transactions over Amateur radio/bluetooth (as available) with a new Tor circuit as a fallback, to further obscure a transactions origin.

In the case that your node might have a metered main connection e.g. 3G along with a secondary transport mechanism, the same prioritisation might be able to help split bandwidth required between multiple transports simultaneously.

For the low bandwidth/metered scenario, I feel this would be an excellent match for the compressed block headers scheme I picked up on in my email here.

remyers commented at 9:53 am on May 18, 2020: none

I love this concept! I will need some more time to have implementation comments, but will try to think about this in the context of the alt net I’m most familiar with (stateless unlicensed UHF Mesh / goTenna). Some preliminary big picture thoughts:

should the Watchdog functionality eventually be broken out as its own RFC? even without an altnet transport wouldn’t a core wallet also want notification of unusual p2p events?
Right now core is aware of ASN buckets, do think different altnet connections could be treated as diverse ASNs? 2a) Should altnet peers be treated as one ASN, or is there a way to ensure diversity to protect against sybil altnet peers.
Do you envision a plugin-driver model so that core would support the high level CDriver/Watchdog, but people could plug-in their own low-level implementations? 3a) In the case of goTenna, we have a low-level C driver that is not open source so would not be appropriate to include in core. Would something like the HWI system make sense for different vender specfic CDriver instances?
A deeper integration to make altnet peers 1st class peers would be difficult, but if a CNetAddr worked more like a URI then peers could be stored as ip:\ipaddress:port and altnet nodes would look like clightning:\peeraddr and use an altnet CDriver named clightning. 4a) A less invasive approach to 1st class would be that any altnet peer just uses 127.0.0.1:cdriver_port, but all peers can include user configurable transport capabilities.

Great work, I’m really excited to try this out.

MarcoFalke commented at 12:36 pm on May 18, 2020: member

Concept ACK, but I am wondering if some of this would be better off outside the Bitcoin Core build system. Obviously stuff that can be done with no additional requirements (additional requirements could be a proxy, another daemon or even new hardware like an antenna) and that is a clear benefit to Bitcoin Core can live inside of Bitcoin Core. For example something like the existing TipMayBeStale logic.

However, for transports that would pull in additional dependencies, a more flexible approach would be to define a common interface. The drivers could then ask Bitcoin Core for raw validation data and statistics, then reply with new headers or blocks. For example, this could be done over the RPC interface. I haven’t thought about all the trade-offs, but it seems beneficial if addons can be attached and removed during run-time as well as being developed independently of the Bitcoin Core development process?

ariard commented at 6:48 am on May 19, 2020: member

Hi @willcl-ark,

I wonder if in the future this framework could include some transport prioritisation which might, for example, relay blocks over an internet/Tor connection but attempt to broadcast and relay transactions over Amateur radio/bluetooth (as available) with a new Tor circuit as a fallback, to further obscure a transactions origin.

Yes introducing drivers capabilities (fHeaders, fTransaction, …) should be seen as primitives to label traffic according to application/user requirements. I can foresee bitcoin-cli ioctl bluetooth --label=tx,headers, you’re envisioning something like this ?

In the case that your node might have a metered main connection e.g. 3G along with a secondary transport mechanism, the same prioritisation might be able to help split bandwidth required between multiple transports simultaneously.

Ah thanks didn’t think about this, effectively when bandwidth has been consumed on one device you may want to fallback on another one. I do think bandwidth tracking should be done drivers-side.

For the low bandwidth/metered scenario, I feel this would be an excellent match for the compressed block headers scheme I picked up on in my email here.

Yes IIRC compressed block headers in the linked issue (#18989). Clearly that would be easier to innovate on message protocol, like also transaction compression.

ariard commented at 7:17 am on May 19, 2020: member

@remyers

should the Watchdog functionality eventually be broken out Do you envision a plugin-driver model so that core would support the high level CDriver/Watchdog, but people could plug-in their own low-level implementations? 3a) In the case of goTenna, we have a low-level C driver that is not open source so would not be appropriate to include in core. Would something like the HWI system make sense for different vender specfic CDriver instances?as its own RFC? even without an altnet transport wouldn’t a core wallet also want notification of unusual p2p events?

Yes Watchdog has its own PR (#18987), with a CWatchdogInterface. It could be used by any kind of application, including the wallet.

Right now core is aware of ASN buckets, do think different altnet connections could be treated as diverse ASNs? 2a) Should altnet peers be treated as one ASN, or is there a way to ensure diversity to protect against sybil altnet peers.

I would like to avoid dealing with ASN/IP address at all. If you assume wireless communications there is no ASN to deal with, peer discovery and policy should be inside the driver. With regards to peer, you may only need a general way to authenticate them, so maybe some driver method enumerating its hosted peers ?

Do you envision a plugin-driver model so that core would support the high level CDriver/Watchdog, but people could plug-in their own low-level implementations? 3a) In the case of goTenna, we have a low-level C driver that is not open source so would not be appropriate to include in core. Would something like the HWI system make sense for different vender specfic CDriver instances?

I clearly envision a plugin-driver model and exactly core would support the high-level CDriver/CWatchdog/AtLogicValidation`. I lean to something like HWI and drivers implementations being hosted outside of core, under bitcoin-core/ for the open-sources ones. With regards to proprietary, I don’t have opinion, if they bind to the interface that’s better for you I guess.

A deeper integration to make altnet peers 1st class peers would be difficult, but if a CNetAddr worked more like a URI then peers could be stored as ip:\ipaddress:port and altnet nodes would look like clightning:\peeraddr and use an altnet CDriver named clightning. 4a) A less invasive approach to 1st class would be that any altnet peer just uses 127.0.0.1:cdriver_port, but all peers can include user configurable transport capabilities.

I don’t want to interfere with the actual p2p stack. But if what you want to address is user configurable transport capabilities that can be done at the rpc-level and yes something like bitcoin-cli ioctl Wifi:XXX --label=tx,addr ?

Thanks for the feedback!

ariard commented at 7:36 am on May 19, 2020: member

However, for transports that would pull in additional dependencies, a more flexible approach would be to define a common interface.

Yes that’s the idea defining a CDriver interface, even if it needs refinement. drivers/ should be elsewhere and you just keep the fetching logic/interface in Core. I don’t think drivers should ask for raw validation data on their own, i.e you may receive your block from actual p2p but want to broadcast it over WiFi. Drivers acting on their own means you poll a lot for nothing due to block variance, and that’s what RPC doesn’t fit IMO. Haven’t thought that much about run-time addition but I completely agree to not encumber Bitcoin Core development process more. You provide a generic framework and drivers devs can work on their own in another repo.

TheBlueMatt commented at 7:24 pm on May 21, 2020: member

but I am wondering if some of this would be better off outside the Bitcoin Core build system. @MarcoFalke sadly, such a design kicks something out to live in purgatory forever. We’ve never materially seen any real adoption of “additional, on the side” contrib things. Even opt-in configuration is rarely used, let alone something with additional setup requirements. I don’t think this carries much value unless its targeting (eventually) being built-in (though not enabled) by default.

TheBlueMatt commented at 7:27 pm on May 21, 2020: member

@TheBlueMatt what’s about a thread pool for every blocking alternative and a CDriver::IsReady when we try to read from it ? Have you in mind other alternative which needs a really specific threading model ? That’s a bit more complexity but you may allocate priority to avoid starvation.

I don’t know that the framework even needs to provide any kind of threading? Why not let the applications decide that. Using the net.h stuff like NetMsgType, CBlockLocator, NetMsgMaker and the connection types really limits the utility here. eg radio stuff probably wants to read/write from a tty, not a “real socket”. Let alone something like DNS which doesn’t even have a file descriptor, only a libc call to poll.

ryanofsky commented at 3:07 pm on May 22, 2020: member

I’m starting to look at this and my first impressions are basically the things Matt’s saying: that the API for transports can probably be very minimal and not deal with threading, and not interact with (or duplicate and fork) existing net_processing and net code.

A real anomaly detection implementation, on the other hand, may need to have access to more things, but hopefully that could happen more by passively receiving events than actively looking up state.

I think the writeups in #18987, #18988, #18989 provide a good description of the features this is trying to support, and the PR code provides a good proof or concept and sketch of design. But one thing that’s missing and would be nice would be a plan for how this might be rolled out. I’m assuming we wouldn’t add a new framework before anything was using it, and I know Matt has already implemented a lot of things that could use this framework. So it’d be great if there was a writeup (or outline like the assumeutxo plan) saying what the steps could be for rolling this out, and linking to Matt’s implementations, and saying how things could be integrated. The dev wiki could be a good place to put this, since it would be speculative and subject to change.

On the PR itself, I know it is exploratory and you don’t expect it to be merged, but was hard not to notice it was copying and pasting a lot of existing code from net/net_processing/validationinteface code and also introducing new code following older conventions (CPrefixedClassNames bHungarianVariables), so I just want to quickly encourage using the current style in new code and trying to avoid duplicate implementations of things like WatchSignalsInstance / MainSignalsInstance in the real version of this.

DrahtBot added the label Needs rebase on May 23, 2020

DrahtBot commented at 12:05 pm on May 23, 2020: member

🐙 This pull request conflicts with the target branch and needs rebase.

ariard commented at 7:21 am on May 27, 2020: member

Using the net.h stuff like NetMsgType, CBlockLocator, NetMsgMaker and the connection types really limits the utility here. eg radio stuff probably wants to read/write from a tty, not a “real socket”. Let alone something like DNS which doesn’t even have a file descriptor, only a libc call to poll.

I think message translation from a wire format suiting communication channel should be done by the driver, it doesn’t make sense at all to make it in the framework. So we may pass NetMsgType to the driver interface without caring further.

With regards to threading right, but isn’t the case in main stack we have a separation between ThreadSocketHandler and ThreadMessageHandler for performance reasons ? We may follow there ?

ariard commented at 7:33 am on May 27, 2020: member

Thanks @ryanofsky for review !

and not interact with (or duplicate and fork) existing net_processing and net code.

Right, what are all the reasons between net_processing and net code separation ?

A real anomaly detection implementation, on the other hand, may need to have access to more things, but hopefully that could happen more by passively receiving events than actively looking up state.

Right, CWatchdog may be consumer of an asynchronous interface too, I keep it simple for a PoC but that’s a good suggestion.

So it’d be great if there was a writeup (or outline like the assumeutxo plan) saying what the steps could be for rolling this out

Yes it’s here : https://github.com/ariard/altnet-proposals/tree/master, still WIP. Note: I’ve been talking with folks actively working on alt-com like BlockSat. As first functional step , I plan to reuse the headers-over-DNS from Matt, it’s pretty straightforward and can be used by anyone without further requirement.

On the PR itself, I know it is exploratory and you don’t expect it to be merged, but was hard not to notice it was copying and pasting a lot of existing code from net/net_processing/validationinteface

Right, it may make sense to refactor MainSignalsInstance to make it generic enough for reuse.

luisan00 commented at 4:26 pm on May 27, 2020: none

Concept ACK, but I am wondering if some of this would be better off outside the Bitcoin Core build system. …

Totally agree. In principle could be better as a separate project. It seems interesting but as you say, a common interface can help in this particular, separating transport logic from the Bitcoin Core logic.

I like the idea and will think carefully the next days in the context that is most familiar to me, ISM and low power mesh networks, and in particular Locha Mesh.

icota commented at 0:58 am on May 30, 2020: contributor

Do you have in mind a different ipv4/ipv6 flow that the one implemented by main net ? Path-aware routing could be cool and definitely something belonging there.

My point is that rather than making a separate framework I would prefer to change net to support the functionality you are suggesting for AltNet.

Look at what I am greeted with when I run getnetworkinfo:

 0"networks": [
 1    {
 2      "name": "ipv4",
 3      "limited": false,
 4      "reachable": true,
 5      "proxy": "",
 6      "proxy_randomize_credentials": false
 7    },
 8    {
 9      "name": "ipv6",
10      "limited": false,
11      "reachable": true,
12      "proxy": "",
13      "proxy_randomize_credentials": false
14    },
15    {
16      "name": "onion",
17      "limited": true,
18      "reachable": false,
19      "proxy": "",
20      "proxy_randomize_credentials": false
21    }
22  ],

Right now this output is produced by iterating over a dumb enum but would it not be cool if you could plug a “satellite” or “radio” driver in there.

This approach to alternative transports seems very intuitive to me but I don’t see the sentiment echoed by anyone else. I haven’t dug deep so it makes me think it’s either not a good idea to begin with and/or involves some very heavy lifting?

Have you considered it?

ariard commented at 1:17 am on June 1, 2020: member

@icota

My point is that rather than making a separate framework I would prefer to change net to support the functionality you are suggesting for AltNet.

I spend some time thinking to integrate this in our currently net/net_processing.

With regards to net you need to abstract any socket operation behind some interface, make peer stateful to know which stack they are relying on and threading may not fit, cf Matt argument on polling.

With regards net_processing, we currently assume bidirectionality, one-transport-fit-all (melding our own tx announcement with public block-relay) and timing based on always existing connection.

Now, considering the user interface it should be unified, you may host both altnet and net behind same commands, i.e in your example “radio” and “blocksat” would present too, just with a field altnet=true. Ideally traffic assignement (“this peer will do block-relay”) should be cross-stack

ryanofsky referenced this in commit d6d835680d on Jun 3, 2020

ryanofsky referenced this in commit 4277501031 on Jun 3, 2020

ryanofsky referenced this in commit 9737cc9b50 on Jun 3, 2020

ryanofsky referenced this in commit 9df8470cd1 on Jun 3, 2020

ryanofsky referenced this in commit 643d602b3f on Jun 11, 2020

Ayms commented at 10:36 am on June 11, 2020: none

I am not expert at all with bitcoin code so I don’t know what approach could be the best

But let’s take another network example, IPFS, despite of their efforts (ie libp2p) it is not so simple to plug “other codebase” to the system, see https://github.com/ipfs/ipfs/issues/439#issuecomment-593116241, even with the same techno

And as we can see the codebase I am referring too here is https://github.com/Ayms/node-Tor

The plan right now for bitcoin would be bitcoin | node-Tor | bitcoin, with (taking @icota approach):

"networks": [
  ...
	{
	  "name": "unix pipe",
	  ...
	},

Yes it’s a bit ugly but I don’t see other way today without using a localhost/socks, which I don’t want, even if suppported, for sure it would be better to pipe directly to the nodejs module (is there a way to do unix pipe today for some potential testing ?)

And more ambitious at the end would be to be able to plug also to something like http://www.peersm.com/Convergence-2020.pdf “A universal and generic architecture to anonymize any application or protocol and turn it into an independent decentralized p2p network inside browsers and servers, with browsers acting as servers”, including WebRTC, browsers, etc

ryanofsky referenced this in commit 556564f853 on Jun 11, 2020

ryanofsky commented at 3:20 pm on June 11, 2020: member

re: #18988 (comment)

Yes it’s a bit ugly but I don’t see other way today without using a localhost/socks, which I don’t want, even if suppported, for sure it would be better to pipe directly

Not completely sure about the context but 52192725a3ec1f814cf40049de45efffc226cd5b from #19160 is an example showing how to spawn a new process and communicate with it bidirectionally over a pipe. It requires a some boilerplate to define a new src/interfaces/ class and interfaces::Init::make method, but after you do that it’s easy to add methods to the interface, and make method calls from either process to the other with call forwarding and serialization and threading handled automatically.

Additionally with 01c31f9dcd15c1009c4b69e820fc0c01b73cd3c9 from the Process Separation branch ryanofsky@ipc-export, bitcoin listens on a <datadir>/sockets/node.sock socket that other processes can connect to and call the same interfaces.

icota commented at 4:19 pm on June 11, 2020: contributor

I almost forgot since it was some time ago but in c-lightning we added local socket (AF_UNIX) support (https://github.com/ElementsProject/lightning/pull/1323, https://github.com/ElementsProject/lightning/pull/1450) and with a couple of processes in between I ran the protocol over NFC.

Pipes and sockets approach can support a lot (but not all) of “alternative transport” use cases and seems like low hanging fruit. It is definitely something to consider.

ariard commented at 8:45 pm on June 17, 2020: member

@icota

Pipes and sockets approach can support a lot (but not all) of “alternative transport” use cases and seems like low hanging fruit. It is definitely something to consider.

After reviewing Russ PR (#19160), you can reuse process spawning and ipc framework to let communicate between a bitcoin-node and bitcoin-altnet one. And then again it could be reused to let bitcoin-altnet talks with any driver, each its own process. These drivers may not use a socket approach for the alternative transport they are supporting.

Pipes and sockets seems to me suited for “internal” communications between our collections of processes, not mandatory for talking with external, or do you see things differently ?

ariard commented at 8:53 pm on June 17, 2020: member

@Ayms

Yes it’s a bit ugly but I don’t see other way today without using a localhost/socks, which I don’t want, even if suppported, for sure it would be better to pipe directly to the nodejs module (is there a way to do unix pipe today for some potential testing ?)

What issue do you have with sockets ? Let’s envision this setup : bitcoin-node <-> bitcoin-altnet <-> tor-driver <-> node-tor where <-> denotes Unix socket as IPC mechanism.

Using pipes you need process to be related and share the same file descriptor table which means integrating node-tor code to bitcoin one which isn’t considered. Unless I miss some unix tricks to pass pipe handles across process ?

icota commented at 3:17 pm on June 18, 2020: contributor

Pipes and sockets seems to me suited for “internal” communications between our collections of processes, not mandatory for talking with external, or do you see things differently ?

When I wrote the comment I wasn’t aware that @laanwj already attempted what I had in mind in #9979. The idea was that (in addition to AltNet that you are building) having P2P through UNIX sockets would allow for a lot of alternative transport use cases, in configurations like: core ---> relay process ---> (some sort of alternative transport) ---> relay ---> other core.

This is also achievable via TCP but IMO it is not as clean. Since the PR was eventually abandoned I’m going to guess that it wasn’t low hanging fruit.

Ayms commented at 11:06 pm on June 18, 2020: none

@ariard , no real issue with sockets and (as you write) it should better be bitcoin <-> node-Tor and not bitcoin -> node-Tor -> bitcoin (basic stdin/stdout), I was talking about unix sockets in my previous message (which I assimilated to unix pipe in terms of wording) and not inter unix processes pipes

I would just modify bitcoin-node <-> bitcoin-altnet <-> tor-driver <-> node-tor by bitcoin <-> node-Tor (via altnet) where node-Tor implements IPC or (better) bitcoin-node <-> altnet nodejs-driver <-> node-tor, so nodejs modules don’t have to implement unix sockets

Where node-js-driver is something like:

altnet.on(‘data’,//process data from unix socket)
altnet.send=//send data to unix socket

Then nodejs programs would just import altnet and turn it into the required object (a Duplex Object for node-Tor), two lines of code and that’s it (unfortunately as mentioned above this does not work very well with some existing examples (ipfs), but for me this is a misdesign of ipfs that should not be reproduced)

Indeed, in @icota example “networks” could be turned into “layers” and probably a good idea would be to be able to pipe between altnet layers when applicable (bitcoin-node <-> altnet some driver <-> my_super_encryption <-> altnet nodejs-driver <-> node-tor, it looks similar to what @icota proposes core ---> relay process ---> (some sort of alternative transport) ---> relay ---> other core

No invention here, this is what unix does since the beginning of times, and similar to #9979 attempt, probably some code already exists to do this (ie easily plug layers from one techno/process to another) @ryanofsky , thanks for all the links, I will look at it in more details when I need it but it still looks a bit complicate, for simple testing, is there a simple way (modif in the bitcoin core code) to redirect/listen the bitcoin protocol via stdin/stdout?

ryanofsky commented at 0:08 am on June 19, 2020: member

@ryanofsky , thanks for all the links, I will look at it in more details when I need it but it still looks a bit complicate, for simple testing, is there a simple way (modif in the bitcoin core code) to redirect/listen the bitcoin protocol via stdin/stdout?

With #10102 you could start unmodified bitcoin-node or bitcoin-wallet binaries via socat if you wanted to control them with the ipc protocol over stdin/stdout:

0socat - system:"src/bitcoin-node -ipcfd 4",fdin=4,fdout=4

I could actually tweak the multiprocess spawning code to communicate over stdin/stdout instead of a separate socketpair descriptor, so you wouldn’t need socat. But one reason the implementation avoids stdin/stdout is that it makes printf debugging awkward, and one stray printf or -printtoconsole anywhere in the codebase messes up the IPC connection.

Also as mentioned #18988 (comment), with 01c31f9 from branch ryanofsky@ipc-export, you can connect to an already running bitcoin-node through a <datadir>/sockets/node.sock socket, which is probably what you would want to do as an external application, rather than starting a bitcoin process.

Let me know what you’re trying to do though, since I’m not really sure what information’s relevant here.

Ayms commented at 10:35 am on June 19, 2020: none

What I want to do is:

bitcoin | node-Tor | bitcoin via stdin/stdout or bitcoin <-> node-Tor via unix sockets
		|
	send/receive
	via Tor or
	node-Tor nodes
		|
	optional: node-Tor RDV point
		|
	node-Tor or Tor nodes
		|
	bitcoin node via Tor exit node
		or
	bitcoin <-> node-Tor process

This is one phase of some funding proposals for the more global Convergence concepts, I will let you know when/if it happens and maybe altnet by that time will have evolved, if too complicate to setup another option is to try this with bitcoin-transactions which implement a part of the bitcoin protocol, and/or to combine both so the node-Tor node above can be a browser also (WebRTC/WS)

ryanofsky commented at 12:47 pm on June 19, 2020: member

What I want to do is:

0bitcoin | node-Tor | bitcoin via stdin/stdout or bitcoin <-> node-Tor via unix sockets

This is really interesting. Maybe it would be good to create a specific issue to discuss this here or in the node-Tor github repo.

If I were trying to connect bitcoind <–> node-Tor, I think the simplest approach to start with would be to get existing bitcoind -onion and -listenonion options to work with node-Tor. To do this, no bitcoind changes should be needed and node-Tor should just need to support simple subsets of the tor control and socks5 protocols (if it doesn’t already). To avoid the need to use plumbing tools like socat, or to modify bitcoind, it would probably be easiest to just start with plain tcp sockets instead of unix sockets or stdin/stdout.

After this worked, I’d start looking into more changes to streamline and package the functionality better:

Changing -onion and -listenonion to support unix sockets or pipes instead of tcp
Using #18988 (this PR) to write a new altnet driver that can speak socks5/torcontrol to node-Tor. No need to mess with #19160 (multiprocess pr) if the altnet driver is written in c++ or can be part of bitcoin core, but you might use #19160 if you want to write the altnet driver in a different language or more independently of the bitcoin core codebase.
Maybe switch from socks5/torcontrol to a custom protocol for bitcoind <–> node-Tor communication. If socks5/torcontrol don’t fit well into the altnet framework, maybe do this at the same time as writing the altnet driver.

Ayms commented at 4:03 pm on June 19, 2020: none

This is really interesting. Maybe it would be good to create a specific issue to discuss this here or in the node-Tor github repo.

Thanks, probably @ariard can tell us if this discussion should be moved to other places, I don’t find it uninteresting to compare the different approaches

node-Tor does support the socks interface since the very beginning but I don’t see the point of using a potentially vulnerable localhost and weak socks protocol for processes that are running on the same machine, we saw not a long time ago that it could lead to security breaches for bitcoin

The Tor project (which is designed for browsing) does not really have the choice today (also for Snowflake), for node-Tor I just don’t want to use it, and just don’t need it, because browsers for example are not browsing anything but acting as Tor protocol relays/nodes (paradoxically they can’t browse anything via WebRTC/WS/xhr interfaces)

So I think that a better approach is pluggable layers with altnet that can be chained including unix sockets and including browser interfaces

What I am proposing can look “ambitious/disruptive or whatever” but I am not far from it and those ideas are not just from yesterday, it’s just a matter of time to do it and that’s just a part of the Convergence project, 3 with unix sockets looks to be the right option for a universal nodejs driver (not talking about a specific driver for node-Tor, it should work for other nodejs programs), writing it myself… most likely no, I could do it but I can’t pretend to be a C/C++ expert for whom it should probably be a piece of cake to do it

ryanofsky referenced this in commit 16a7aa17a4 on Jun 29, 2020

ryanofsky referenced this in commit 187b769fdd on Jul 7, 2020

ryanofsky referenced this in commit 7928b7eb72 on Jul 10, 2020

ryanofsky referenced this in commit b24ca99478 on Aug 10, 2020

ryanofsky referenced this in commit 1525c389cc on Aug 11, 2020

ryanofsky referenced this in commit 3eb10c37b6 on Aug 17, 2020

ryanofsky referenced this in commit 907f95b07d on Aug 24, 2020

ryanofsky referenced this in commit f73918eb2d on Aug 26, 2020

ryanofsky referenced this in commit cb33a694b3 on Sep 1, 2020

ryanofsky referenced this in commit c832c85561 on Sep 24, 2020

ryanofsky referenced this in commit c915cea0db on Oct 1, 2020

ryanofsky referenced this in commit a8fbbd7364 on Oct 1, 2020

ryanofsky referenced this in commit 2f521b2c24 on Oct 2, 2020

ryanofsky referenced this in commit bf525a8d60 on Oct 21, 2020

ryanofsky referenced this in commit f663f7031c on Nov 25, 2020

ryanofsky referenced this in commit 7723ab68ca on Nov 25, 2020

ryanofsky referenced this in commit 1d1688f600 on Dec 8, 2020

ryanofsky referenced this in commit 5a695fd248 on Dec 8, 2020

ryanofsky referenced this in commit fe184ae2df on Dec 11, 2020

ryanofsky referenced this in commit fe0a68197a on Dec 18, 2020

laanwj referenced this in commit ac219dcbcc on Apr 27, 2021

fanquake commented at 1:30 pm on May 12, 2022: member

Given that #18987 (which this is based on) has been closed, I’m going to close this for now. I think further discussion of this in issues is potentially going to be more productive.

fanquake closed this on May 12, 2022

ariard commented at 11:46 pm on May 12, 2022: member

Still eager to collect feedbacks on the design. Blocker is currently the internal plumbing to communicate across processes. State of the discussion is here : https://github.com/chaincodelabs/libmultiprocess/issues/56

DrahtBot locked this on May 12, 2023

RFC: Introducing AltNet, a pluggable framework for alternative transports #18988

The Design

The Demo

Conflicts