Add a "block relay" to bitcoind with it's own port #7049

@TheBlueMatt has shown that a relay network helps with convergence (fewer orphans) but the implementation depends on an external centralised network of relay servers and a separate client that talks to bitcoind.

Bitcoin is peer2peer, so of course there will be network disagreement over the state of the blockchain and thus convergence to the resolution of that disagreement will always take time. There is no 100% solution to this design of bitcoin, instead the rules need to be (as they already are) simple to decide disagreement. Centralisation, to reduce the amount of disagreement, isn't an acceptable long term option.

An obvious option to help reduce divergence would be to add another port to bitcoind for a block relay network to allow for send/receive of blocks with the following if possible:

1. Compression - binary data, and also compression of that data - since it's a new non-network-forking option, the data can be designed for speed and compressed as much as possible since that would be an obvious gain. Transaction compression may also be an option - most nodes have most transactions (pretty much all but the coinbase) already available to them when most blocks arrive.

2. Early relay from trusted connections - e.g. send the block on if the header hash has been verified and version checks have been done (takes only nanoseconds to hash a block header on any modern desktop CPU or better) - mining finding blocks isn't that common so the risk of passing on invalid blocks weighs very low against the obvious network gains. Also of course, any bitcoind on the network can already distribute 'bad' blocks to any other bitcoind they are connected to, the current core client just stops them being relayed by delaying for the completion of the full block/transaction check. The level, defined within the new protocol, of checks done and accepted, could also be part of the trust level required to accept/transmit/ignore data.

This would effectively replace @TheBlueMatt relay network with bitcoind's own block relay network that would run in parallel with the current bitcoin network, however, it should also allow, in the configuration, for external/internal networks to accept the external block network at 'few' points and distribute it quickly to 'many' internal network points. Edit: and in case it isn't immediately obvious, this would allow for networks like @TheBlueMatt to be created and used as required. e.g. such a network similar to the one that already exists that you could configure in a higher trust rating based on your trust of any such available network.

implementation depends on an external centralised network of relay servers

Ah, that's a common misunderstanding-- and not true. Matt does run a well maintained public network of nodes running the protocol; but it has no dependence on it. And anyone can run the same software.

We've considered it a goal to include more efficient block relaying in core though, and have mostly just been waiting for the protocols implementing it to mature.

Compression - binary data, and also compression of that data

The bitcoin p2p protocol is already binary. Traditional data compression is not terribly useful except in the presence of address reuse, and not needed over and above mempool relative differential transmission-- which is critical to get any efficiency gain at all.

Early relay

This is orthogonal to more efficient transmission and could just be done in the ordinary p2p protocol with, a "unverified block" message. Luke-jr implemented this a couple years ago but it provided no meaningful speedup at the time (presumably due to other delay factors which we've since eliminated). So long as all parties involved know they're getting something non-validated, there isn't much harm in passing it along; as a combination of POW, chain consistency (e.g. extends the current tip), and serialization rules is enough to prevent denial of service.

and a separate client that talks to bitcoind.

Our preferred method of adding new protocols is via gatewaying from another process, for reliability and security reasons.

such a network similar to the one that already exists that you could configure in a higher trust rating based

In general our experience suggests that trust settings are black magic that cannot be configured correctly even by experts-- that even trusted things fail, and their doing so magnifies fault domains, so we should avoid them wherever possible. Fortunately, I don't see anything you've suggested that has any need for such settings.

implementation depends on an external centralised network of relay servers

Ah, that's a common misunderstanding-- and not true.

Ah, that's an incorrect statement of yours-- and not true.

Just as there are many bitcoin 'centralised' pools, many 'centralised' relays are still 'centralised' relays. Moving it to being part of the bitcoin protocol is what is called 'decentralised' and then only if not using centralised 'networks' like the current relay.

Early relay

This is orthogonal to more efficient transmission and could just be done in the ordinary p2p protocol with, a "unverified block" message.

Indeed and your suggestion that people should be using multiple daemons to solve this problem: #7015 (comment) requires this "early relay" since otherwise even a local set of (as you said):

to work around this today (run multiple daemons).

will relay to each other slowly since they must wait for full verification before passing the block on to any other local daemon

and a separate client that talks to bitcoind.

Our preferred method of adding new protocols is via gatewaying from another process, for reliability and security reasons.

That of course would be handled in an intelligent way - just as there are compiled in options for bitcoin core already - cli, qt, server (even compiling 'out' wallet) In this case, since the client would require a lot of bitcoin core, it would be a bad design to make that some other project requiring a second implementation of the rules already in bitcoin core. It could indeed be another program compilable with bitcoin core. The trust design would allow levels - and thus the 'relay client' would be a lot of bitcoin core including the code that performs the various checks in core applied before relaying the block, according to the trust level requirements.

such a network similar to the one that already exists that you could
configure in a higher trust rating based

In general our experience suggests that trust settings are black magic that cannot be configured correctly even by experts-- that even trusted things fail, and their doing so magnifies fault domains, so we should avoid them wherever possible. Fortunately, I don't see anything you've suggested that has any need for such settings.

I'm not sure what grade school you experienced your methods of software design, or who wrote software that you consider black magic, but dumbing down software and limiting it's performance because YOU believe you are unable to understand the concept of trust, sounds like a bad idea to me.

In fact even in the current design there is a rather obvious point of trust that could implement the relaying of non-fully verified blocks to your own list of bitcoinds - that would be a block submitted to the RPC - that already has a trust given to it to allow access to the RPC

The use of not relaying something, is to save the target from checking it, when you consider it invalid. Relaying something, that you may rarely consider bad, means that those who receive it must check it ... as they already do with everything else they receive.

As stated here: #7015 (comment) There is indeed a very clear need for it in your 'opinion' of how to solve that problem, without resorting to a simple improvement of the code, that I stated there: #7015 (comment)

In that example given, that raised the issue, and your 'supposed' solution to it, in fact what currently happens is work is supplied that is already stale even though bitcoin core code has the data that knows this in an extremely high percentage of those cases, but due to some unexplained locking failure of understanding, it's decided ok to increase network divergence.

If you had multiple bitcoinds, and didn't submit the block to the getblocktemplate bticoind, i.e. only submit the block to another one (or more) to process, and then be given back to the getblocktemplate source bitcoind via those, that would double the amount of time spent working on stale work.

block relay network has been implemented in #15759