[WIP] net: implement a StratumV2 Template Provider in core

Fi3 commented at 4:46 pm on September 20, 2021: none

This PR provides a simple example of how to import and use the sv2-ffi Rust crate which exports the Stratum V2 (Sv2) functions required by a Template Provider as a C library.

Stratum V2

An introduction to Sv2

Sv2 specifiaction

Template Provider

Sv2 defines a service (role) called the Template Provider (TP), whose functionality is defined as follows:

0Generates the custom block templates to be passed to the Job Negotiator for eventual mining. 
1This is usually a Bitcoin Core full node (or possibly another node implementation).

Phase 1: PR Goals

This phase encompasses the concepts that must be agreed upon before proceeding with Phase 2.

Need agreement that implementing a TP in Bitcoin Core is a good idea.
Need agreement that the implementation of a TP in Bitcoin Core should use the Rust crates in this workspace.

Phase 2: PR Implementation

This phase will commence once the concepts in Phase 1 are agreed upon. Phase 2 is the final goal of this project and encompasses the the implementation steps required for a functional TP in Bitcoin Core.

Extensive review of the sv2-ffi API and other Rust sources.
Review of the guix build process.
Find a location for the Sv2 crates (possibly under the Bitcoin GitHub Account) to live *1.
Implement the TP.

If TP in Core gets a concept NACK

A TP can be implemented as an independent service and can communicate with core via RPC.

If Rust in Core gets a concept NACK

A TP can be implemented in core without using any Rust dependency.

*1 Right now the Rust sources are in this PR and are packaged by a script before doing the guix build. Ideally these sources should live in another project repository and should be packaged by the guix script which builds a downloadable binary.

Fi3 commented at 4:49 pm on September 20, 2021: none

To build it:

0./contrib/guix/guix-clean
1env HOSTS='x86_64-linux-gnu' ./contrib/guix/guix-build

To test it:

0 ./guix-build-34bebad831a9/distsrc-34bebad831a9-x86_64-linux-gnu/src/bitcoind

An example of how to import sv2-ffi in a generic c++ project can be found here

in src/rusty/build-for-guix.sh:9 in bb059e4be3 outdated

0@@ -0,0 +1,36 @@
1+#! /bin/sh
2+
3+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
4+
5+rm -rf ${SCRIPT_DIR}/target
6+mkdir ${SCRIPT_DIR}/target
7+mkdir ${SCRIPT_DIR}/target/package
8+
9+cargo package --manifest-path ${SCRIPT_DIR}/sv2-ffi/Cargo.toml --allow-dirty

TheBlueMatt commented at 4:58 pm on September 20, 2021:

I don’t think Bitcoin Core build should ever call cargo for rust stuff - we should call rustc directly with the relevant libraries.

Fi3 commented at 9:22 am on September 21, 2021:

I’m gonna remove the cargo dependency. And subtree all the rust sources.

luke-jr commented at 5:30 pm on September 20, 2021: member

Concept NACK for Rust, until a secure bootstrap without trusted third party binaries is practical.

DrahtBot added the label P2P on Sep 20, 2021

DrahtBot added the label Scripts and tools on Sep 20, 2021

Fi3 commented at 9:23 am on September 21, 2021: none

Concept NACK for Rust, until a secure bootstrap without trusted third party binaries is practical.

Isn’t that already achieved with guix?

Fi3 force-pushed on Sep 21, 2021

Fi3 renamed this:
~~net: implement a StratumV2 Template Provider in core~~
[WIP] net: implement a StratumV2 Template Provider in core
on Sep 21, 2021

ariard commented at 1:33 am on September 23, 2021: member

I think this work is interesting, that said it might need more thought on how to interface with Core.

AFAICT, the proposed PR just link the library in the build system and create a new template encoder in bitcoind. Though it doesn’t introduce a new thread where the logic would co-run with other Core’s current threads (ThreadSocketHandler, ThreadOpenConnection, …). I initially followed this approach when I proposed AltNet with #18988. After gathering feedbacks from other contributors, a separated process sounds a more flexible and safer approach.

Recently, I’ve made some progress in this direction by building on top of the multiprocess project (see the hacky branch https://github.com/ariard/bitcoin/commits/2021-07-altnet-lightning). Process separation is promising, though going further I’m aiming to have AltNet pluggin modules written in Rust, while interfacing in Core over the Cap’n Proto newer interfaces. The current bottleneck is re-writing libmultiprocess in Rust, see https://github.com/chaincodelabs/libmultiprocess/issues/56 for more information. It’s more work than expected though I’ve a branch half-written doing that, hope to make it public soon(tm).

I think such modules or the one proposed in the current PR could live beyond the Core repository to avoid burdening the maintenance effort and the concerns about Rust chain of trust, while consuming the stable interfaces to ease deployment across the ecosystem.

Glad to see progress on the miner infrastructure, that’s really needed!

TheBlueMatt commented at 5:22 pm on September 23, 2021: contributor

Strong Concept ACK. At a high level, this is incredibly important work for a few reasons:

having a good block template producer in Bitcoin Core is critical for moving Bitcoin’s mining decentralization forward over the coming years, the importance of which cannot be understated,
Bitcoin Core having more visibility/control into how and when block templates get produced gives Bitcoin Core a ton more flexibility going forward to do things like pre-construct next-next-block templates in advance and push them out quicker, gives Bitcoin Core a ton more control of when transactions get included in a block by enabling “push” support for template updates instead of only being polled, etc,
the inefficiencies of getblocktemplate have been a real source of pain for pool operators for some time, I’m excited its going away.

As for multi-process, I don’t think this is a good candidate for multi-process - block template building is incredibly latency-sensitive, plus needs full access to the mempool and block template construction. At most, I’d think we could think of doing the block template construction in Core and then handing the template off to a new process which does the StratumV2 protocol work, but (a) that adds a bunch more latency for no reason, and (b) you’d probably just use the StratumV2 protocol to communicate with that other binary, so why? In general, one reason to consider rust for the protocol side of this is precisely because you don’t need to worry as much about it being in the same binary - the actual network logic itself isn’t going to corrupt memory.

sipa commented at 6:41 pm on September 23, 2021: member

Some thoughts.

I don’t object in general to adding a Rust-based dependency, as long as we can treat it as such. I don’t personally know Rust, and suspect that many contributors/reviewers aren’t as comfortable with Rust, so I think my criterion for this is whether it’s something that we can treat as external, existing, known-to-work, code that’s maintained separately - like say libevent or LevelDB, and not something that would expect detailed review from this project. Evidence that this same code is used in other contexts/projects would certainly help. The fact that this would be code that’s only accessible through a local, optional, interface makes the safety analysis a lot easier.
I haven’t followed the state of affairs with StratumV2. If it’s expected that a significant part of the mining ecosystem will end up depending on this protocol, I agree it makes sense to include it in Bitcoin Core, and agree with @TheBlueMatt that direct integration is the lowest-latency solution.
I believe Guix indeed solves the concern of having bootstrapping the Rust compiler, at least to the same extent that our C++ compilers for release builds are bootstrapped too. Right, @dongcarl?

So conditionally on getting some confirmation on my assumptions above, (somewhat hesitant) Concept ACK.

luke-jr commented at 6:48 pm on September 23, 2021: member

No, Guix does not solve the bootstrapping problem, because Guix itself has a bootstrapping problem. You can’t get Guix going without running their trusted blobs.

sipa commented at 6:54 pm on September 23, 2021: member

No, Guix does not solve the bootstrapping problem, because Guix itself has a bootstrapping problem. You can’t get Guix going without running their trusted blobs.

That’s correct (for now), but the situation is not Rust-specific. Our release builds are created with Guix, so the C++ compiler used for releases is subject to the same concerns.

dongcarl commented at 6:58 pm on September 23, 2021: contributor

I believe Guix indeed solves the concern of having bootstrapping the Rust compiler, at least to the same extent that our C++ compilers for release builds are bootstrapped too. Right, @dongcarl?

Yes I think this is exactly right. Building Rust in Guix is not any less bootstrappable than building anything else (C++) in Guix.

Some context: Guix gained the ability to compile Rust v1.19 from mrustc v0.8 in 2018 (blog post), and this bootstrap method has been actively maintained, and now compiles Rust v1.29 from mrustc v0.9 (newer versions of rust (say v1.n) simply build using the previous version (v1.(n-1)) until it reaches v1.29).

michaelfolkson commented at 7:39 pm on September 23, 2021: contributor

Concept ACK on implementing a Stratum v2 Template Provider in Core ignoring language.

On the C++/Rust question it is dependent on other contributors being comfortable with it. This was discussed in today’s Core dev meeting and has come up repeatedly in the past e.g. #17090. I would guess a C++ PR would be an easier/quicker merge but there may be a Rust approach that can work.

luke-jr commented at 8:53 pm on September 23, 2021: member

That’s correct (for now), but the situation is not Rust-specific. Our release builds are created with Guix, so the C++ compiler used for releases is subject to the same concerns.

We support users building their own, without using Guix’s trust-required toolchain.

ariard commented at 11:53 pm on September 23, 2021: member

As for multi-process, I don’t think this is a good candidate for multi-process - block template building is incredibly latency-sensitive, plus needs full access to the mempool and block template construction.

Ah I see the low-latency requirement, yes effectively IPC/serialization to access the mempool is likely going to be a worrying performance penalty.

That said, one advantage splitting nicely the block template/mempool subsystems behind interfaces like it has been done for GUI/wallet/node is the ability offered to the end-user to have either one fully-fledged binary or a collection of process in function what make sense as a deployment.

One motivation could be to have fallback mempools connected to your block template construction in case of a malicious partition targeting your main one.

Fi3 commented at 3:20 pm on October 7, 2021: none

Update build system following @dongcarl suggestions.

TP is enabled via a config flag: –enable-template-provider For now when compile with guix TP is enabled.

jamesob commented at 3:30 pm on October 7, 2021: member

Big concept ACK on supporting the use of StratumV2, but slight concept NACK on bundling it with the introduction of Rust, barring some additional context/rationale. It’d be nice to see a compelling reason why Rust should be used for this particular feature other than the standard potpourri of “Rust is safer/more ergonomic in $X_GENERAL_WAY.” Edit: I do see that this change pulls in a stratum Rust library, so maybe that’s why?

I have a number of questions about how Rust will dovetail with the existing codebase (and this PR might not be the appropriate place to discuss them), but some that come to mind in the context of this change are

how will use of Rust work with existing locks and lock annotations?
- E.g., are we going to lose the use of our deadlock detection tooling within Rust code?
what’s the real cost of maintenance/additional complexity in the build system?
how much third-party code does this actually entail dragging in? AFAIU, concurrency, e.g., is not a simple thing in Rust and often entails the use of third-party libs like tokio.

If Rust is really a critical part of doing this feature, maybe we should look at how to offer the raw data necessary (in a performant way) to construct any number of block template formats outside of Core, by supplying data in a generic format over e.g. Unix sockets.

DrahtBot commented at 8:54 pm on October 7, 2021: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#25076 (guix: native GCC 10 toolchain for Linux builds by fanquake)
#25037 (build: Create noinst_LTLIBRARIES conditionally by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Fi3 commented at 10:09 am on October 8, 2021: none

If Rust is really a critical part of doing this feature, maybe we should look at how to offer the raw data necessary (in a performant way) to construct any number of block template formats outside of Core, by supplying data in a generic format over e.g. Unix sockets.

Hi @jamesob:

why rust: main reason to use rust it to not rewrite an already working set of libraries, the libraries have been written to be used both in core (for the implementation of the TP) and in rust (for the implementation of the other Sv2 roles)
how will Rust work with existing locks: do not know I will check
cost of maintenance/additional complexity in the build system: IMHO is pretty insignificant. Support TP in the build system add:
1. few lines in the config file in order to check if there is the flag --enable-template-provider and if the system that will build core has at least rustc 1.51.
2. An m4 function pulled from the gnu repo ax_compare_version.
3. A make file that call rustc with the correct flag and build the rust library then add the builded library to LDFLAGS so that is linked with bitocoind (this is the most complex addition to the build system and are only 159 LOC that do a very simple task)
4. It modify 2 line in the guix build and add rustc 1.51 in the manifest
how much third-party: the rust code added in order to build the TP in core do not have any external dependency but std.
doing it externally: this was my original idea but, as stated also in the above post by @TheBlueMatt: block template building is incredibly latency-sensitive, plus needs full access to the mempool and block template construction

maflcko commented at 10:15 am on October 17, 2021: member

No objection to rust code. After all it is not different than wallet or qt code, which only a subset of all devs can understand, write and review. I guess the only thing needed here is review(ers).

DrahtBot added the label Needs rebase on Oct 20, 2021

ccdle12 commented at 4:34 pm on January 3, 2022: contributor

Just leaving a note that I’m currently working on an implementation of the template provider that integrates with the the rust sv2 project

Add sv2 via guix 7bfeca76e0

Remove cargo dependency fa876129b1

Remove unneccesary sources 3cbb132617

Remove local TP rust crates to be replaced by a git subtree 7ee49541b9

Add sv2 Rust crates for TP implementation 8792e6a253

Update guix builds to reflect src/rusty directory structure 04b3b5c7cf

Update build system

It conditionally compile the Template Provider logic into core if:
* the system that compile core has rustc at least at version 0.51
* configure is called `enable-template-provider`

Template Provider is optional, with default to no, so no rustc is required to compile core.
When compiled with guix the Template Provider is compiled in. As guix do support rustc.

9af61f3c06

Update guix rust version to latest avaiable 4e91f0ff9f

Fi3 force-pushed on May 12, 2022

DrahtBot removed the label Needs rebase on May 12, 2022

DrahtBot commented at 9:09 am on May 18, 2022: contributor

🐙 This pull request conflicts with the target branch and needs rebase.

Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a “draft”.

DrahtBot added the label Needs rebase on May 18, 2022

DrahtBot commented at 10:51 am on August 1, 2022: contributor

Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
Is it no longer relevant? ➡️ Please close.
Did the author lose interest or time to work on this? ➡️ Please close it and mark it ‘Up for grabs’ with the label, so that it can be picked up in the future.

ccdle12 commented at 9:27 am on August 3, 2022: contributor

We are currently working on changes, should be ready for review soon.

glozow added the label Mining on Aug 27, 2022

achow101 marked this as a draft on Oct 12, 2022

DrahtBot commented at 1:44 am on January 10, 2023: contributor

There hasn’t been much activity lately and the patch still needs rebase. What is the status here?

Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
Is it no longer relevant? ➡️ Please close.
Did the author lose interest or time to work on this? ➡️ Please close it and mark it ‘Up for grabs’ with the label, so that it can be picked up in the future.

Fi3 commented at 4:37 pm on January 10, 2023: none

we are implementing it in pure c++ without the use of the rust ffi lib so the PR is outdated

Fi3 closed this on Jan 10, 2023

bitcoin locked this on Jan 10, 2024