Introduce auxiliary block requests

jonasschnelli commented at 9:30 am on November 16, 2016: contributor

This PR will allow auxiliary block requests. This will be required to run SPV wallets as well as this solves some other interesting use cases.

CAuxiliaryBlockRequest

A new “lock-free” class that handles the auxiliary blocks-download/processing
Contains a single global CAuxiliaryBlockRequest pointer (switching to a queue would be trivial).

SyncTransaction signal

A new parameter will be added that defines if the transactions are coming from a validated or non-validated block

New RPC call “requestblocks”

Allows to request blocks, if available on disk, they will be optionally passed tough the SyncTransaction signal (Wallet/ZMQ). If some or all of the blocks are not available, they will be downloaded (prioritized over “normal” IBD downloads). The blocks will be downloaded in parallel using the current block download mechanisms. CAuxiliaryBlockRequest will ensure that the blocks get processes in the correct order (as requested).

If someone has a good testplan for this, please stand up.

jonasschnelli added the label P2P on Nov 16, 2016

jonasschnelli added the label RPC/REST/ZMQ on Nov 16, 2016

jonasschnelli renamed this:
~~Introduce out-of-band block requests~~
Introduce auxiliary block requests
on Nov 16, 2016

jonasschnelli commented at 10:00 am on November 16, 2016: contributor

Renamed “Out-of-band Block Requests” to “Auxiliary Block Request” to avoid confusion.

jonasschnelli force-pushed on Dec 16, 2016

jonasschnelli commented at 3:19 pm on December 16, 2016: contributor

Did the non-trivial post main.cpp split rebase. Travis issue is related to #9364.

Short explanation how this works.

This PR allows someone to request blocks during IBD (or catch-up). These requested blocks can be newer the the current chaintip (validated blocks tip).
- Example: requestblocks start '["0000000000000000000a394efd567b68581485fa91bbec42b24a4268fda73a42"]'
If one requests blocks with a auxiliary block request (RPC requestblocks), bitcoin-core checks if the blocks are already available on disk and eventually download the missing ones from other peers.
Downloads of auxiliary block requests are prioritized over “normal IBD block downloads”.
Optionally, requestblocks will stuff the requested blocks through the SyncTransaction signal (example. ZMQ).
CAuxiliaryBlockRequest ensures that the blocks get passed in the right sequence through the signal (download can be asynchronous, but signal passing is in order [example: download can be D-F-E-G-B-C-A, but signal passing is always A-B-C-D-E-F-G])
You can check the status of the auxiliary request (if all blocks has been downloaded) by requestblock status ** During download, it can look like:

0{
1  "request_present": 1,
2  "created": 1481888029,
3  "is_cancled": false,
4  "requested_blocks": 80,
5  "loaded_blocks": 65,
6  "processed_blocks": 11
7}

Once the download/processing has been completed, you can get details from the block with things like getblock 0000000000000000000a394efd567b68581485fa91bbec42b24a4268fda73a4.

Use cases

This could serve as the core infrastructure for the fullblock hybrid SPV mode.
One could process (only header/PoW checked) blocks with this PR (attach an SPV wallet over ZMQ as example)

jonasschnelli commented at 3:44 pm on December 16, 2016: contributor

I forgot to mention that this could – in general – be useful for pruned nodes.

jonasschnelli force-pushed on Dec 18, 2016

jonasschnelli force-pushed on Dec 19, 2016

jonasschnelli commented at 5:53 pm on December 19, 2016: contributor

Added a hidden RPC command (setautorequestblocks) as well as a debug startup configuration (-autorequestblocks) to stop requesting blocks (IBD / catch-up).

This allows regression-tests of the auxiliary block requests.

Added a simple RPC test.

rebroad commented at 12:49 pm on December 30, 2016: contributor

I’m not really understanding why this PR is needed or what new functionality (that was needed) it does… sorry.

Add CAuxiliaryBlockRequest, a class to handle auxiliary blocks downloads

This is required for features that need to download and process block higher/further-away then the current validation depth

e1825ec223

Pass CBlockRequest blocks through SyncTransaction signal

+ Adds a validate=true|false to the SyncTransaction signal

81c6001d44

CBlockRequest: make SyncTransaction() optional 87f62504be

Add requestblocks - request out-of-band blocks download - RPC call a41ead7df3

[Wallet] don't consume non-validated transactions 7c2e134446

Add -autorequestblocks debug option and setautorequestblocks hidden RPC call

This allows efficient testing of auxiliary block requests

6b60e7e7e1

[QA] Add auxiliary block request test 9a43921ec0

jonasschnelli force-pushed on Jan 2, 2017

jonasschnelli commented at 2:38 pm on January 2, 2017: contributor

Force push fixed a rebase error.

jtimon commented at 2:21 pm on January 3, 2017: contributor

Concept ACK

Little CAuxiliaryBlockRequest refactor 5598be6f25

jonasschnelli force-pushed on Jan 4, 2017

jtimon commented at 6:37 pm on April 18, 2017: contributor

Needs rebase

ryanofsky commented at 6:42 pm on May 1, 2017: member

Maybe this PR should be closed? #9483 is newer and includes these changes. Also I posted a big batch of review comments there in February.

jonasschnelli commented at 3:02 pm on July 11, 2017: contributor

Closing in favour of #10794

jonasschnelli closed this on Jul 11, 2017

jonasschnelli removed this from the "In progress" column in a project

DrahtBot locked this on Sep 8, 2021

Introduce auxiliary block requests #9171

CAuxiliaryBlockRequest

SyncTransaction signal

New RPC call “requestblocks”

Short explanation how this works.

Use cases