Rust-based Backup over-REST block downloader #16762

pull TheBlueMatt wants to merge 13 commits into bitcoin:master from TheBlueMatt:2019-08-rusty-block-download changing 20 files +1189 −124
  1. TheBlueMatt commented at 8:30 pm on August 29, 2019: member

    This is based on #15798.

    I’m increasingly worried about censorship and other attacks in the P2P network (see, eg, https://twitter.com/TheBlueMatt/status/1160620919775211520). While obviously making the net_processing p2p implementation more robust should be the focus (eg #15759), adding more redundancy in every way is also key. To avoid needless review burden on such systems, I think rust is an ideal candidate: it is wholly optional so we don’t just right to requiring Rust for Bitcoin Core, but building with it still provides real benefit, plus we can use compile-time checks to catch any overflows and memory corruption bugs to reduce review burden.

    This PR implements a simple API for Rust code to fetch headers and blocks and then implements a sample client that can sync the chain using a REST endpoint. As a demo, I provide a rest endpoint over cloudflare at http://cloudflare.deanonymizingseed.com/rest/ so you should be able to fully sync the chain (albeit very slowly) with bitcoind -blockfetchrest=http://cloudflare.deanonymizingseed.com/rest/ -connect=0

    There’s a few things to do before this is mergable, but I’d like feedback on the idea, and would like to also have a fully parallel, in rust, P2P net layer that is similarly simple and provides a nice redundant implementation in the future.

  2. fanquake added the label Brainstorming on Aug 29, 2019
  3. fanquake added the label Needs Conceptual Review on Aug 29, 2019
  4. fanquake added the label P2P on Aug 29, 2019
  5. TheBlueMatt force-pushed on Aug 29, 2019
  6. TheBlueMatt force-pushed on Aug 29, 2019
  7. TheBlueMatt force-pushed on Aug 29, 2019
  8. DrahtBot commented at 11:53 pm on August 29, 2019: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #17783 (util: Fix -norpcwhitelist, -norpcallowip, and similar corner case behavior by ryanofsky)
    • #17581 (refactor: Remove settings merge reverse precedence code by ryanofsky)
    • #17580 (refactor: Add ALLOW_LIST flags and enforce usage in CheckArgFlags by ryanofsky)
    • #17493 (util: Forbid ambiguous multiple assignments in config file by ryanofsky)
    • #17452 (test: update fuzz directory in .gitignore by jonatack)
    • #17398 (build: Update leveldb to 1.22+ by laanwj)
    • #17383 (Refactor: Move consts to their correct translation units by jnewbery)
    • #17227 (Qt: Add Android packaging support by icota)
    • #16722 (build: Disable warnings for leveldb subtree by default by hebasto)
    • #15606 ([experimental] UTXO snapshots by jamesob)
    • #15367 (feature: Added ability for users to add a startup command by benthecarman)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  9. TheBlueMatt force-pushed on Aug 30, 2019
  10. TheBlueMatt force-pushed on Aug 30, 2019
  11. in src/Makefile.am:538 in cf221a2410 outdated
    536@@ -529,7 +537,7 @@ nodist_libbitcoin_util_a_SOURCES = $(srcdir)/obj/build.h
    537 bitcoind_SOURCES = bitcoind.cpp
    538 bitcoind_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES)
    539 bitcoind_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
    540-bitcoind_LDFLAGS = $(RELDFLAGS) $(AM_LDFLAGS) $(LIBTOOL_APP_LDFLAGS)
    541+bitcoind_LDFLAGS = -Wl,--start-group $(RELDFLAGS) $(AM_LDFLAGS) $(LIBTOOL_APP_LDFLAGS)
    


    jonasschnelli commented at 7:38 am on August 30, 2019:

    fanquake commented at 7:44 am on August 30, 2019:
    @jonasschnelli Yes ld on macOS doesn’t support --start-group or --end-group. This was a bit of a hack to fix a circular dependency issue Matt was seeing on his Debian machine. If you want to compile on macOS you should just be able to drop the -Wl,--start-group additions, as that has been working for me. We’ll need to fix this properly soon.
  12. practicalswift commented at 12:46 pm on August 30, 2019: contributor

    Interesting idea: cautious concept ACK – more redundancy is good and it is hard to see how this would make us worse off in any way as long as the existing P2P logic is kept unchanged. (Intentionally not commenting on the implementation and more specifically the pros and cons of pulling in Rust code: that is an implementation detail that I’m sure will be discussed separately post concept ACK/NACK.)

    Censorship scenarios worth thinking about:

    Even if we do the over-REST block downloads over HTTPS it should be noted that the TLS handshake leaks the hostname of the connected to service to passive eavesdroppers. (Unless TLS 1.3 + Encrypted SNI is used: super rare in practice though.)

    The hostname is obviously also leaked when doing the DNS lookup. (Unless DoH (DNS over HTTPS) or DoT (DNS over TLS) is used: quite rare in practice.)

    This allows for active censorship attacks if the hostnames used are only used for Bitcoin purposes (such as in the cloudflare.deanonymizingseed.com example – I understand it is just used here as an example).

    (The same obviously goes for the IP-address used: if the IP-address connected to is only used for Bitcoin purposes then access to that IP-address could trivially be null routed without any non-Bitcoin users noticing.)

    I guess one way to achieve (partial) censorship resistance is to download over HTTPS URLs where the hostnames used are mostly used for non-Bitcoin purposes (such as github.com – just used here as an example: I’m sure we can come up with other examples). That would leak the hostname but since the hostname and the IP address is used for other purposes it would still be hard for an attacker to block it without having non-Bitcoin users complain.

    Another route would be to use TLS 1.3 + Encrypted SNI + DoH/DoT to connect to a service whose IP-address(es) are used mostly for non-Bitcoin purposes (such as CloudFlare which luckily supports both TLS 1.3 and Encrypted SNI). That wouldn’t leak the hostname and the IP-address would be hard to null route for an attacker without having non-Bitcoin users complain.

  13. TheBlueMatt commented at 6:33 pm on August 30, 2019: member

    Intentionally not commenting on the implementation and more specifically the pros and cons of pulling in Rust code: that is an implementation detail that I’m sure will be discussed separately post concept ACK/NACK

    Right, there’s a few things that would need to happen to get this in mergeable state (notably buildsystem things), so no need for that just yet.

    Censorship scenarios worth thinking about:

    Right, two things I’d like to support in the blocks-over-REST module: domain fronting avoids most of the gnarly issues with SNI/DNS lookups breaking your privacy, though sadly many providers don’t like it anymore, and Tor support, possibly even with some default onion service recommended as a backup cause privacy is less of a concern there.

    I’d also like to build a whole second P2P implementation (though with an explicit goal of being inefficient and simple, with the same trick as here where you wait 30 seconds before deciding to download anything to avoid wasting bandwidth), but first need to land some bit of rust downloading, and this seemed like an easier target.

  14. TheBlueMatt force-pushed on Aug 30, 2019
  15. kristapsk commented at 7:46 pm on August 30, 2019: contributor
    But why Rust not C++?
  16. TheBlueMatt commented at 5:59 am on August 31, 2019: member

    But why Rust not C++?

    Because we can trivially turn on compile-time checking preventing a ton of classes of vulnerabilities (especially for network-facing code), and runtime checking for many other classes (like buffer overflows, at least those the compiler can’t prove aren’t possible).

  17. fanquake commented at 6:01 am on August 31, 2019: member

    I’m fairly Concept ACK on this. It’s been in discussion for a little while, and I’m pretty sure it came up at the last Core dev meetup as well?

    But why Rust not C++? @kristapsk This PR isn’t really the place to have that discussion and is only going to distract from what’s actually trying to be achieved. Although Matt points out one big advantage:

    plus we can use compile-time checks to catch any overflows and memory corruption bugs to reduce review burden.

    If you’d like to discuss Rust inside Bitcoin Core further, I’d suggest taking a look at #15798. Here are Cory’s thoughts on Rust from that thread:

    Why rust? I don’t know. Maybe not. But I think it’s a fair assumption to say that Rust code will eventually end up in Bitcoin Core as the result of adding a new dependency. Adoption is happing quickly. So, I think it prudent to look ahead and not only be prepared, but actively help usher it in. Already I have a few things that I would like to work on and upstream to rust-lang to make our integration nicer, and I assume that more will be uncovered as it actually gets used.

    I built and tested that this functions using the instructions above. The backup downloader kicked off after a 10 minute wait

     0src/bitcoind -blockfetchrest=http://cloudflare.deanonymizingseed.com/rest/ -connect=0
     1
     22019-08-31T05:37:06Z nBestHeight = 311015
     32019-08-31T05:37:06Z init message: Loading P2P addresses...
     42019-08-31T05:37:06Z Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there
     52019-08-31T05:37:06Z Loaded 63409 addresses from peers.dat  159ms
     62019-08-31T05:37:06Z init message: Starting network threads...
     72019-08-31T05:37:06Z DNS seeding disabled
     82019-08-31T05:37:06Z net thread start
     92019-08-31T05:37:06Z msghand thread start
    102019-08-31T05:37:06Z addcon thread start
    112019-08-31T05:37:06Z init message: Done loading
    122019-08-31T05:47:10Z Synchronizing blockheaders, height: 313014 (~53.91%)
    132019-08-31T05:47:13Z Synchronizing blockheaders, height: 315013 (~54.23%)
    142019-08-31T05:47:15Z Synchronizing blockheaders, height: 317012 (~54.55%)
    152019-08-31T05:47:17Z Synchronizing blockheaders, height: 319011 (~54.87%)
    16< snip >
    172019-08-31T05:52:27Z Synchronizing blockheaders, height: 574883 (~97.17%)
    182019-08-31T05:52:34Z Synchronizing blockheaders, height: 576882 (~97.49%)
    192019-08-31T05:52:36Z Synchronizing blockheaders, height: 578881 (~97.81%)
    202019-08-31T05:52:39Z Synchronizing blockheaders, height: 580880 (~98.14%)
    212019-08-31T05:52:41Z Synchronizing blockheaders, height: 582879 (~98.46%)
    222019-08-31T05:52:44Z Synchronizing blockheaders, height: 584878 (~98.76%)
    232019-08-31T05:52:47Z Synchronizing blockheaders, height: 586877 (~99.09%)
    242019-08-31T05:52:49Z Synchronizing blockheaders, height: 588876 (~99.40%)
    252019-08-31T05:52:56Z Synchronizing blockheaders, height: 590875 (~99.73%)
    262019-08-31T05:52:59Z Synchronizing blockheaders, height: 592551 (~100.00%)
    272019-08-31T05:53:35Z Pre-allocating up to position 0x3000000 in blk00158.dat
    282019-08-31T05:53:35Z Pre-allocating up to position 0x100000 in rev00158.dat
    292019-08-31T05:53:35Z UpdateTip: new best=0000000000000000246039d52e90fb6b7cd442dda19861fb948c28e3a44f7a4e height=311016 version=0x00000002 log2_work=79.744641 tx=42644694 date='2014-07-16T13:51:22Z' progress=0.095646 cache=0.5MiB(3475txo)
    302019-08-31T05:53:38Z UpdateTip: new best=000000000000000014c552ced25645452a0b8632753e9288e2decedd273c5828 height=311017 version=0x00000002 log2_work=79.744747 tx=42645541 date='2014-07-16T14:10:17Z' progress=0.095648 cache=1.0MiB(7067txo)
    312019-08-31T05:53:40Z UpdateTip: new best=0000000000000000260ef522f4875fb0d5207e4e3ac97ce373a47209bcee52b6 height=311018 version=0x00000002 log2_work=79.744853 tx=42645990 date='2014-07-16T14:18:18Z' progress=0.095649 cache=1.2MiB(8621txo)
    322019-08-31T05:53:42Z UpdateTip: new best=0000000000000000007dabdebadfde7beaef31d13133600d0d88695146e2fac6 height=311019 version=0x00000002 log2_work=79.744959 tx=42646068 date='2014-07-16T14:19:50Z' progress=0.095649 cache=1.2MiB(8821txo)
    332019-08-31T05:53:45Z UpdateTip: new best=00000000000000001fc78d459e2480177ceea92d53f75cdbd856501b60cbdc3a height=311020 version=0x00000002 log2_work=79.745065 tx=42646933 date='2014-07-16T14:34:32Z' progress=0.095651 cache=1.6MiB(11938txo)
    34< snip >
    352019-08-31T06:00:41Z UpdateTip: new best=000000000000000017de2992c0de3e98d711253582fa7ee626645e395e4fd0e5 height=311697 version=0x00000002 log2_work=79.815121 tx=42899222 date='2014-07-20T16:20:21Z' progress=0.096217 cache=67.4MiB(497634txo)
    362019-08-31T06:00:44Z UpdateTip: new best=000000000000000022aeedbb76913a3666d997897c23efc8e92fe38ec79728a7 height=311698 version=0x00000002 log2_work=79.815222 tx=42899470 date='2014-07-20T16:25:14Z' progress=0.096217 cache=68.0MiB(502949txo)
    372019-08-31T06:00:47Z UpdateTip: new best=000000000000000027a68e70c9c02985173c56e8c8ce210c02eb975530d77f0d height=311699 version=0x00000002 log2_work=79.815323 tx=42899561 date='2014-07-20T16:27:06Z' progress=0.096218 cache=68.3MiB(505709txo)
    382019-08-31T06:00:51Z UpdateTip: new best=000000000000000038730cd937f594452e96b97a496afc03a84fc90a18119113 height=311700 version=0x00000002 log2_work=79.815424 tx=42900165 date='2014-07-20T16:40:48Z' progress=0.096219 cache=68.8MiB(509386txo)
    

    FWIW I’ve also rebased and added another commit for a more recent Rust onto https://github.com/fanquake/bitcoin/tree/theuni-with-rust-example-working-rebased. Will add another commit on top for the --start-group workaround. Given that you’ve dropped cbindgen here, I can also maintain a branch with that dropped.

  18. naumenkogs commented at 5:04 pm on September 3, 2019: member
    Concept ACK on adding more redundancy. Agree that Tor support is desirable, and also measures suggested by @practicalswift are important, but this feature even before those measures is already a big improvement (without trading any privacy comparing to the current threat model with unencrypted p2p I believe).
  19. MarcoFalke referenced this in commit 8e00a68552 on Sep 3, 2019
  20. sidhujag referenced this in commit b3459434c9 on Sep 3, 2019
  21. laanwj commented at 9:15 am on September 4, 2019: member
    Concept ACK!
  22. elichai commented at 6:47 am on September 5, 2019: contributor

    Could we try and use something like https://github.com/dtolnay/no-panic on parts of the code? (if we can, this will give us compile time guarantees of no panics)

    Another Idea, checking nullness at the boundaries and then use https://doc.rust-lang.org/std/ptr/struct.NonNull.html (can even use NonNull::new() to check for nullness)

    Other than that I like this and would love to review it when it’s reviewable :)

  23. TheBlueMatt commented at 4:53 pm on September 5, 2019: member
    I’d rather deliberately use panics so that we can enable overflow checking and catch the panics to kill the Rust thread(s) but leave the rest of Core running. Good call on NonNull, will update to use that!
  24. TheBlueMatt force-pushed on Sep 6, 2019
  25. TheBlueMatt force-pushed on Sep 6, 2019
  26. TheBlueMatt force-pushed on Sep 6, 2019
  27. TheBlueMatt force-pushed on Sep 6, 2019
  28. TheBlueMatt marked this as ready for review on Sep 6, 2019
  29. TheBlueMatt commented at 7:19 pm on September 6, 2019: member
    Aside from the final commit working around one last build issue, I believe this is ready for review. Doesn’t make sense to land for 19, but would be nice to land early in the cycle for 20.
  30. TheBlueMatt force-pushed on Sep 6, 2019
  31. theuni commented at 8:53 pm on September 6, 2019: member
    I’m taking a look at a proper fix for the circular dependency.
  32. TheBlueMatt force-pushed on Sep 6, 2019
  33. practicalswift commented at 5:13 am on September 7, 2019: contributor

    https://github.com/bitcoin/bitcoin/blob/3f1ceb95edeb9f58b0a702520a2c452012c63974/src/rusty/src/rest_downloader.rs#L254-L262

    Is there any particular reason that the Rust code is using a hand-rolled HTTP client implementation?

    https://github.com/bitcoin/bitcoin/blob/3f1ceb95edeb9f58b0a702520a2c452012c63974/src/rusty/src/rest_downloader.rs#L246-L248

    TLS support would be more or less free if we didn’t roll our own implementation? :-)

    TLS would be really nice to have since our goal with this PR is to increase censorship resistance.

  34. fanquake added this to the milestone 0.20.0 on Sep 7, 2019
  35. DrahtBot added the label Needs rebase on Sep 7, 2019
  36. TheBlueMatt force-pushed on Sep 7, 2019
  37. TheBlueMatt commented at 5:33 pm on September 7, 2019: member

    Rebased.

    Yea, probably makes sense to switch to a non-handrolled-HTTP-parser eventually, but: a) I haven’t yet had the time to look into existing rust ones to see if I can find anything suitable (ie doesn’t pull in a million dependencies, implying its probably not async/tokio based, and easy to review), which in my previous experience implies this may be hard, b) if we add any dependencies we have to have a process for vendoring the entire recursive dependency tree, which is gonna be its own hurdle, so I figured leave that for a follow-up PR.

  38. DrahtBot removed the label Needs rebase on Sep 7, 2019
  39. in .gitignore:142 in 14bc05537f outdated
    136@@ -137,3 +137,7 @@ db4/
    137 osx_volname
    138 dist/
    139 *.background.tiff
    140+
    141+# Rust
    142+src/rusty/Cargo.lock
    


    elichai commented at 9:10 pm on September 7, 2019:

    I think tracking changes to Cargo.lock might be a good thing. (especially if we will have some dependencies in the future)

    (I know it’s against the official recommendation)


    TheBlueMatt commented at 8:37 pm on September 8, 2019:
    I think we’ll vendor all our dependencies, so hopefully no need, but we can look into it more once we have a dependency.
  40. in src/init.cpp:1838 in 14bc05537f outdated
    1830@@ -1824,5 +1831,13 @@ bool AppInitMain(InitInterfaces& interfaces)
    1831         g_banman->DumpBanlist();
    1832     }, DUMP_BANS_INTERVAL * 1000);
    1833 
    1834+    // ********************************************************* Step 14: kick off backup block downloaders
    1835+
    1836+#if ENABLE_RUSTY
    1837+    for (const std::string& uri : gArgs.GetArgs("-blockfetchrest")) {
    1838+        rust_block_fetch::init_fetch_rest_blocks((const unsigned char*)uri.c_str());
    


    elichai commented at 9:20 pm on September 7, 2019:
    why not use the regular const char*? why cast it into an unsigned char pointer? You could then use rust’s CStr instead of manually looping and offsetting the pointer: https://doc.rust-lang.org/std/ffi/struct.CStr.html#method.from_ptr

    TheBlueMatt commented at 8:43 pm on September 8, 2019:
    Cause I didn’t know CStr existed :p
  41. in src/rusty/src/rest_downloader.rs:212 in 14bc05537f outdated
    207+
    208+#[no_mangle]
    209+pub extern "C" fn init_fetch_rest_blocks(uri: *const u8) -> bool {
    210+    let mut uri_vec = Vec::with_capacity(50);
    211+    for i in 0..10000 {
    212+        let c = unsafe { *uri.offset(i) };
    


    elichai commented at 9:28 pm on September 7, 2019:

    A. I’m not sure dereferencing a char* casted to *const u8, is safe. B. if you pass this as *const c_char and then use rust’s CStr (as I wrote above) you can replace these ~10 lines with:

    0let uri_str = match CStr::from_ptr(uri).to_str() {
    1    Ok(r) => r,
    2    Err(_) => return false,
    3};
    
  42. in src/rusty/src/rest_downloader.rs:60 in 14bc05537f outdated
    51+
    52+    Some((ssl, host, port, path))
    53+}
    54+
    55+#[test]
    56+fn test_split_uri() {
    


    elichai commented at 9:30 pm on September 7, 2019:
    currently the test is pretty small, but maybe still feature gate it with #[cfg(test)] so that it won’t be compiled normally?
  43. in src/rusty/src/rest_downloader.rs:15 in 14bc05537f outdated
     5+use std::panic::catch_unwind;
     6+
     7+use crate::bridge::*;
     8+use crate::await_ibd_complete_or_stalled;
     9+
    10+fn split_uri<'a>(uri: &'a str) -> Option<(bool, &'a str, u16, &'a str)> {
    


    elichai commented at 9:31 pm on September 7, 2019:
    Could you add a doc explaining what the function is doing?
  44. in src/rusty/src/rest_downloader.rs:25 in 14bc05537f outdated
    16+    };
    17+    let mut host_path = match uri_iter.next() {
    18+        Some(r) => r,
    19+        None => return None,
    20+    };
    21+    while !host_path.is_empty() && host_path.as_bytes()[0] == '/' as u8 {
    


    elichai commented at 9:39 pm on September 7, 2019:
    Maybe replace with host_path.starts_with("/")? (which also solves possible utf-8 problems)
  45. in src/rusty/src/rest_downloader.rs:26 in 14bc05537f outdated
    17+    let mut host_path = match uri_iter.next() {
    18+        Some(r) => r,
    19+        None => return None,
    20+    };
    21+    while !host_path.is_empty() && host_path.as_bytes()[0] == '/' as u8 {
    22+        host_path = &host_path[1..];
    


    elichai commented at 9:41 pm on September 7, 2019:
    Then you can replace this with host_path = host_path.trim_start_matches("/");

    TheBlueMatt commented at 3:06 am on September 9, 2019:
    Hmm, thats only available in 1.30. For now lets avoid it until we have a clear min version.

    elichai commented at 3:01 pm on September 9, 2019:
    So you have trim_left_matches() which predates it (is in rust since 1.0) https://doc.rust-lang.org/std/primitive.str.html#method.trim_left_matches
  46. in src/rusty/src/rest_downloader.rs:31 in 14bc05537f outdated
    22+        host_path = &host_path[1..];
    23+    }
    24+    let mut host_path_iter = host_path.splitn(2, "/");
    25+    let (host_port_len, host, port) = match host_path_iter.next() {
    26+        Some(r) if !r.is_empty() => {
    27+            let is_v6_explicit = !r.is_empty() && r.as_bytes()[0] == '[' as u8;
    


    elichai commented at 9:52 pm on September 7, 2019:
    Same, unless we have something that if !uri.is_ascii() return None; at the start of the function we should really try and not index into strings

    elichai commented at 9:54 pm on September 7, 2019:
    and btw you can do b'[' instead of '[' as u8 :)
  47. in src/rusty/src/rest_downloader.rs:231 in 14bc05537f outdated
    220+    // Sadly only non-SSL is supported for now
    221+    if let Some((false, _, _, _)) = split_uri(&uri_str) { } else { return false; }
    222+    std::thread::spawn(move || {
    223+        // Always catch panics so that even if we have some bug in our parser we don't take the
    224+        // rest of Bitcoin Core down with us:
    225+        let _ = catch_unwind(move || {
    


    elichai commented at 10:03 pm on September 7, 2019:
    Could we easily log a panic easily somehow? (although this introduces a lot of complications to pass the datadir location somehow, could we maybe use some tmp location just as a jump start?)

    TheBlueMatt commented at 3:04 am on September 9, 2019:
    Right, it would be really nice to be able to call LogPrintf from rust code, but I’d prefer to leave that to a future PR.
  48. in src/rusty/src/rest_downloader.rs:226 in 14bc05537f outdated
    216+    let uri_str = match String::from_utf8(uri_vec) {
    217+        Ok(r) => r,
    218+        Err(_) => return false,
    219+    };
    220+    // Sadly only non-SSL is supported for now
    221+    if let Some((false, _, _, _)) = split_uri(&uri_str) { } else { return false; }
    


    elichai commented at 10:06 pm on September 7, 2019:

    I think this should be reversed to check if SSL is true. Because if this returns None we might as well also return false and finish this early, because we unwrap this few lines after

    Better yet, run it once and check for if ssl {return False;} so that we won’t need to run this twice(not that it really matters that much…)


    TheBlueMatt commented at 3:03 am on September 9, 2019:
    Hmm? The pattern hers is if let Some() {} else { return; }. It reads a bit weird but its the easiest way to write it out.
  49. in src/init.cpp:375 in 14bc05537f outdated
    370@@ -367,6 +371,9 @@ void SetupServerArgs()
    371 #endif
    372     gArgs.AddArg("-blockreconstructionextratxn=<n>", strprintf("Extra transactions to keep in memory for compact block reconstructions (default: %u)", DEFAULT_BLOCK_RECONSTRUCTION_EXTRA_TXN), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
    373     gArgs.AddArg("-blocksonly", strprintf("Whether to reject transactions from network peers. Transactions from the wallet, RPC and relay whitelisted inbound peers are not affected. (default: %u)", DEFAULT_BLOCKSONLY), ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
    374+#if ENABLE_RUSTY
    375+    gArgs.AddArg("-blockfetchrest=<uri>", "A REST endpoint from which to fetch blocks. Acts as a redundant backup for P2P connectivity", ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
    


    elichai commented at 10:18 pm on September 7, 2019:
    maybe add an example of how the uri should be formatted? (especially because currently there’s no feedback from split_uri)

    TheBlueMatt commented at 3:02 am on September 9, 2019:
    I went ahead and added the deanonymizingseed.com one, if folks complain I can use example.com.
  50. in src/rusty/src/rest_downloader.rs:243 in 14bc05537f outdated
    232+
    233+                let mut stream;
    234+                macro_rules! reconnect {
    235+                    () => { {
    236+                        stream = match TcpStream::connect_timeout(&match (host, port).to_socket_addrs() {
    237+                            Ok(mut sockaddrs) => match sockaddrs.next() { Some(sockaddr) => sockaddr, None => continue 'reconnect },
    


    elichai commented at 10:31 pm on September 7, 2019:
    Ideally we would match against the exact error match so that if it returns something like InvalidData then there’s no need to keep the loop running. (so stop unless it’s a timeout, broken pipe, would block etc.)

    TheBlueMatt commented at 3:00 am on September 9, 2019:
    Ehh, I’d rather keep polling. Better to keep polling so that when the rest endpoint fixes their misconfiguration (eg its returning a 500 or 404 or whatever) the downloader picks back up.
  51. in src/rusty/src/rest_downloader.rs:274 in 14bc05537f outdated
    263+                        // We got exactly the header we requested, ie it is *also* the tip for the
    264+                        // remote node, go on to block fetching!
    265+                        break 'header_sync;
    266+                    } else if headers.len() == 0 {
    267+                        let genesis_tip = BlockIndex::genesis();
    268+                        if genesis_tip == provider_state.get_current_best() { // Maybe they're on a different network entirely?
    


    elichai commented at 10:40 pm on September 7, 2019:
    Add a comment that this should only ever trip on the second connection?

    TheBlueMatt commented at 2:59 am on September 9, 2019:
    I mean that seems pretty obvious given the code?
  52. in src/rusty/src/rest_downloader.rs:275 in 14bc05537f outdated
    264+                        // remote node, go on to block fetching!
    265+                        break 'header_sync;
    266+                    } else if headers.len() == 0 {
    267+                        let genesis_tip = BlockIndex::genesis();
    268+                        if genesis_tip == provider_state.get_current_best() { // Maybe they're on a different network entirely?
    269+                            continue 'reconnect;
    


    elichai commented at 10:41 pm on September 7, 2019:
    If we’re on completely separate chains why reconnect?

    TheBlueMatt commented at 2:58 am on September 9, 2019:
    The server may also be broken and be returning empty responses, so best to keep trying.
  53. in src/rusty/src/bridge.rs:105 in 14bc05537f outdated
    100+    }
    101+
    102+    pub fn hash(&self) -> [u8; 32] {
    103+        let hashptr = unsafe { rusty_IndexToHash(self.index) };
    104+        let mut res = [0u8; 32];
    105+        unsafe { std::ptr::copy(hashptr, (&mut res).as_mut_ptr(), 32) };
    


    elichai commented at 10:51 pm on September 7, 2019:
    res.as_mut_ptr() should also work
  54. elichai commented at 10:57 pm on September 7, 2019: contributor

    Reviewed all the rust code and FFI except read_http_resp which haven’t had time to review it with the HTTP specs.

    Don’t know enough about bitcoin’s P2P to comment if the logic itself makes a lot of sense. Didn’t try to run it yet.

  55. TheBlueMatt force-pushed on Sep 8, 2019
  56. TheBlueMatt force-pushed on Sep 9, 2019
  57. TheBlueMatt force-pushed on Sep 9, 2019
  58. in src/rusty/src/rest_downloader.rs:222 in 2cc8b15de5 outdated
    211+}
    212+
    213+#[no_mangle]
    214+pub extern "C" fn init_fetch_rest_blocks(uri: *const c_char) -> bool {
    215+    let uri_str: String = match unsafe { CStr::from_ptr(uri) }.to_str() {
    216+        Ok(r) => r.to_string(),
    


    elichai commented at 3:04 pm on September 9, 2019:
    Great catch :) that way we won’t have a dangling pointer at some point
  59. TheBlueMatt force-pushed on Sep 9, 2019
  60. TheBlueMatt force-pushed on Sep 28, 2019
  61. laanwj added the label Feature on Sep 30, 2019
  62. TheBlueMatt force-pushed on Sep 30, 2019
  63. TheBlueMatt commented at 11:57 pm on September 30, 2019: member
    Now based on #16834 instead of the other way around, but otherwise this should be mostly good to go!
  64. TheBlueMatt force-pushed on Oct 6, 2019
  65. TheBlueMatt force-pushed on Oct 7, 2019
  66. DrahtBot added the label Needs rebase on Oct 9, 2019
  67. TheBlueMatt force-pushed on Oct 10, 2019
  68. TheBlueMatt force-pushed on Oct 16, 2019
  69. TheBlueMatt force-pushed on Oct 16, 2019
  70. DrahtBot removed the label Needs rebase on Oct 16, 2019
  71. DrahtBot added the label Needs rebase on Oct 26, 2019
  72. TheBlueMatt force-pushed on Oct 26, 2019
  73. DrahtBot removed the label Needs rebase on Oct 26, 2019
  74. TheBlueMatt force-pushed on Oct 29, 2019
  75. TheBlueMatt force-pushed on Nov 3, 2019
  76. TheBlueMatt force-pushed on Nov 3, 2019
  77. TheBlueMatt force-pushed on Nov 3, 2019
  78. DrahtBot added the label Needs rebase on Nov 8, 2019
  79. TheBlueMatt force-pushed on Nov 12, 2019
  80. DrahtBot removed the label Needs rebase on Nov 12, 2019
  81. DrahtBot added the label Needs rebase on Nov 19, 2019
  82. TheBlueMatt force-pushed on Nov 19, 2019
  83. DrahtBot removed the label Needs rebase on Nov 20, 2019
  84. DrahtBot added the label Needs rebase on Nov 20, 2019
  85. TheBlueMatt force-pushed on Nov 21, 2019
  86. DrahtBot removed the label Needs rebase on Nov 21, 2019
  87. DrahtBot added the label Needs rebase on Dec 4, 2019
  88. Adds a rust library to bitcoin to demonstrate linking and building rust code.
    The demonstration library compiles a rust hello world example and auto-generates
    a header which can be included in C++ code.
    
    Co-Authored-By: Jeremy Rubin <j@rubin.io>
    Co-Authored-By: Cory Fields <cory-nospam-@coryfields.com>
    Various changes by: Matt Corallo <git@bluematt.me>
    100ce28f5a
  89. build: show rust enabled in configure output 4830d9e0c8
  90. build: gitignore src/rusty_test 828a9822a0
  91. Add basic ffi bindings for Rust header download
    Also, break circular dependency with a new helper lib courtesy of
    Cory Fields <cory-nospam-@coryfields.com>.
    9bd5f6fba0
  92. Add headers-over-DNS fetcher in Rust f0763b28ea
  93. Limit total memory allocated by all Rust code to 128MB eb21b69810
  94. Enable rustc building + testing on xenial Tsan Travis job 94051addd9
  95. Move CNode's FindNextBlocksToDownload state into its own strut
    This starts the process of moving the last bits of
    validation-critical logic out of net_processing - the fallback
    logic of which blocks to download is incredibly critical to
    validation and would likely also be used for any parallel block
    sync systems. Further, assumeutxo will require additional ability
    to select blocks for download, which is a validation-specific
    concept and shouldn't really live in net_processing.
    
    This moves a few responsibilities to the FindNextBlocksToDownload
    callsite instead of passing in a CNodeId, but otherwise has no
    functional changes.
    ccbd928cf5
  96. Make FindNextBlocksToDownload a member func on BlockProviderState 0696ab1bab
  97. Make FindNextBlocksToDownload not use mapBlockIndex for in-progress
    This makes it more general than just net_processing
    6fc556d1f8
  98. Move BlockProvider to validation.{h,cpp} from net_processing e513835e20
  99. Add ffis to access BlockProviderState and hand blocks to C++ 1361107c89
  100. Add a rust-based backup over-REST block downloader 2c0dda33c5
  101. TheBlueMatt force-pushed on Dec 5, 2019
  102. DrahtBot removed the label Needs rebase on Dec 5, 2019
  103. DrahtBot commented at 11:43 pm on January 7, 2020: member
  104. DrahtBot added the label Needs rebase on Jan 7, 2020
  105. luke-jr commented at 5:08 am on January 18, 2020: member

    would be nice to land early in the cycle for 20.

    Rust is still a concept NACK

  106. TheBlueMatt commented at 7:48 pm on March 5, 2020: member
    Closing due to lack of interest.
  107. TheBlueMatt closed this on Mar 5, 2020

  108. humbleDasher referenced this in commit 2996f593ef on Nov 13, 2021
  109. pravblockc referenced this in commit 0b4f279949 on Nov 18, 2021
  110. DrahtBot locked this on Feb 15, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-22 00:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me