Added Blockchain Bootstraping Tutorial #3747

super3 commented at 1:55 AM on February 26, 2014: contributor

in doc/bootstrap.md:None in f2aa10ffb3 outdated

  58 | +Now start the Bitcoin client software. It should show "Importing blocks from disk" like the image below. 
  59 | +![Fig5](/img/bootstrap5.png)
  60 | +Wait until the import finishes. The client will download the last days not covered by the import. Congratulations you have successfully imported the blockchain!
  61 | +
  62 | +### Is this safe?
  63 | +Yes, the above method is safe. The download contains only raw block chain data and the client verifies this on import. Do not download the blockchain from unofficial sources, especially if they provide blk00&#42;&#42;&#42;.dat files. These files are not verified and can contain malicious edits. There is some protection against this by means of blockchain checkpoints, but be careful with unofficial downloads.

sipa commented at 2:04 AM on February 26, 2014:

Please don't call checkpoints a security feature, that's a misconception. They're only there to make skipping signature checks safe in the current synchronization mechanism, but they will not help against importing an invalid pre-indexed block chain database.

super3 commented at 2:11 AM on February 26, 2014:

Addressed. Anything else?

laanwj commented at 6:32 AM on February 26, 2014: member

https://github.com/super3/bitcoin/blob/bootstrap-doc2/doc/bootstrap.md

Seems the images don't work inline. You likely need to use a relative path, not an absolute one.

super3 commented at 6:34 AM on February 26, 2014: contributor

@laanwj Fixed.

laanwj commented at 6:46 AM on February 26, 2014: member

Overall, looks good.

I think it should be preferred to use the direct magnet link that @jgarzik mentions in the bitcointalk topic (inside the signed area) instead of a torrent file from http sourceforge. All torrent clients support magnet links these days. Though you can keep the sourceforge description as fallback in case the magnet link doesn't work.

laanwj commented at 7:00 AM on February 26, 2014: member

Also we should likely make the contents of the img/ directory is part of installers and distributions ( https://github.com/bitcoin/bitcoin/blob/master/Makefile.am#L19 ).

Hmm I just realized that nothing on the user's computer can probably show .md files... At some point we want to add a document converter to an OS-dependent format in the build/install process.

super3 commented at 7:05 AM on February 26, 2014: contributor

I'm making the assumption that most people are reading these docs through the repo. If we want an OS-dependent format I say just parse the Markdown to .html and call it a day.

laanwj commented at 7:07 AM on February 26, 2014: member

It's a new problem though, we don't have any other .mds with images, the others work fine in a notepad or such :) But doesn't have to be solved in this pull, reading it through github works fine.

super3 commented at 7:11 AM on February 26, 2014: contributor

Found some time to contribute so there will probably be quite a few more .mds with images as I make improvements to the docs. Will keep that in mind.

I'll add the magnet link tomorrow. Thanks for the feedback.

laanwj commented at 7:44 AM on February 26, 2014: member

An empty wallet. This is very important as pending transactions in an existing wallet can get lost!

This process cannot lose any transactions in the wallet (nothing ever deletes from the wallet apart from salvagewallet / zapwallettxes). I always suggest backing up the wallet before doing anything like manually copying files into/out of the Bitcoin folder, but this is overstated.

If you were to restart block synchronization from the beginning (ie by adding a step to remove the current blocks and blockchain directories) transactions later in the chain will be temporarily invisible/unconfirmed. But they will not be lost.

super3 commented at 9:04 PM on February 27, 2014: contributor

@laanwj Fixed.

Delta1977 commented at 9:25 PM on February 27, 2014: contributor

you can describe it as a rescan is needed in the case of an used wallet.
can you add a point : a checksum check of the bootstrap.dat ?

super3 commented at 10:21 PM on February 27, 2014: contributor

This guide was intended for bootstrapping a new wallet. That would require additional steps and instructions for manipulating the QT client. If that necessary I'd rather get this pull fixed up and in before adding additional stuff.
If the torrent is updated then the checksum changes. Which means every time @jgarzik updates the torrent, we will have to update this guide as well. I figure most people won't even bother checking.

sipa commented at 1:33 PM on February 28, 2014: member

Meh about checksum check. Worst case you wasted some bandwidth.

super3 commented at 2:56 AM on March 1, 2014: contributor

Any other suggestions? Hoping for ACKs.

int03h commented at 3:01 AM on March 1, 2014: none

I can't support this because the blockchain should be self maintained and checkpointed. Its a great guide.. dont get me wrong .. but is this something we should encourage ?

super3 commented at 3:08 AM on March 1, 2014: contributor

@int03h Another Bitcoin Atlanta guy. Hello!

I'll copy what @jgarzik had to say "Not quite... this data is the precisely the same data you see on the P2P network.

The client validates bootstrap.dat data to the same level it validates data downloaded from the P2P network.

If all checkpointing code is disabled, this data remains fully valid and useful."

int03h commented at 3:13 AM on March 1, 2014: none

howdee s3 ... ( I am in buckhead ) .. close to bitpay .. we should get coffee .. or something stronger :) .. I am fundamentally against torrents because it devalues the "process of bitcoin". It somehow intimates that you can download the whole blockchain like movie and own it all .. I know its full retard .. but people are like that ;)

super3 commented at 3:17 AM on March 1, 2014: contributor

I'm near the dome. I'm off in two weeks. Lets do it.

I mean Bitcoin-Qt is not really for the end user anymore. I doubt the end user would dig in here to find this tutorial. This is more for people who might be running Bitcoin Armory, or want to play around with rpc commands on a full node and not wait 3 days for a sync.

int03h commented at 3:37 AM on March 1, 2014: none

yes.. lets get a beer! :) :+1:

16GB is nothing these days. Maybe its really not a big deal ?

int03h commented at 3:38 AM on March 1, 2014: none

http://www.bittorrent.com/sync dying to paste this into a thread somewhere .. here it is .. its a "Private" cloud ... using torrents..

super3 commented at 3:49 AM on March 1, 2014: contributor

@int03h Indeed! 16 GB is not that big of a deal, but when you have to do it in order, and block by block it will take a long time. Until we have header first implemented this is the only real way to do a blockchain sync that doesn't take 48 hours.

int03h commented at 4:04 AM on March 1, 2014: none

I always have a spare node handy to sync from. It only takes about 4 hours if I addnode that node ..local node discovery is terrible. I really really wish we could get the peer discovery fixed instead of making people torrent the blockchain. ( that said .. I have a torrent to a checkpoint too on bitsync ! :)

laanwj commented at 7:48 AM on March 1, 2014: member

NACK on adding a checksum check. Let's not make this an unnecessarily long process (everyone hates long processes). And bitcoind/-qt validates it itself anyway (as if it came from the network), there's nothing to win for an attacker by mangling it.

ACK on document itself

super3 commented at 3:48 PM on March 6, 2014: contributor

Squashed. If I could get some more ACKs that would be great.

in doc/bootstrap.md:None in ac23d02dc7 outdated

   0 | @@ -0,0 +1,64 @@
   1 | +### Bootstrap the Blockchain Synchronization
   2 | +
   3 | +Normally the Bitcoin client will download the transaction and network information, called the blockchain, in the network by syncing with the other clients. This can be a process that can take multiple days as the [Bitcoin block chain](https://blockchain.info/charts/blocks-size) has grown to more than 15 gigabytes, and is growing almost a gigabyte every month. Luckily there is a safe and fast way to speed up this process. We’ll show you how to bootstrap your blockchain to bring your client up to speed in just a few simple steps.

Michagogo commented at 3:52 PM on March 6, 2014:

"in the network"? Sounds a bit odd to me. Maybe "from"?

in doc/bootstrap.md:None in ac23d02dc7 outdated

   6 | +
   7 | +A fresh install of the Bitcoin client software.
   8 | +
   9 | +### Download the blockchain via Bittorent
  10 | +
  11 | +Jeff Garzik, Bitcoin core developer, offers an [torrent file](https://bitcointalk.org/index.php?topic=145386.0) for bootstrapping purposes that is updated often. Bittorrent is a protocol that speeds up the downloading of large files by using the other clients in the network. Example free and safe open-source clients are [Deluge](http://deluge-torrent.org/) or [QBittorent](http://www.qbittorrent.org/). A guide to installing and configuring the torrent clients can be found [here](http://dev.deluge-torrent.org/wiki/UserGuide) for Deluge and [here](http://qbforums.shiki.hu/) for QBittorent. A further in-depth tutorial on Bittorent can be found [here](http://www.howtogeek.com/howto/31846/bittorrent-for-beginners-how-get-started-downloading-torrents/).

Michagogo commented at 3:53 PM on March 6, 2014:

Any specific reason to endorse these two clients? There are many out there, and I wouldn't say they're hard to find.

super3 commented at 5:33 PM on April 10, 2014:

Not really, but couldn't hurt to include some relevant resources.

in doc/bootstrap.md:None in ac23d02dc7 outdated

  33 | +    
  34 | +The download page should look like this, with a countdown to the download. If it does not work directly click the download. Save the file to a folder you use for downloads.
  35 | +![Fig3](img/bootstrap3.png)
  36 | +
  37 | +### Importing the blockchain
  38 | +Exit the Bitcoin Client software if you have it running. Be sure not to have an actively used wallet in use. Now we are going to copy the download of the blockchain to the Bitcoin client data directory. You should run the client software at least once so it can generate the data directory. Copy the downloaded bootstrap.dat file into the Bitcoin data folder.

Michagogo commented at 3:55 PM on March 6, 2014:

No need to have the datadir already existing... The user can create an empty directory and drop the file in without a problem.

super3 commented at 5:31 PM on April 10, 2014:

True, but this adds unnecessary steps.

in doc/bootstrap.md:None in ac23d02dc7 outdated

  40 | +**For Windows users:**
  41 | +Open explorer, and type into the address bar:
  42 | +
  43 | +	%APPDATA%\Bitcoin
  44 | +    
  45 | +This will open up the data folder. It should look like the image below. Copy over the blockchain.dat from your download folder to this directory.

Michagogo commented at 3:55 PM on March 6, 2014:

What's blockchain.dat?

laanwj commented at 7:21 AM on March 14, 2014:

Looks like a typo that should be bootstrap.dat

in doc/bootstrap.md:None in ac23d02dc7 outdated

  59 | +Now start the Bitcoin client software. It should show "Importing blocks from disk" like the image below. 
  60 | +![Fig5](img/bootstrap5.png)
  61 | +Wait until the import finishes. The client will download the last days not covered by the import. Congratulations you have successfully imported the blockchain!
  62 | +
  63 | +### Is this safe?
  64 | +Yes, the above method is safe. The download contains only raw block chain data and the client verifies this on import. Do not download the blockchain from unofficial sources, especially if they provide blk00&#42;&#42;&#42;.dat files. These files are not verified and can contain malicious edits.

Michagogo commented at 3:56 PM on March 6, 2014:

Huh? Bootstrap.dat is exactly as safe/unsafe as blk*.dat files.

laanwj commented at 4:11 PM on March 6, 2014:

The overall advice to not copy the entire bitcoin directory from others is sound. To be precise it's the data in the chainstate directory that is 'trusted' and should not be copied from other people. Copying just the blocks files should be OK, but recommending to use just the bootstrap.dat makes sense to newbies.

Michagogo commented at 5:03 PM on March 6, 2014:

Yes, but there isn't supposed to be any "official source". That doesn't say you shouldn't copy the entire datadir, it says not to download the blockchain from "unofficial sources", and it emphasizes specifically not to accept blk00_**.dat files. Those files are perfectly safe, and also _\ is redundant.

super3 commented at 5:35 PM on April 10, 2014:

Just removed the "safe" part.

in doc/bootstrap.md:None in ac23d02dc7 outdated

  55 | +
  56 | +	~/.bitcoin/
  57 | +    
  58 | +### Importing the blockchain
  59 | +Now start the Bitcoin client software. It should show "Importing blocks from disk" like the image below. 
  60 | +![Fig5](img/bootstrap5.png)

Michagogo commented at 3:57 PM on March 6, 2014:

Only relevant to Bitcoin Core GUI, not the Daemon.

laanwj commented at 4:06 PM on March 6, 2014:

This tutorial is aimed at the kind of users that will likely be using the GUI

laanwj commented at 7:31 AM on April 4, 2014: member

@super3 If you have some time, please do the final changes (like the blockchain.dat/bootstrap.dat typo) so that this can be merged.

super3 commented at 7:51 AM on April 4, 2014: contributor

Got a bit busy. I'lll try to fix it up over the weekend.

super3 commented at 5:35 PM on April 10, 2014: contributor

Did I miss anything before I squash?

Added Blockchain Bootstraping Tutorial

Per @laanwj request in #3724

Fixing Checkpoints Comment

Fixed Image Paths

Removed requirement.

Update Torrent Links.

Update Links 2

Link would not work.

2 Small Fixes

Changed in to from

@laanwj suggestion on safety

b698417fc8

super3 commented at 6:34 AM on April 12, 2014: contributor

Fixed and squashed.

BitcoinPullTester commented at 6:51 AM on April 12, 2014: none

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/b698417fc87a43adfefdeaa01c30f658cf2487cf for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

laanwj referenced this in commit 76db4e235d on Apr 12, 2014

laanwj merged this on Apr 12, 2014

laanwj closed this on Apr 12, 2014

super3 deleted the branch on Apr 12, 2014

DrahtBot locked this on Sep 8, 2021