This uses boost::filesystem::create_hard_link to hard-link the pre-0.8 blkNNNN.dat files to blocks/blkNNNNN-1.dat
A hard link is the semantics we want: a copy would use twice the disk space, and a move would mean you have to re-download blocks if you switch back to 0.7.
The hard link failing is a soft error-- in that case, you just re-download the blocks.
According to my research, this should work on Windows, unless you're running a FAT32 filesystem.
Looks good to me. Note, however, that ultraprune currently has problems in some cases with hard linked files. IMO those should be fixed regardless, so not a blocker for this.
744 | + printf("Hardlinked %s -> %s\n", source.c_str(), dest.c_str()); 745 | + linked = true; 746 | + } catch (filesystem::filesystem_error & e) { 747 | + // Note: hardlink creation failing is not a disaster, it just means 748 | + // blocks will get re-downloaded from peers. 749 | + printf("Error hardlinking blk%04u.dat : %s\n", i, e.what());
IMHO this could be an InitWarning() and the message should be a little more user friendly and translatable.
I don't want users to be told that hardlinking failed-- most end-users won't know what a hard-link is, and if it DOES fail I don't expect them to do anything (they'll just fall-back to downloading the whole chain again).
This is definitely the easiest solution imaginable, assuming it works for Windows users.
Importing instead of reindexing is probably sometimes a better choice (it helps with heavily fragmented block files, for example, and results in the intended smaller files), but I suppose those cases can be handled manually.
Automatic sanity-testing: FAILED BUILD/TEST, see http://jenkins.bluematt.me/pull-tester/662bdaa9c36c0b0da29edd82fa9bebbee9a1ecaa for binaries and test log.
This could happen for one of several reasons:
739 | + filesystem::path source = GetDataDir() / strprintf("blk%04u.dat", i); 740 | + if (!filesystem::exists(source)) break; 741 | + filesystem::path dest = blocksDir / strprintf("blk%05u.dat", i-1); 742 | + try { 743 | + filesystem::create_hard_link(source, dest); 744 | + printf("Hardlinked %s -> %s\n", source.c_str(), dest.c_str());
source.string().c_str(), etc. if you want to be compatible with older filesystem boost lib.
Thanks @sipa -- updated to use string().c_str() and rebased.
Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/f4445f9982a760869c430f3d4b1302f7eb509bd8 for binaries and test log.
If the linking fails, can it copy or move the file instead rather than download from the genesis block?
I wonder how difficult it would be to have the code just use the old blk000?.dat files directly when they exist.
The reason to move to a new naming scheme was:
Hardlinking removes that second advantage, but by the time we're actually going to support pruning, I suppose many setups will not be using an upgraded pre-0.8 datadir anyway.
Regarding using the old files: currently, the block index encodes positions in files as filenum+byteoffset. I suppose using some trick like making negative filenums refer to old files, we can encode both, but the reindexing/loading code should be checked carefully... a dirty hack in any case.
I was thinking more like "if blk000?.dat exist, start new files with the next number. when reading, if the blk0000?.dat is missing, check blk000?.dat" so the files would be stored as the same filenum :)
In case hardlinking fails, I don't think there's any harm in adding the found source files automatically to the list of files to be -loadblock='ed. They will be downloaded anyway otherwise, and loadblock happens in a background thread anyway, so it can only speed things up.
RE: doing something smart if the hardlink fails:
I'm not planning on testing that, and I don't believe in adding untested code even if it should OBVIOUSLY work.
If somebody wants to implement and test that as a separate pull, great, but it is not on my priority list because it would benefit so few (maybe zero) people.
Test plan with testing bounties: https://github.com/gavinandresen/QA/blob/master/HardLinksUpgrade.md
First bounty claimed.
@gavinandresen Can you explain in the "build binaries yourself" case which branch we need to build? The main "bitcoin" repository does not appear to contain 0.8 related stuff.
The hard-linking does not appear to work on my system (Ubuntu 12.10 64bit). In two different cases, my .bitcoin directory grew from 6.3 GB to nearly 12 GB and my df (disk free) significant reduced. The files in blocks are of different size than the blk*.dat files and have different inode numbers.
The whole process tool several hours.
@sipa 2013-01-10 22:20:55 Bitcoin version v0.7.1-297-g429915b-beta (2013-01-06 07:26:43 -0800)
Maybe I should mention that I was using the Ubuntu package (0.7.2-quantal2) before. which is a little bit different to the official build in its dependencies.
@Goonie I'm afraid I confused you. You built from git head, it seems. This page is a pull request: a change requested to be merged in git head, but not yet there. The branch to pull from is in gavin's repository (https://github.com/gavinandresen/bitcoin-git.git), in branch blkfile_upgrade.
It's also included in my 'turbo' branch (https://github.com/sipa/bitcoin.git, branch turbo), together with several other pull requests.
@sipa Ok, this time I'm using v0.7.1-269-gf4445f9-beta. It looks like its using hard links in the blocks directory. It managed to reindex about 211700 blocks in 80 minutes. Then suddenly the status bar text disappeared. From the icons on the lower right and from debug.log I can tell its still accepting blocks, but at a relatively low rate of 1 block per second (still better than before).
@sipa Another issue: the "Show QR code" context menu option silently fails. I remember it was working before because I was using QR codes to scan with Bitcoin Wallet.
@Goonie The GUI is using an estimated block count, which it get's from the connected nodes and the last checkpoint, to be able to compute a "Blocks left" number and to display the progressbar. If you have more blocks than the estimated block count there is nothing to predict anymore and so the progressbar get's hidden.
ACK