Leveldb as subtree #2616

I've recreated a branch representing the original LevelDB from which Bitcoin based its copy (it was version 1.7.0 from October 2012), and reapplied the commits from Bitcoin using cherry-pick. It stands in its own repository now so that anyone can use LevelDB with the fixes that Bitcoin applies. It has a reconstituted commit log so that it is a "true" fork from the original Google repository, and that regular Git commands will work.

This pull request removes the detached leveldb sources from the Bitcoin repository and adds them back using "git-subtree", which retains the commit log for the entire LevelDB branch from which Bitcoin started its fork. Using this method it will be possible to both submit Bitcoin's LevelDB changes back to the upstream (which resides on Google code) and cherry-pick or merge newer branches from LevelDB back into the Bitcoin code base.

All of this was done using the 'git-subtree' command (instead of git-submodule which I hate).

The repository containing Bitcoin's version of LevelDB is located here, in the branch 'bitcoin-fork' https://github.com/vinniefalco/LevelDB

Now, anyone can use LevelDB and benefit from the changes that Bitcoin provided. Hopefully, this will make it easier for Bitcoin developers to merge improvements and fixes from the upstream LevelDB back into Bitcoin.

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/80744b3beeab9b25be46e122e6648ba9a0c46e5f for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

I see the advantages of this. But one question, will the subtree automatically be cloned when cloning the bitcoin repository? Or does this add extra steps or even need extra tools?

Yes, it will be cloned. The files in the subtree are plain, ordinary repository files. You can modify and commit them, rename, them, delete them, etc... What git-subtree does is create a special 'merge' commit that ties in the commit log from the source repository (https://github.com/vinniefalco/LevelDB in this case) so that later on, you can pull a newer version of the subtree. And you can also 'split' any changes that are made to the LevelDB files and push them back to the original repository. It's the ideal alternative to the mess that is git-submodule. No tools or steps are needed (although note that git-subtree is part of the official Git distribution now).

I plan on integrating Bitcoin's LevelDB changes into the latest tip of the LevelDB master branch so once that's ready, it will be possible to bring Bitcoin's version of LevelDB up to date. Here's a little promo for git-subtree:

http://makingsoftware.wordpress.com/2013/02/16/using-git-subtrees-for-repository-separation/

We considered putting the LevelDB code in a separate repository, before switching to it. I believe people complained about submodules back then, though I can't remember the reasons. I'm using submodules in another project, and though I'm not too comfortable with it, it seems to work.

I don't know about subtrees - will read - but if it requires separate tools (at least Ubuntu doesn't seem to have it), it will complicate our deterministic build environment.

EDIT: as subtree seems to leave a copy of the code inside the including project, it shouldn't cause any build issues.

No separate tools are needed, the files are ordinary files. Use of "git-subtree" will only be required if someone wants to pull a newer branch from a LevelDB repository, to bring Bitcoin's version of it up to date. Or if someone wants to split Bitcoin's changes out so they can submit a pull request to the LevelDB repository. "git-subtree" is part of Git now so if it's missing in someone's Ubuntu they only need install the latest Git (git-subtree is just one shell script). But this is only if they want to do the two things I mentioned - for normal development it is not needed.

Nifty! I think we should pull this soon after releasing 0.8.2.

Note that this pull request can be skipped if you accept number #2702 , which does everything this one does and also brings LevelDB up to 1.10.0, and fixes a deadlock scenario on Windows.

Here's a decent comparison highlighting the benefits of git-subtree over submodules: http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/

A one-sided comparison...

This blog post accurately summarizes my bad experiences with git submodules. Note that I have used both so I am speaking directly from experience:

http://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/

Just to understand, are we then using a location for all this, which is then not inside bitcoin/bitcoin anymore?

@Diapolo No. git-subtree does not store the location of the repo. The foreign repo is only accessed at the time you issue the appropriate git-subtree command. Other than when you are running a git-subtree command, the files are normal repository files like everything else. There is nothing magical or special about them, other than they have a few extra lines in the commit message that help the git-subtree script figure things out. Git-subtree only does things that you could do manually yourself, the structure of the repository is not hacked.

The reason that I have created my own LevelDB repo (at https://github.com/vinniefalco/LevelDB) is so that I could recreate the "bitcoin-fork" branch, which includes all of the changes made by the Bitcoin developers replayed on top of the commit log from the original LevelDB (stored at google code). You can see this recreated history here: https://github.com/vinniefalco/LevelDB/commits/bitcoin-fork

Notice how sipa's changes are all there, and if you keep going back in time you will find the place where Bitcoin forked LevelDB. When LevelDB was added to bitcoin, they did it by just manually plopping the files into the source code hierarchy and making a commit. By doing so, the history of LevelDB was not carried over and therefore it is difficult to submit changes back to the upstream (the google code repo in this case).

What I have done is to recreate the commit log including bitcoin's changes, and turn it into a proper branch called 'bitcoin-fork' located in my repository (URL above). Then I used git-subtree to replace the detached LevelDB sources in the bitcoin repo with an identical set of sources that also carries the history with it. This allows me to recreate the fork with any new changes that you guys make to LevelDB, so that I can submit it to the upstream.

The repository here: https://github.com/vinniefalco/LevelDB/commits/bitcoin-fork

Is not needed by the bitcoin developers. You can simply continue on the way you did before, editing LevelDB sources as needed. At any time, I can pull your changes and split them into my bitcoin-fork branch. I can also cherry pick commits from the upstream LevelDB repo, commit them to my bitcoin-fork branch, and then submit a pull request so they can get into Bitcoin. This is how I brought in the 1.10.0 update as well as the fix for the Windows deadlocking.

If you guys want to do this yourself I can show you how to use subtree. It is quite easy. If you want to use your own repository to store your fork of leveldb (e.g. at bitcoin/leveldb) you could fork mine, or clone mine and push it to your own empty repo (at bitcoin/leveldb) so that it appears as an original and not a fork (at which point I can delete my repo and fork yours). Its worth repeating that none of this affects the subtree, because the repo URI is not stored anywhere. It's just a remote in your local repo (on any developer's computer who wants to issue git-subtree commands). You don't need to use git-subtree to edit the resulting files. You only need to use git-subtree if you want to do the following:

1. Pull in commits from the external repository
2. Submit a pull request for local changes to the upstream (i.e. the google code repo or bitcoin's fork of it)

Or you could just continue on and pretend that LevelDB has no external repository, making changes to the sources as if they were your own - I can always step in and maintain the 'bitcoin-fork' branch no matter what you do, no action is required on your part unless you want to learn git-subtree and maintain the fork branch yourself.

@diapolo that'd be the case with submodule. With subtree, all the files are still in this repository, there is simply a link back to the original repository to make it easier to stay up to date (merging) or push changes back upstream.

@laanwj No that's not quite right. The URI / URL to the external repository is not stored anywhere in the bitcoin repository. Only the hashes of the commits are referenced, since that's all that matters for having an intact commit log. Anyone who wants to stay up to date / merge, or push changes back to the external repository needs to first 'git fetch' from the external repository to bring all the refs in locally. It does not matter from which repository they come from as long as it contains the appropriate objects - they are referenced by hash.

Ok, thanks for explaining.

ACK in any case

ACK

ACK

If you guys make changes to the level DB sources in the future, let me know and I can either split the changes out and push them to my LevelDB repo in the bitcoin-fork branch, or I can show you how to do it yourself using either my repo or your own.

Uh. I merge this, and I get a repository with no leveldb in it and no ability to build Bitcoin.

Note sure how you merged it. I'm in #bitcoin-dev to help you diagnose it. I will attempt to merge it myself and see if I run into problems.

I did the following:

git checkout master git pull # bring my local master branch up to date git merge leveldb-as-subtree # merge the pull request

The output should appear as follows:

sh-3.1$ git merge leveldb-as-subtree
Already up-to-date!
Merge made by the 'recursive' strategy.
sh-3.1$ git log -1
commit c1dd3db5dd62a53d4cf45f715f60481ff75e00bf
Merge: 3fbd87e 80744b3
Author: Vinnie Falco <vinnie.falco@gmail.com>
Date:   Thu May 30 16:32:44 2013 -0700
    Merge branch 'leveldb-as-subtree'