Dependency on GitHub #20227

After reading this post with the title «Important Open Source projects should not use GitHub» I thought instantly about Bitcoin and it's source code. It is the most important open source project and it is hosted on GitHub.

Was this topic ever discussed? I would think so after Microsoft aquired GitHub, but I couldn't find it. What happens if GitHub requires a Microsoft account to login to GitHub, are all developers willing to create a Microsoft account?

I know, I don't offer a solution at this point. I heard about Sourcehut as an Alternative to GitHub. The problem is, the source code of Bitcoin will (with the current available git hosting providers) always be centralized. Is that the reason why Bitcoin didn't move away from GitHub?

This is more like a discussion, if this is the wrong place for that or there is already an open issue. Please close this one and point me in the right direction. Thanks!

We had this discussions already a few times in the past.

Our dependency on GitHub is not huge. We use it as a pure workflow and communication tools which can exchanged relatively painfree.

Switching away from GitHub just because of eventual expected friction makes little sense to me (as of today). AFAIK we'r still mirroring everything,... including the GitHub communication (comments, pulls, etc.). I also regularly fetch all pulls/issues html (which is nowadays incomplete with the lazy loading approach of comments and reviews).

Maybe we should reinvestigate in mirroring "everything". Weekly fetching of all issues/pulls including all comments, etc. by the JSON API.

Also, there are viable GitHub alternatives (like GitLab) which probably makes it harder for Microsoft to add things like "everyone needs a Microsoft account". But sure, that could change.

We had this discussions already a few times in the past.

Dialectic https://en.wikipedia.org/wiki/Dialectic should never end. My antithesis would be, how would Satoshi thought about that? From all we know he used platforms like MS to reach out but was clear pro decentralization and anti censorship. So to host this project on a platform that proven had begun to exclude certain third world country's devs. Is a no no and since Bitcoin is one of the most relevant projects in our century maybe even the greatest at all and as political as something can be political, I would feel much more comfortable, if the move could be done fast. I,e like the Tor project had done. https://gitlab.torproject.org/tpo

We can not assume MS won't or must do what it had always has done, is naive. They must be assumed in a framework they can't escape.

@jonasschnelli The entirety of Github is archived in real time, it's trivial to filter the output with jq for just bitcoin repository events.

https://www.gharchive.org/

Just to add some recent context too. (I'm not sure if it is general practice to attribute quotes from public IRC channels outside of scheduled meetings, ie bitcoin-core-dev or keep quotes anonymized. I will do the latter here just in case)

GitHub concerns

"there are now "73 hidden items" in #19988 that can no longer be loaded (the link does nothing) :/"

"it's been the case for a few days now, only the number of hidden comments keeps growing. i'm reviewing without the discussion."

"did anyone report this to github yet? we can't keep using the platform if this is the case"

"They seem to pushing a workflow where you approve the PR using the CLI but where discussion and review is done outside of the PR"

" i use gh cli a little and keep updating it, but the features added so far aren't what i'm hoping for. no getting the comments yet."

"whoever the PM is who is driving their features roadmap has very different priorities than ours"

"Ugh, is there any reason why GitHub would delete whole issues? #17298 is gone"

"One thing I notice is that its title is identical to the title of an older issue which makes me wonder if GitHub did some kind of erroneous database cleanup or something, although that would be crazy"

"I've seen this happen at least three times. I can look up all the issues that were deleted, if needed."

Thoughts on experimenting with a GitHub alternative

"iirc the idea was to move away from any centralised service, if a migration were to happen"

" I guess we could have moved the gui monotree to GitLab for comparison. Downside would be that all gui ppl need to create another account for review/pulls "

"If we need to move, i'd suggest moving everything, having split environments is even more annoying"

Thoughts on GitLab specifically

"Gitlab seems to handle things fine, e.g. freedesktop uses their own gitlab instance to host some active high-profile projects such as Mesa, no big issues from what i know. They were also really, really careful to transition from mailinglist-based FOSS development, they never trusted GitHub"

"the globocorps and startup missions i was on until late 2018 used gitlab and mattermost instead of github and slack, which i was happy about, and we didn't have any issues at all -- i definitely preferred gitlab"

"I have a GitLab repo that exceeded an arbitrary repo size limit years ago; opened an issue on their tracker, no response; pokes on Twitter, was told they'd look at it, still no response.. years later"

Above are various people's comments. I'll add my own. I think we need to seriously consider alternatives. If it was just the terrible reviewing comment experience then it wouldn't be worth it. But the general experience seems to be deteriorating and the direction they appear to be moving in (GitHub CLI etc) with discussion done outside of PRs doesn't gel at all well with our processes.

I'm very sympathetic to the view that the timing needs to be right though and that we need to make sure that any alternative we potentially move to doesn't offer up a different set of problems.

Adding to the list of issues:

https://twitter.com/lrvick/status/1320246266270519297

And censorship issues that we have already seen in past:

https://techcrunch.com/2019/07/29/github-ban-sanctioned-countries/

UX issues: https://twitter.com/jonatack/status/1315623229747322880

Asked a question on Stackexchange in case GitHub censored this issue as well : https://bitcoin.stackexchange.com/questions/99634/what-are-some-of-the-github-alternatives-that-you-consider-for-bitcoin-core-and

Asked a question on Stackexchange in case GitHub censored this issue as well :

Come on.

Github, like it or not, has to comply with US regulations. They can not refuse to comply with a valid DMCA notice.

@wpeckr prepared for the worst

Microsoft won't like if one of the important open source project moves from GitHub to one of the alternatives.

It's good that we can discuss GitHub alternatives on GitHub.

It's good that we can discuss GitHub alternatives on GitHub.

Are u sure u see all my comments or the correct source of a PR (SHA-1) in your Browser? If core does not self host fast, it is a fox that thinks it can get a nice scorpion in his mouth about the water.

https://github.com/bk2204/git/blob/transition-stage-4/Documentation/technical/hash-function-transition.txt

@Saibato Yes I can read your comments.

the correct source of a PR (SHA-1) in your Browser

Can you explain the above thing?

Irony is we can discuss GitHub alternatives here but the question I asked on StackExchange might be closed if 2 more people vote for it as OPINION BASED lol

This is the question I asked: What are some of the GitHub alternatives that you consider for Bitcoin Core and other Bitcoin related projects?

Can you explain the above thing?

I see it that way, if u can not be sure what is merged, is what u reviewed since that is in Git archived with chains of hash functions, its a rigged task, if ur platform has a lot of computing power and incentives to use it by the legal frame they can not escape.

Diversity and multi path;s and platforms is key to free speech, ( btw that;s the reason centralized structures want conferences, fixed timeframes and roadmaps and channeled discussions)

We can assume even at MS are ppl that want change, but ... , and at least I want not temped them and force them to be disloyal.

I guess they let us by now only go bcs they think they can scrum or agile learn but when they figure out what bitcoin really did, i guess the will get in rage.

Since world dominance and leverage, is and was, i must assume Bill's playgrounds goal,

btw. Not that ruled by freedom loving Cowboys is that a bad thing, but at least stylish smart and hipp like Jobs with Apple did.

@TheBlueMatt on IRC:

gitlab seems pretty complicated, right?

hmm? you mean to self-host it? I didnt look into it too deeply. I think actually the easiest thing to set up would be to a) create a webform that lets you upload an ssh pubkey which then creates a user which is restricted to git-shell, b) restirct each user via GIT_NAMESPACE to a subtree of refs which is prefixed by their ssh pubkey hash or so, c) use gitweb to display all the pushes people have, d) limit upload pack size and do something to limit how many pushes/hour each remote IP can do. I think the above basically lets you do an anonymous git project management with an hour or two of setup and without all the complication of something like gitlab or gitea where users have ot actually create accounts with emails and passwords and stuff.

iirc actually restricting users to git-shell with ssh isn't as trivial as it should be

no, you just set their login shell, its one command or you can just edit /etc/passwd's shell entry and done. you also have to create an empty git-shell-commands file or something stupid but thats easy.

github i think is using a custom ssh daemon

ah, i could see that. i mean ssh is shit, but i dont think git-shell is the biggest of your worries in that case. and if you're worried just do a separate git-upload server and then pull from it every minute on the actual git server. but I think you have the same issue with gitlab or whatever. the other thing I was thinking about is doing git pushes via https, and then just setting a cookie for user auth. still anonymous, but users have to configure a per-repo cookieFile and set the flag to let the server set cookies locally. would be a tiny bit more work for users, but the server-side setup would be similar effort. just via nginx authentication modules instead of ssh. if you're feeling un-lazy, you can set it up for git.bitcoin.ninja and then I'll just let people use that server for stuff :)

We aren't looking for another git mirror of the repo (anyone can run those without asking for permission). We use GitHub to discuss patches and attach review comments to them. I have never used git-shell, but I'd surprised if it solved that problem.

It is great to see that we have so many volunteer sysadmins ready to a.) take on the migration away from GitHub the day the project would need such help, and b.) to take on the continuous sysadmin work that would be required to keep the new system patched, hardened and otherwise in par with the GitHub offering.

Thanks for volunteering! However, currently it appears that the consensus opinion from regular contributors is that the project is not in need of any such sysadmin help for migrating off GitHub. (That might change in the future of course.)

However, there is a sysadmin task that I think is somewhat urgently needed by the project: we need a Bitcoin Core installation of ClusterFuzz/CIFuzz to catch issues such as the recent wtxid crash bug before merge (or at least soon after merge). See comment in #20332 (comment) for more information. Sysadmin volunteers welcome!

I'm not a great fan of the sarcasm @practicalswift, perhaps you can tone that down. I collected together comments from long term contributors in the above comment. Multiple regular contributors have concern about the current state of GitHub performance and the future trajectory. If they didn't I wouldn't bother engaging with this topic (whatever my personal views).

I am interested in what it would need to migrate and whether this is possible/viable. It certainly sounds like you think it isn't viable which is interesting. Other long term contributors did not think it would be that disruptive when I chatted to them on IRC. Perhaps their view has changed in the intervening period.

@michaelfolkson I'm not sarcastic: I'm happy if we have sysadmin resources standing by to take on work. My point was simply that judging from this and previous discussions it seems that migration away from GitHub seems not be requested, or a priority at the moment. Thus I suggested an alternative project where these sysadmin resources could make a meaningful difference for end user security right now. That's all :)

I am agree to find an alternative to github. Thanks to open this issue!

Not sure if everyone read the tweet by @achow101 mentioning the script he wrote to download all issues and PRs (includes issue/PR comments, PR commits, repo wiki, releases, release assets, milestones, and labels) for a github repo. Maybe related to this issue and helps someone reading the conversation in future: https://github.com/achow101/github-dl

Thanks everyone for the input. The discussion has been moved to the wiki: https://github.com/bitcoin-core/bitcoin-devwiki/wiki/GitHub-alternatives-for-Bitcoin-Core