Migrating Off GitHub #35574

Over the past several years, but especially over the past several months, GitHub has rather rapidly deteriorated both in uptime, usability, and general reliability. It seems increasingly like the time for projects to move away for simple reliability reasons.

On the LDK end, we've seen the rapid increase in ban-aggressiveness bite us a bunch - at least three or four people have opened new GitHub accounts, opened legitimate (if AI-aided) PRs, only to get immediately banned (in which case GitHub deletes their PRs from our repo, with no ability to access their code at all).

This is, of course, in addition to the site suffering a bunch of downtime over the past few months, and the website generally getting slower and slower over the years as they add more and more useless client-side JavaScript.

Finally, recently, GitHub decided to ban the entire lightningdevkit org from GitHub Actions for unspecified "ToS Violations" (even escalating via a large corporate account was insufficient to get them to even tell us what we did that they want us to stop doing, when most of our CI is already self-hosted).

Of course I don't blame GitHub for some of this - they're seeing a massive increase in AI agent traffic and are struggling to contain the increase in load, but of course none of us have to suffer from their degraded service they are offering for free - self-hosting forgejo or using a commercial instance of it is easy enough. If anything moving off is helping reduce their increasing load.

Still, given the massive increase in issues over the past few months, and the fact that AI is only going to make this worse over time, I can't see a reason why any open-source project should remain on GitHub. We're, of course, planning on retaining our existing repository for visibility and to let people file issues, while moving actual work (and CI) to a self-hosted instance.

Of course feel free to close and ignore this, but I wanted to at least raise this and document what went into our decision. rust-bitcoin decided to start migrating a few months ago and we'll likely share an instance with them.

Are there obvious alternatives like BitBucket or GitLab that can take over the CI and won't fail under similar pressure? Or are you thinking we self host a GitGogs?

We leverage open protocols for as many things as possible for a project like Bitcoin. I recommend using Nostr NIP34.

It works without platform lock-in since you sign every issue or comment or PR with your sovereign keypair, just like with PGP. Nostr however brings much richer context because one keypair can be used with many applications. More context, more trust.

Nostr also leverages repository announcements so one logical repo can be used with any number of redundant git servers regardless of implementation. You sign where code might be, people fall back if need be. No hosting lock-in either.

Open-source NIP34 clients include:

• https://budabit.org (git + added community features like chat rooms, forum threads like Discord)
• https://gitworkshop.dev : github clone without platform lock-in
• and many others

Bitcoin deserves to be developed fully on open protocols.

This is kind of a pointless issue because anyone can already collaborate together on Bitcoin using any platforms or tools they want. There has never anything preventing this from occurring.

I think one way to bootstrap a private server is to mirror content from the Bitcoin org GitHub, and then sync back when you're ready to distribute a patch.

(I also like the idea of the "vouch" system where collaborators vouch for a new person instead of allowing random spam to be posted immediately to everyone.)

Are there obvious alternatives like BitBucket or GitLab that can take over the CI and won't fail under similar pressure? Or are you thinking we self host a GitGogs?

The original post says:

self-hosting forgejo or using a commercial instance of it is easy enough. If anything moving off is helping reduce their increasing load.

From what I've read, Gogs was sort the OG, Gitea forked Gogs, then Gitea owner pissed off the community with commercial decisions, then that forked to be forgejo. Forgejo seems like a legit project though and also supports a lot of open protocols such as Activity Pub and gasp RSS. Moving to a project that supports RSS seems reason enough.

Also the UI is so much more responsive in forgejo because there isn't a huge pile of client-side JS...

@kanzure

This is kind of a pointless issue because anyone can already collaborate together on Bitcoin using any platforms or tools they want. There has never anything preventing this from occurring.

This doesn't tell the whole story. You can collaborate and author features, but to get them merged, you need to pass CI, which is on GitHub. GitHub is the main place where development happens: PRs are created and reviewed here, and issues are filed here.

@TheBlueMatt

While what happened to LDK is unfortunate, there's no guarantee that scalability and customer support issues would not happen with a different provider. A lot of solutions are being proposed that seem okay now, but it's not clear whether they will be okay in the future (when the next batch of users migrate or when the management changes, etc.).

A better option would be a distributed or self-hosted solution. But given limited resources on the project, developers should minimize time spent on maintenance work. GitHub silently handles all the infrastructure and security issues at the moment. I would welcome a distributed solution (built on Nostr or something else), but the worry with those is that they might not be that polished, which would hinder development.

If Bitcoin Core is not experiencing any issues, I would advise not to move prematurely, just think of a backup plan.

While what happened to LDK is unfortunate, there's no guarantee that scalability and customer support issues would not happen with a different provider. A lot of solutions are being proposed that seem okay now, but it's not clear whether they will be okay in the future (when the next batch of users migrate or when the management changes, etc.).

A better option would be a distributed or self-hosted solution. But given limited resources on the project, developers should minimize time spent on maintenance work. GitHub silently handles all the infrastructure and security issues at the moment. I would welcome a distributed solution (built on Nostr or something else), but the worry with those is that they might not be that polished, which would hinder development.

Of course. I don't think it makes sense to jump to anything but a self-hosted option, and LDK is going to migrate to a self-hosted Forgejo instance that rust-bitcoin is starting to use.

If Bitcoin Core is not experiencing any issues, I would advise not to move prematurely, just think of a backup plan.

Coming from someone who got their CI shut down in the midst of a release process, i strongly recommend migrating before you are forced to, as its much easier when you can control the timeline...

@kanzure

This is kind of a pointless issue because anyone can already collaborate together on Bitcoin using any platforms or tools they want. There has never anything preventing this from occurring.

This doesn't tell the whole story. You can collaborate and author features, but to get them merged, you need to pass CI, which is on GitHub. GitHub is the main place where development happens: PRs are created and reviewed here, and issues are filed here.

@TheBlueMatt

While what happened to LDK is unfortunate, there's no guarantee that scalability and customer support issues would not happen with a different provider. A lot of solutions are being proposed that seem okay now, but it's not clear whether they will be okay in the future (when the next batch of users migrate or when the management changes, etc.).

A better option would be a distributed or self-hosted solution. But given limited resources on the project, developers should minimize time spent on maintenance work. GitHub silently handles all the infrastructure and security issues at the moment. I would welcome a distributed solution (built on Nostr or something else), but the worry with those is that they might not be that polished, which would hinder development.

If Bitcoin Core is not experiencing any issues, I would advise not to move prematurely, just think of a backup plan.

I agree with this, have both options

Having it as a "backup" isn't really an option, sadly. AFAIK forgejo doesn't currently have an option to continuously sync issues/PRs/comments from a github repo, only do a one-time migration or keep a copy of the git tree alone. That implies you either have to do a cut-over and switch or just keep hoping github doesn't break any more than it already has.

Concept ACK, reviewing PRs when Github hides comments, takes forever to load, and times out is terrible. I am open to anything that does not require downloading something, loads comments, and has a clean stable interface.

Concept ACK. Using GitHub made sense because it made our lives easier, but that seems to hold up less and less because of outages, bans, and a workflow (e.g. moderation, hidden comments, ...) that doesn't seem to perfectly fit the needs of many of us.

I don't have experience running self-hosted alternatives, but if there is an alternative that do not require large amount of administration and maintenance, and people are willing to take this on, that would be a nice alternative. I'd also be open to moving to another hosted service (e.g GitLab) if it means less maintenance or better functionality. Sure, it might mean we have to move again in x years if/when they change their terms or service, but similar concerns (maintenance, functionality, ...) exist for a self-hosted solution as well, so I wouldn't rule either out.

Having it as a "backup" isn't really an option, sadly. AFAIK forgejo doesn't currently have an option to continuously sync issues/PRs/comments from a github repo, only do a one-time migration or keep a copy of the git tree alone. That implies you either have to do a cut-over and switch or just keep hoping github doesn't break any more than it already has.

We have some existing repo metadata being archived (by the one and only @0xB10C) so I patched Forgejo to be able to import it and act as a full "read-only" mirror of this repo.

It's live, updated hourly, and available to browse here: https://git.fish.foo/bitcoin/bitcoin

I'm not advocating that we use this instance (or migrate now), but it is a nice demonstration that, as far as I can tell, we do actually have the data needed to do it in case of a GH ban, or whenever we fancy it (which may well be sooner rather than later).

As has been discussed before in the previous times this topic has come up, the biggest issue with self hosting is who is going to pay for and host it. Maintaining a self hosted instance that anyone can access will basically require hiring a sysadmin. I agree that github's recent uptime has been tragically awful, but I'm not certain that we would actually be able to do better without essentially hiring someone full time to maintain the self hosted instance. But it would also be nice to have to deal with these annoying "features" that github keeps adding.

A couple decentralized options have been mentioned, but I would like to voice my opposition to using those. I don't want to use anything which requires developers to have to install some other specific software just to access the git repo, not to mention how/whether those options at all integrate with CI.

Speaking of CI, we were recently forced to migrate CI to a new service. I don't think it would be a good use of time to have to migrate CI yet again, so if we decide to move off of github, then we should prefer some solution where the CI changes are minimal.

As has been discussed before in the previous times this topic has come up, the biggest issue with self hosting is who is going to pay for and host it. Maintaining a self hosted instance that anyone can access will basically require hiring a sysadmin. I agree that github's recent uptime has been tragically awful, but I'm not certain that we would actually be able to do better without essentially hiring someone full time to maintain the self hosted instance. But it would also be nice to have to deal with these annoying "features" that github keeps adding.

Given rust-bitcoin (well, me, but also soon others) is already doing it, I don't see a reason why Bitcoin Core couldn't use the same infra.

A couple decentralized options have been mentioned, but I would like to voice my opposition to using those. I don't want to use anything which requires developers to have to install some other specific software just to access the git repo, not to mention how/whether those options at all integrate with CI.

Sadly, AFAICT, none of them support moderation, which is also a deal-breaking IMO.

Speaking of CI, we were recently forced to migrate CI to a new service. I don't think it would be a good use of time to have to migrate CI yet again, so if we decide to move off of github, then we should prefer some solution where the CI changes are minimal.

Luckily the forgejo Actions config file format is ~identical to the github one. See eg https://git.rust-bitcoin.org/rust-bitcoin/rust-psbt/pulls/111/files (note that "runs on" change is just cause I was running a different VM for it).

Another fun fact, in doing our migration we discovered a bug in Github's API where we had (at least) an issue which was totally fine but didn't appear in the issue list/search APIs.

As has been discussed before in the previous times this topic has come up, the biggest issue with self hosting is who is going to pay for and host it.

I have a computer in my basement that we could use. Jokes aside, given the companies that sponsor devs on this project AFAIK, I'd be surprised if none of those organization had some rack space someplace.

Maintaining a self hosted instance that anyone can access will basically require hiring a sysadmin

As it stands, it seems like this project spends time fighting AI bots (even if it's time spent coding Draftbot to detect activity), so the time spent doing that could instead be on self-hosting an instance. This was a reason I've seen that projects are moving off of Github to something self-hosted. In that way, the projects are not such an easy target for bots and junk PRs/issues. The instance Matt setup has a sort of POW thing in the interface that makes it less feasible for bots to spam.

Many of us who work on Git Nostr were inspired to build a github alternative to meet the needs of this specific community. I created ngit, gitworkshop and the GRASP protocol. Imo we are headed in the right direction but are not a viable solution for many projects in this space right now. We fall short on moderation, CI intergraton, migration tooling and stability. I'd encourage you to keep an eye on our progress and give us feedback.

About CI: Bitcoin Core used to run on persistent self-hosted runners (only the linux part). There was zero downtime over the years while it was running, but other devs preferred hosted runners due to maintenance (security updates, shuffling servers, bus factor, ...). Going back to self-hosted for Linux may be workable, but I can't imagine anyone to voluntarily run a macOS VM, let alone a Windows VM?

Edit: Also the armhf that runs on an aarch64 VM natively may be difficult

Please correct me and speak up, if my imagination is wrong. Otherwise, the only alternative I see is to mirror all CI pushes from the self-hosted repo to a hosted repo with a CI subscription and then sync the results to the self-hosted platform? Though, if the hosted repo sits on a platform that we think is going be deleted soon, I wonder what the point is.

To cross link, there is also a related wiki page (https://github.com/bitcoin-core/bitcoin-devwiki/wiki/GitHub-alternatives-for-Bitcoin-Core), but it hasn't been updated in more than 2 years.

It's live, updated hourly, and available to browse here: https://code.fish.foo/bitcoin/bitcoin

Nice. I think someone ( cc @fjahr ?) tried the same for GitLab a few years back, but I don't recall the outcome.

It could make sense to document outcomes of experimental migrations in the wiki, so that it is easier to evaluate them?

FWIW, if the project decides to migrate or mirror, I assume it makes sense to do it for all Bitcoin-related projects that are maintained by the core group, for example:

• https://github.com/bitcoin-core
• https://github.com/corecheck/corecheck
• https://github.com/bitcoin-dev-tools
• https://github.com/oss-garage

I'm sure there's more that I'm forgetting or don't know about.

Nice. I think someone ( cc @fjahr ?) tried the same for GitLab a few years back, but I don't recall the outcome.

It could make sense to document outcomes of experimental migrations in the wiki, so that it is easier to evaluate them?

There is https://delvingbitcoin.org/t/gitlab-backups-for-bitcoin-core-repository/624 and https://gist.github.com/fjahr/9a28abefe0ab8413d96aa1dd7903c5d4. The instance seems to be running at https://gitlab.sighash.org/bitcoin/bitcoin (last activity in Feb 2025).

Going back to self-hosted for Linux may be workable, but I can't imagine anyone to voluntarily run a macOS VM, let alone a Windows VM?

We're happy to forge (🥁 ) the way here. We have a loose plan of just spinning up AWS VMs for windows and mac jobs but haven't gotten far enough yet to test it out. Will let you know how it goes.

FWIW, the Forgejo CI runner is way nicer than the github one, it has native docker integration so it kinda Just Works if you give it an image.