build: Make networking work inside LXC builder in gitian-building.md #7060

These are changes I needed to get gitian building to work with a fresh Debian 8.2 VM, which is the version we tell to use.

• Set up NAT and forwarding, so that LXC container can access network beyond host
• Remove explicit cgroup setup - these are mounted automatically now

Also needed a change to gitian: https://github.com/devrandom/gitian-builder/issues/105 [skip ci]

utACK

It also looks like we now need ca-certificates and python as dependencies in the descriptors for LXC gitian builds. Apparently for KVM they are added to the base image, but not for LXC.

Can you also give this a try @theuni? Want to make sure that I'm not crazy (or somehow have a subtly different environment) that I need these changes.

@laanwj I can't try until tomorrow, but I believe i've ended up doing similar things to get it somewhat working in the past. I never got it reproducibly working 100% though, so if this does the trick, that's great!

There were three major issues in my case:

• LXC container couldn't reach the internet (solved by setting up masquerading/forwarding, as well as the gitian change to set 'auto' gateway)
• TLS certificates couldn't be verified (solved by adding ca-certificates package)
• Python missing error (I think during MacOsX build, adding python package)

We need gitian-building instructions that work. If this too doesn't solve it reproducibly, then switching the VM image to Ubuntu may be the best plan (though it's nice if Gitian works from Debian as well). I'm not actually sure that these problems don't exist with an Ubuntu VM.

switching the VM image to Ubuntu

Ideally this should work on any linux with the right hardware?

Ideally this should work on any linux with the right hardware?

For gitian itself you're right.

Though with all the possible things that may be different between setups, a single "follow these steps" plan cannot work on any linux. To make sure it is actually tested and reproducible (which even now is lacking!) I'm just targeting one distro with this guide. (same answer to questions like "why not KVM" etc - this is not meant to be a comprehensive book on all the capabilities of gitian, just one tested way to get from a VM to built executables)

We really need to be going the opposite direction here. gitian build environments should be banned from accessing the network at all, both because they shouldn't need it and its good hygene to not allow it and because we really should be supporting building on airgapped machines and always denying network access is a good way to ensure we continue to do that.

My goal was just to get the guide working as it is specified now. I agree with you long term @TheBlueMatt but right now, having a broken guide is more harmful than one that isn't perfect.

Feel free to write it as you think is better.

But if it isn't fixed before 0.12, I'd rather just remove it than leave it in a broken state. I mean, people follow these steps exactly and it doesn't work, can you think of anything more frustrating?

Agree with @laanwj

just remove it than leave it in a broken state

Or mark it [outdated]?

Is it no longer possible to build without network access? If so, can we just change the documentation to disallow network access instead of allow it? If not, when did we merge something with such a huge regression? Would it be easy to fix?

Or mark it [outdated]?

If someone wants to resurrect it it's in git history.

Is it no longer possible to build without network access?

It is possible, to be precise it's even mentioned in the guide in a separate section: https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md#building-fully-offline

However it requires extra steps, building with network access in the container is easier.

@theuni Can you go through this one time to test it please? Would be nice to have this up-to-date before we start building 0.12 RCs.

@laanwj Yep, will do in a few hours.

Sadly I didn't make it to this today. I'll revisit asap after the conference.

@laanwj Just stepped through a 0.12 build for win32/linux and hit no snags. I haven't tried osx or the offline path (followed the guide from top to bottom, whoops). I'll get those going in the background next.

From top to bottom makes a lot of sense :)

ACK

Tested ACK regarding adding the python package to the OS X descriptor. That let it work for me when it failed before (building in LXC on Ubuntu Trusty). I don't know if adding it is necessary for the non-OS X builds, and I haven't tried any of the network changes. In response to @TheBlueMatt, I don't think it's actually needed. I don't actually know if my container can access the Internet, but afaik it only needs it to download the depends files, and we have a step in release-process.md that downloads the files and seeds them into the gitian cache.