Gitian builds: use 'sort' in deterministic way #4274

I have come to a solution to the first of my issues with a deterministic build in LXC containers: while trying to figure out why the source archive would have a different hash, I noticed that the sort output of the VirtualBox VM would differ from the sort output of the LXC container VM. See this gist.

The solution is fairly simple: instead of sort, use numeric sorting + stable:

sort -n -s

From the man page:

  -n, --numeric-sort
              compare according to string numerical value
  -s, --stable
              stabilize sort by disabling last-resort comparison

With my testcase (connecting via ssh and doing a sort) using these 2 options gave me same results on both LXC and VM.

The change needs to be applied to the gitian-linux.yml (and actually everywhere else the sort command is being used!), and yes - it will change the hashes, but for the better of determinism.

IRC user coryfields is helping me to troubleshoot this issue, trying to correlate it with potential locale problems. In this regard, I think the guide should better not advise to choose any locale/timezone during the host installation, or better a bug in separate project gitian-builder could be submitted to pretend that LXC environments never get polluted from parent

Yes, good point on using a locale independent sort everywhere. I already stumbled on that once but solved it some other way.

Can you submit this as a pull?

@laanwj sure, let me couple this with a docs fix for #4269 in a single PR

the docs fix is in that PR, let me know if we still want a commit for deterministic sort. It will help developers, but has surely less importance than the LC_* pollution issue.

I am closing this since https://github.com/devrandom/gitian-builder/issues/56 fixes this and other issues second a different approach