build: Avoid hardcoded libfaketime dir in gitian #17409

Without this gitian prints warnings for me:

ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

$ ls /usr/lib/aarch64-linux-gnu/faketime/libfaketime.so.1 
/usr/lib/aarch64-linux-gnu/faketime/libfaketime.so.1

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

No conflicts as of last run.

Code review ACK fab9850ef4a76e4f09278909a2865e5175661653 (haven't tested build determinism yet)

Note that this requires a gbuild that includes this patch: https://github.com/devrandom/gitian-builder/pull/223

Note that this requires a gbuild that includes this patch:

So should this get a mention in our docs somewhere, if you've bumped the minimum required version of gitian-builder?

docs somewhere

I don't think we have docs somewhere that document the minimum required version of gitian. Also, gitian is not versioned, and as long as the diff is applied to your local clone of gitian, the build should succeed.

@fanquake We don't officially support versions that are not the latest version:

https://github.com/bitcoin/bitcoin/blame/8021392b825c74312173f15eb937ba6d4aec3841/contrib/gitian-build.py#L245 https://github.com/bitcoin/bitcoin/blame/8021392b825c74312173f15eb937ba6d4aec3841/doc/release-process.md#L112

So I think this is ready for merge (assuming DrahtBot and other peoples build succeed)

I think the implicit requirement is to use the latest version of gitian-builder for new releases (I mean, latest commit). This is likely not documented anywhere,but it has always been the case

<!--a722867cd34abeea1fadc8d60700f111-->

Gitian builds

I think the implicit requirement is to use the latest version of gitian-builder for new releases

Ok. I feel like at one point we might have specified only a minimum bound, but I can't find any record of it.

Upstream https://github.com/devrandom/gitian-builder/pull/223 has now been merged.

<!--a722867cd34abeea1fadc8d60700f111-->

Gitian builds

The two above gitian builds don't seem to match :cry:

What do you mean they don't match? The commit hashes are different, so all builds must differ as well.

Oh, I thought you were trying to show determinism by running gitian-builder twice on the same commit.

I ran it on the same commit as DrahtBot and got a match for osx: (linux and windows didn't compile due to #17323 )

$ sha256sum ./scratch/gitian/bitcoin-binaries/25c136d3/*
1e63e8af030e810f6fced9b494274ee5821781687ce3a302730cead48dfd44bf  ./scratch/gitian/bitcoin-binaries/25c136d3/bitcoin-0.19.99-osx64.tar.gz
ed6a358f1b7fa39712174981c9e211e7288801e0711e3be4f6bd4c82f8b0cc66  ./scratch/gitian/bitcoin-binaries/25c136d3/bitcoin-0.19.99-osx-unsigned.dmg
b7fd3e751120b911c4b2e3998726b673a8a8edab6ad2cfe01526053e33c22d3d  ./scratch/gitian/bitcoin-binaries/25c136d3/bitcoin-0.19.99.tar.gz
3ada76f58943e505ce55cdb1f3e24fe599a323c44a826049e0839c48a461f258  ./scratch/gitian/bitcoin-binaries/25c136d3/bitcoin-core-osx-0.20-res.yml
143682b4f38681f400623973adb6bf37f00c800621086193a82e8aee04c96f24  ./scratch/gitian/bitcoin-binaries/25c136d3/linux-build.log
c70240190107edc5b757bff5004264d23dc116088883c878e3abb25caa5855d5  ./scratch/gitian/bitcoin-binaries/25c136d3/osx-build.log
71f1b8ead4f53b1b6806543b50e39ece600c88b5d5f3306d902acdc0ad0f71ef  ./scratch/gitian/bitcoin-binaries/25c136d3/win-build.log

Could you add a comment about $LIB being interpreted in a special way by LD_PRELOAD?

Done

ACK fab9850ef4a76e4f09278909a2865e5175661653, ran two -linux builds and they match ACK 333362991c5219e6d9ede5fa54328bcbb0277cf2

156d83b11f469a2b4d0f2f6e2e926ec5527a893178fa0b029e62acb030990622  bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz
357a099db4ca37f1076c9d88cacc64b0b905d84305c89d014277ad3c274388a7  bitcoin-0.19.99-aarch64-linux-gnu.tar.gz
44a02d1397312e16f6a029b0a3429ff617e1e1c8862e49b15730a3afe169bfa2  bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz
9f6eb58afdbf93ef1983e89b074a1dd7d0e52f97272c510a052ecfd7cedccc38  bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz
100ec7d9da684075c2bd7b24df0f008fb1c6a0b7977534d84196bf6a358eeeea  bitcoin-0.19.99-i686-pc-linux-gnu-debug.tar.gz
d541290225f9dcb645a5d7026a55212c74f370a9b07c4bccbe693690080347a6  bitcoin-0.19.99-i686-pc-linux-gnu.tar.gz
0bb2e865157e71af718d792774e6f8ed4eba80ad9f3790e171cf371f65fa30bd  bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz
ba3aa72aa649cafcbe51470931c0d8150f9d8c1a631e9708e83e8b25ac2dfcf2  bitcoin-0.19.99-riscv64-linux-gnu.tar.gz
5f3e64d037da40ffb78ef48ad5958e94542b776ff3e7769fd7aefb81eade3607  bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz
164e6348a02e056aa75a8c79f604d5cd026aadcac0ad9e49139dc04b4f05d7d0  bitcoin-0.19.99-x86_64-linux-gnu.tar.gz

156d83b11f469a2b4d0f2f6e2e926ec5527a893178fa0b029e62acb030990622  bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz
357a099db4ca37f1076c9d88cacc64b0b905d84305c89d014277ad3c274388a7  bitcoin-0.19.99-aarch64-linux-gnu.tar.gz
44a02d1397312e16f6a029b0a3429ff617e1e1c8862e49b15730a3afe169bfa2  bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz
9f6eb58afdbf93ef1983e89b074a1dd7d0e52f97272c510a052ecfd7cedccc38  bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz
100ec7d9da684075c2bd7b24df0f008fb1c6a0b7977534d84196bf6a358eeeea  bitcoin-0.19.99-i686-pc-linux-gnu-debug.tar.gz
d541290225f9dcb645a5d7026a55212c74f370a9b07c4bccbe693690080347a6  bitcoin-0.19.99-i686-pc-linux-gnu.tar.gz
0bb2e865157e71af718d792774e6f8ed4eba80ad9f3790e171cf371f65fa30bd  bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz
ba3aa72aa649cafcbe51470931c0d8150f9d8c1a631e9708e83e8b25ac2dfcf2  bitcoin-0.19.99-riscv64-linux-gnu.tar.gz
5f3e64d037da40ffb78ef48ad5958e94542b776ff3e7769fd7aefb81eade3607  bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz
164e6348a02e056aa75a8c79f604d5cd026aadcac0ad9e49139dc04b4f05d7d0  bitcoin-0.19.99-x86_64-linux-gnu.tar.gz

same for windows

f349e675ffec85081ad0bd3704b10ff87664888860dfe73c4898add9ebc7a68b  bitcoin-0.19.99-win-unsigned.tar.gz
78465e91cb11d1a145108c9b5d16500f477f9f4c643fdc5d0d5873abd7257b7c  bitcoin-0.19.99-win64-debug.zip
503a17d636c89fa9b9964ccc84300b228787cb54acd76400cef0f2364d4e3add  bitcoin-0.19.99-win64-setup-unsigned.exe
6bda2c10eebee5cc87eb9cd13e6f77927e1b502e1656320154346f3581112ade  bitcoin-0.19.99-win64.zip
691e4cc359b7fdf4f0d47d36dd4a1b4bc95146d6c9ae75838a4c35a8b3343701  src/bitcoin-0.19.99.tar.gz
6149a730b4d1953c7dc9d54679065728296bc5e2747bc364878d698738edbca7  bitcoin-core-win-0.20-res.yml

f349e675ffec85081ad0bd3704b10ff87664888860dfe73c4898add9ebc7a68b  bitcoin-0.19.99-win-unsigned.tar.gz
78465e91cb11d1a145108c9b5d16500f477f9f4c643fdc5d0d5873abd7257b7c  bitcoin-0.19.99-win64-debug.zip
503a17d636c89fa9b9964ccc84300b228787cb54acd76400cef0f2364d4e3add  bitcoin-0.19.99-win64-setup-unsigned.exe
6bda2c10eebee5cc87eb9cd13e6f77927e1b502e1656320154346f3581112ade  bitcoin-0.19.99-win64.zip
691e4cc359b7fdf4f0d47d36dd4a1b4bc95146d6c9ae75838a4c35a8b3343701  src/bitcoin-0.19.99.tar.gz
6149a730b4d1953c7dc9d54679065728296bc5e2747bc364878d698738edbca7  bitcoin-core-win-0.20-res.yml

Compiled windows and it matches yours as well:

Compiling fab9850 Windows
--- Building for bionic linux64 ---
Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up.
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results from target
Grabbing cache from target
Generating report
f349e675ffec85081ad0bd3704b10ff87664888860dfe73c4898add9ebc7a68b  bitcoin-0.19.99-win-unsigned.tar.gz
78465e91cb11d1a145108c9b5d16500f477f9f4c643fdc5d0d5873abd7257b7c  bitcoin-0.19.99-win64-debug.zip
503a17d636c89fa9b9964ccc84300b228787cb54acd76400cef0f2364d4e3add  bitcoin-0.19.99-win64-setup-unsigned.exe
6bda2c10eebee5cc87eb9cd13e6f77927e1b502e1656320154346f3581112ade  bitcoin-0.19.99-win64.zip
691e4cc359b7fdf4f0d47d36dd4a1b4bc95146d6c9ae75838a4c35a8b3343701  src/bitcoin-0.19.99.tar.gz
2b9ea9a8a42fdf90224b41f6de277502dfe1eb7020700d2764fcf9263bf74e59  bitcoin-core-win-0.20-res.yml

(except for res.yml, of course, which is expected)

Same for linux:

Compiling fab9850 Linux
--- Building for bionic linux64 ---
Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up.
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results from target
Grabbing cache from target
Generating report
156d83b11f469a2b4d0f2f6e2e926ec5527a893178fa0b029e62acb030990622  bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz
357a099db4ca37f1076c9d88cacc64b0b905d84305c89d014277ad3c274388a7  bitcoin-0.19.99-aarch64-linux-gnu.tar.gz
44a02d1397312e16f6a029b0a3429ff617e1e1c8862e49b15730a3afe169bfa2  bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz
9f6eb58afdbf93ef1983e89b074a1dd7d0e52f97272c510a052ecfd7cedccc38  bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz
100ec7d9da684075c2bd7b24df0f008fb1c6a0b7977534d84196bf6a358eeeea  bitcoin-0.19.99-i686-pc-linux-gnu-debug.tar.gz
d541290225f9dcb645a5d7026a55212c74f370a9b07c4bccbe693690080347a6  bitcoin-0.19.99-i686-pc-linux-gnu.tar.gz
0bb2e865157e71af718d792774e6f8ed4eba80ad9f3790e171cf371f65fa30bd  bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz
ba3aa72aa649cafcbe51470931c0d8150f9d8c1a631e9708e83e8b25ac2dfcf2  bitcoin-0.19.99-riscv64-linux-gnu.tar.gz
5f3e64d037da40ffb78ef48ad5958e94542b776ff3e7769fd7aefb81eade3607  bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz
164e6348a02e056aa75a8c79f604d5cd026aadcac0ad9e49139dc04b4f05d7d0  bitcoin-0.19.99-x86_64-linux-gnu.tar.gz
691e4cc359b7fdf4f0d47d36dd4a1b4bc95146d6c9ae75838a4c35a8b3343701  src/bitcoin-0.19.99.tar.gz
867588207f5c985ae8daac9baa6442083a63e133651becae61af8e62e7d51d45  bitcoin-core-linux-0.20-res.yml

The build on my arm64 with a single thread just completed:

Compiling fab9850ef4a76e4f09278909a2865e5175661653 Windows
--- Building for bionic linux64 ---
Stopping target if it is up
Making a new image copy
Starting target
Checking if target is up..
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
Grabbing results from target
Grabbing cache from target
Generating report
a0a12b428c9ce7137d966022be15e49bf3e2538738bce173509598bc829dd5fa  bitcoin-0.19.99-win-unsigned.tar.gz
b410bc4a2fa908f996b617efaca5cc9b3acae6920ee30f2f0cac8504a87f556b  bitcoin-0.19.99-win64-debug.zip
f45c898bac6c2550cb7dd9e2138f6beaf288ec58d59ea619fd30b55a8ac766c3  bitcoin-0.19.99-win64-setup-unsigned.exe
fd822a029b2a43975a5ff8c4707d16d5726dda69d24de11fb78c74d299518081  bitcoin-0.19.99-win64.zip
691e4cc359b7fdf4f0d47d36dd4a1b4bc95146d6c9ae75838a4c35a8b3343701  src/bitcoin-0.19.99.tar.gz
ba59634819dc55558f470116834fdf94a23dc21990538d74ff7a45c28844fa72  bitcoin-core-win-0.20-res.yml
Done.
gpg: keybox '/home/ubuntu/.gnupg/pubring.kbx' created
gpg: skipped "no_signer": No secret key
gpg: signing failed: No secret key
Traceback (most recent call last):
        1: from bin/gsign:93:in `<main>'
bin/gsign:11:in `system!': failed to run gpg --detach-sign -u "no_signer" "../gitian.sigs/fab9850ef4a76e4f09278909a2865e5175661653-win-unsigned/no_signer/bitcoin-core-win-0.20-build.assert" (RuntimeError)
Traceback (most recent call last):
  File "./contrib/gitian-build.py", line 261, in <module>
    main()
  File "./contrib/gitian-build.py", line 249, in main
    build()
  File "./contrib/gitian-build.py", line 69, in build
    subprocess.check_call(['bin/gsign', '-p', args.sign_prog, '--signer', args.signer, '--release', args.version+'-win-unsigned', '--destination', '../gitian.sigs/', '../bitcoin/contrib/gitian-descriptors/gitian-win.yml'])
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bin/gsign', '-p', 'gpg --detach-sign', '--signer', 'no_signer', '--release', 'fab9850ef4a76e4f09278909a2865e5175661653-win-unsigned', '--destination', '../gitian.sigs/', '../bitcoin/contrib/gitian-descriptors/gitian-win.yml']' returned non-zero exit status 1.
ubuntu@ubuntu:~/bitcoin-core$ uname -a
Linux ubuntu 4.15.0-1041-raspi2 [#44](/bitcoin-bitcoin/44/)-Ubuntu SMP PREEMPT Wed Jul 3 15:45:20 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

Does this outdate the docs: https://github.com/bitcoin-core/docs/blob/master/gitian-building/gitian-building-setup-gitian-debian.md#setting-up-the-gitian-image?

Yes, they will need to be fixed up

I guess this requires a new base image? Using the newest gitian master that includes (https://github.com/devrandom/gitian-builder/pull/223) calling bin/make-base-vm --lxc --arch linux64 --suite bionic results on my end in:

E: Invalid Release file, no entry for main/binary-linux64/Packages

Ok, it might be better to just revert this pull request and give up on getting this run on non-amd64

No, it isn't necessary to generate a new image. linux64 is an abstract name, the concrete build arch is still the same. What I did was just symlink the old base image. After that, the LXC build worked.

bin/make-base-vm --lxc --arch linux64 --suite bionic

The LXC parameters should be kept the same, there is no architecture-independent release of ubuntu :smile:

What I did was just symlink the old base image.

Yes, that works, but is clumsy. At the very least the instructions should be updated to include this step.

Well, less clumsy than waiting half an hour to generate a new base image, but sure… I think ideally gitian should pick the right base image for the platform automatically.

I can confirm that a symlink helps. But how should new (or existing) gitian builder figure that out?

Yeah, either

• the docs need to be updated, or
• gitian-builder needs to be made smarter, or
• the change needs to be reverted