build: Replace genisoimage with xorriso #20470

dongcarl commented at 12:06 AM on November 24, 2020: member

This is a redo of fanquake's #18151, which, aside from switching us from the deprecated genisoimage to the maintained xorriso, is also necessary for Guix to achieve determinism without using faketime.

xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are more maintained, and has the right toggles for us to achieve output determinism without using blunt tools like faketime.

In this commit, we use xorrisofs from the build environment rather than building it ourselves using depends. This is not necessary and can be changed in the future.

From wiki.debian.org/genisoimage?action=recall&rev=11 :

The classical command line interface for production of ISO 9660 filesystem images is the option set established by program mkisofs. For reasons of licensing and other problems with its author, Debian ships a fork of mkisofs, called genisoimage, which was split off in 2006 and then developed independently.

Meanwhile, genisoimage gets no new features and not even bug fixes. It is first choice only if its options -udf or -hfs are needed.

Replacement in most uses cases, especially for bootable ISO 9660 filesystems, archiving, and backup, is xorrisofs which starts the -as mkisofs emulation mode of program xorriso.

dongcarl added the label Build system on Nov 24, 2020

dongcarl added the label Needs gitian build on Nov 24, 2020

dongcarl force-pushed on Nov 24, 2020

fanquake added this to the "PRs" column in a project

DrahtBot commented at 8:45 AM on November 24, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#20638 (build: Fix macOS code signing by pre-allocating space for the code signature during gitian build by achow101)
#17920 (guix: Build support for macOS by dongcarl)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

laanwj commented at 8:48 AM on November 24, 2020: member

Concept ACK

practicalswift commented at 8:54 AM on November 24, 2020: contributor

Concept ACK: maintained is better than unmaintained :)

hebasto commented at 8:25 AM on November 25, 2020: member

Concept ACK.

DrahtBot commented at 11:38 AM on November 30, 2020: member

Gitian builds

File	commit e2ff5e7b35d71195278d2a2ed9485f141de33d7a<br>(master)	commit 4cd306c4d67aac6616196da0ed1ec91d6abd46a4<br>(master and this pull)
bitcoin-core-linux-22-res.yml	`44e52af36236879f...`	`9e35f5cf2a311cd8...`
bitcoin-core-osx-22-res.yml	`71d5fb0e6642d6b0...`	`39c3b4d8606449d9...`
bitcoin-core-win-22-res.yml	`0148a2f5e7295184...`	`c44f40f485521b1d...`
*-aarch64-linux-gnu-debug.tar.gz	`383e84aaa96f342a...`	`c2373d9b17ef64aa...`
*-aarch64-linux-gnu.tar.gz	`3558211891c0484a...`	`bbe6f3a350eee12e...`
*-arm-linux-gnueabihf-debug.tar.gz	`5ebbb475a590480a...`	`b8620629a4943471...`
*-arm-linux-gnueabihf.tar.gz	`60951d893e2688b6...`	`6a29c12f830614d5...`
*-osx-unsigned.dmg	`7cba6edda598c96c...`	`7746491c14d8e536...`
*-osx64.tar.gz	`ea2ab5c893780a2e...`	`40b977f268fd9103...`
*-riscv64-linux-gnu-debug.tar.gz	`0c20a43923efcd5f...`	`e0c2d9f76e8f7bdd...`
*-riscv64-linux-gnu.tar.gz	`8d3632d81c77eff1...`	`1d6d0f4afed9e98d...`
*-win64-debug.zip	`72c35e133cf9a0db...`	`44bb9249ea02fb4c...`
*-win64-setup-unsigned.exe	`55c17bfff54273e5...`	`6bb09951569edd6b...`
*-win64.zip	`f7bf6737c7f741c3...`	`22a04573095f31ee...`
*-x86_64-linux-gnu-debug.tar.gz	`abf91baef5c44686...`	`2100bfc3e7ad0630...`
*-x86_64-linux-gnu.tar.gz	`75fda490174df335...`	`17ce53433f37c554...`
*.tar.gz	`793a2c7af39452d5...`	`42b4a292b2236755...`
linux-build.log	`9c5461a914e1f4c0...`	`58c11f1d8d0b6a9e...`
osx-build.log	`5adaf67afbaf00fc...`	`c91d144bb6cd0bcc...`
win-build.log	`756101ecccffabf0...`	`3488da316ac6068e...`
bitcoin-core-linux-22-res.yml.diff		`872de92646873957...`
bitcoin-core-osx-22-res.yml.diff		`c2beeb988be1ec6f...`
bitcoin-core-win-22-res.yml.diff		`70161d6cfad01933...`
linux-build.log.diff		`1e0eb4be52ea6b97...`
osx-build.log.diff		`15491d7a0b331268...`
win-build.log.diff		`d6882ad0a0026167...`

DrahtBot removed the label Needs gitian build on Nov 30, 2020

jonasschnelli commented at 8:16 PM on December 1, 2020: contributor

Tested ACK f2cfb5c794c83a6e8771c8f08e5fa8aefcf269fc - Did a gitian build and verified the OSX disk image https://bitcoin.jonasschnelli.ch/gitian/build/347

dongcarl force-pushed on Dec 9, 2020

dongcarl added the label Needs gitian build on Dec 9, 2020

dongcarl commented at 10:24 PM on December 9, 2020: member

Pushed b9d60ddefcf14c0ccf38c1a841c7558396e5e738 -> 75bfa827b30985ec7ca3b161a6dc0e1fb2f0432b, specify that we need to use the faketime-wrapped xorrisofs to ./configure in Gitian

fanquake commented at 1:06 PM on December 10, 2020: member

Looks good. I tested with 48e1b68a2e757fee2dbcd724deee075385e797c5 on top of master (86f20071931b803b5f26ed8f685d98d4919fb7a7).

One thing I noticed is that the naming of the DMGs is now inconsistent. i.e a make deploy on macOS still produces Bitcoin-Core.dmg, whereas when cross-compiling you'll end up with bitcoin-21.99.0.dmg (along with the .iso). We should probably make this consistent. #20422 recently changed the name of the DMG produced on macOS (to Bitcoin-Core.dmg over Bitcoin-Qt.dmg`). Probably worth rebasing this branch so that you're working on top of those changes.

Will be nice if xorrisofs can do DMG compression in the future.

Note: we should also add a *.iso entry to .gitignore as part of this change.

dongcarl force-pushed on Dec 10, 2020

dongcarl commented at 12:51 AM on December 11, 2020: member

Addressed fanquake's concerns. Here are the Gitian hashes for fdb76e62ce8d4d7a072d2e05895360a1cb7e7b6d

ee16ae845d4ea95842abd59e81b91c3a84924149c94c526c2f68d46f08b40cf3  bitcoin-fdb76e62ce8d-osx-unsigned.dmg
a96e1af388a4c1616489d4d52076d567c1ab33557bd0e56ea21fbd766739d33c  bitcoin-fdb76e62ce8d-osx-unsigned.tar.gz
c2691b955bf1d781f19c41908f9152264502a299b827b30d40049b8f567d35e6  bitcoin-fdb76e62ce8d-osx-unsigned.temp.iso
fc298e3f2f592398a386d9cf6eabaed8c41e7bf949636014ef78120430ef1351  bitcoin-fdb76e62ce8d-osx64.tar.gz
2e3e88d07f07bd42457ec22bf43fae6cfe51582faae50c0e0b0d41aae2c6d809  src/bitcoin-fdb76e62ce8d.tar.gz

in Makefile.am:140 in fdb76e62ce outdated

 135 | @@ -135,8 +136,11 @@ $(APP_DIST_DIR)/Applications:
 136 |  
 137 |  $(APP_DIST_EXTRAS): $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt
 138 |  
 139 | -$(OSX_DMG): $(APP_DIST_EXTRAS)
 140 | -	$(GENISOIMAGE) -no-cache-inodes -D -l -probe -V "$(OSX_VOLNAME)" -no-pad -r -dir-mode 0755 -apple -o $@ dist
 141 | +$(OSX_TEMP_ISO): $(APP_DIST_EXTRAS)

theuni commented at 3:58 PM on December 11, 2020:

This could be .INTERMEDIATE, don't know why it wasn't before.

theuni commented at 3:59 PM on December 11, 2020: member

Concept ACK. I never had much luck figuring out who the true upstreams were for these tools. Much better to use something supported.

build: Run libdmg-hfsplus's DMG tool in make deploy

Previously, the compression of the .iso file to a .dmg file was done
outside of `make deploy' in order to use the faketime-wrapped version of
libdmg-hfsplus's DMG tool.

Specifying the faketime-wrapped version of the DMG tool to ./configure
fixes this and simplifies build scripts.

22437fc72e

dongcarl force-pushed on Dec 11, 2020

dongcarl commented at 4:44 PM on December 11, 2020: member

Pushed fdb76e62ce8d4d7a072d2e05895360a1cb7e7b6d -> 25d39b61095d7c1ccbc5ec08a9c738aa56ade421: addressed #20470 (review)

616680767fe519b013f31a8c915719d15aeb45647929a2185e9d7d4b34ed057a  bitcoin-25d39b61095d-osx-unsigned.dmg
19e44ee5454510a7ce77b61ffdfd988f55ec1da8497b0c63803605601ef45ef0  bitcoin-25d39b61095d-osx-unsigned.tar.gz
ab639cd94bcb1ca168312927e1708de84a844be94e9672951d71de1446297f39  bitcoin-25d39b61095d-osx64.tar.gz
574fac8995ab9cb900aeb263e5b920d7ae3d1385523a4dd9cbe58c56184e1761  src/bitcoin-25d39b61095d.tar.gz

DrahtBot commented at 10:35 AM on December 12, 2020: member

Gitian builds

File	commit 736eb4d8083862a6c3dd79e65afca6217cf7939d<br>(master)	commit 33d40f23650b790631289092f24ca962a9fed00c<br>(master and this pull)
*-aarch64-linux-gnu-debug.tar.gz	`2be2a11a222f573c...`	`74c1a539c73cf28d...`
*-aarch64-linux-gnu.tar.gz	`aa6be788b43c77ab...`	`cce35d5a5cc98494...`
*-arm-linux-gnueabihf-debug.tar.gz	`bc084de532268d02...`	`e020fafd3eefd2d2...`
*-arm-linux-gnueabihf.tar.gz	`8bf8847238bbb772...`	`776c5a95b8aef35c...`
*-osx-unsigned.dmg	`0e0b5353ad614bc4...`	`91daf88d0bac46dc...`
*-osx64.tar.gz	`138e5e00bfec397f...`	`6ce3ca2f85f19001...`
*-riscv64-linux-gnu-debug.tar.gz	`f77b9fc97e1691c7...`	`cc61ad54b008cfed...`
*-riscv64-linux-gnu.tar.gz	`45248a9a8de4982e...`	`5204a540dbf32e48...`
*-win64-debug.zip	`09d982b554816f6b...`	`b4052a10e53c7604...`
*-win64-setup-unsigned.exe	`3c905f96f0f756e0...`	`4844a665c8c9ae50...`
*-win64.zip	`8c7fc6e3a8ea93fe...`	`b7772ae95be473df...`
*-x86_64-linux-gnu-debug.tar.gz	`e47b5bfa0a774d6b...`	`d25b8e240357d68c...`
*-x86_64-linux-gnu.tar.gz	`fe3f8b7e4e601768...`	`83a4306e193a8adc...`
*.tar.gz	`12158696afd16c3a...`	`c04d253bd527c8e0...`
bitcoin-core-linux-22-res.yml	`70d8be4a51bc67f7...`	`bd865fc816add89a...`
bitcoin-core-osx-22-res.yml	`97d889ba103beac9...`	`e6a03a2e9e8fabee...`
bitcoin-core-win-22-res.yml	`169a1d1fafcb841e...`	`7f3603516013ac23...`
linux-build.log	`46b0db1f12f1fed8...`	`5fcf65342eac3280...`
osx-build.log	`42dd0c03e04b575f...`	`e19160b8ae60c902...`
win-build.log	`9a4cfdc2af2389a0...`	`70fa1e1642d1fc3a...`
bitcoin-core-linux-22-res.yml.diff		`13b5676969e41e52...`
bitcoin-core-osx-22-res.yml.diff		`9b78fa0489fca201...`
bitcoin-core-win-22-res.yml.diff		`0a70271891fb1fc9...`
linux-build.log.diff		`fe013e611195940f...`
osx-build.log.diff		`db6c123c606f097d...`
win-build.log.diff		`227eb1413cdf08a3...`

DrahtBot removed the label Needs gitian build on Dec 12, 2020

in contrib/gitian-descriptors/gitian-osx.yml:42 in 25d39b6109 outdated

  38 | @@ -38,9 +39,9 @@ script: |
  39 |  
  40 |    WRAP_DIR=$HOME/wrapped
  41 |    HOSTS="x86_64-apple-darwin18"
  42 | -  CONFIGFLAGS="--enable-reduce-exports --disable-bench --disable-gui-tests GENISOIMAGE=$WRAP_DIR/genisoimage"
  43 | +  CONFIGFLAGS="--enable-reduce-exports --disable-bench --disable-gui-tests XORRISOFS=$WRAP_DIR/xorrisofs DMG=$WRAP_DIR/dmg"

hebasto commented at 3:10 AM on December 16, 2020:

style nit:

  CONFIGFLAGS="--enable-reduce-exports --disable-bench --disable-gui-tests XORRISOFS=${WRAP_DIR}/xorrisofs DMG=${WRAP_DIR}/dmg"

in Makefile.am:141 in 25d39b6109 outdated

 135 | @@ -135,8 +136,12 @@ $(APP_DIST_DIR)/Applications:
 136 |  
 137 |  $(APP_DIST_EXTRAS): $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt
 138 |  
 139 | -$(OSX_DMG): $(APP_DIST_EXTRAS)
 140 | -	$(GENISOIMAGE) -no-cache-inodes -D -l -probe -V "$(OSX_VOLNAME)" -no-pad -r -dir-mode 0755 -apple -o $@ dist
 141 | +.INTERMEDIATE: $(OSX_TEMP_ISO)
 142 | +$(OSX_TEMP_ISO): $(APP_DIST_EXTRAS)
 143 | +	$(XORRISOFS) -D -l -V "$(OSX_VOLNAME)" -no-pad -r -dir-mode 0755 -o $@ dist

hebasto commented at 3:39 AM on December 16, 2020:

059b84205b2002d238b1147714fb78cff2041818, nit: xorrisofs has -D set by default.

fanquake commented at 4:31 AM on December 16, 2020:

I'd rather we be explicit.

in configure.ac:672 in 25d39b6109 outdated

 668 | @@ -669,7 +669,8 @@ case $host in
 669 |             AC_PATH_TOOL([DSYMUTIL], [dsymutil], dsymutil)
 670 |             AC_PATH_TOOL([INSTALLNAMETOOL], [install_name_tool], install_name_tool)
 671 |             AC_PATH_TOOL([OTOOL], [otool], otool)
 672 | -           AC_PATH_PROGS([GENISOIMAGE], [genisoimage mkisofs],genisoimage)
 673 | +           AC_PATH_PROGS([XORRISOFS], [xorrisofs],xorrisofs)

hebasto commented at 3:41 AM on December 16, 2020:

059b84205b2002d238b1147714fb78cff2041818, style nit:

           AC_PATH_PROGS([XORRISOFS], [xorrisofs], xorrisofs)

in contrib/gitian-descriptors/gitian-osx-signer.yml:40 in 25d39b6109 outdated

  36 | @@ -36,5 +37,5 @@ script: |
  37 |    tar -xf ${UNSIGNED}
  38 |    OSX_VOLNAME="$(cat osx_volname)"
  39 |    ./detached-sig-apply.sh ${UNSIGNED} signature/osx
  40 | -  ${WRAP_DIR}/genisoimage -no-cache-inodes -D -l -probe -V "${OSX_VOLNAME}" -no-pad -r -dir-mode 0755 -apple -o uncompressed.dmg signed-app
  41 | +  ${WRAP_DIR}/xorrisofs -D -l -V "${OSX_VOLNAME}" -no-pad -r -dir-mode 0755 -o uncompressed.dmg signed-app

hebasto commented at 3:43 AM on December 16, 2020:

059b84205b2002d238b1147714fb78cff2041818, nit: xorrisofs has -D set by default.

hebasto approved

hebasto commented at 4:27 AM on December 16, 2020: member

ACK 25d39b61095d7c1ccbc5ec08a9c738aa56ade421, tested cross-compiling on Linux Mint 20 (x86_64), and made the gitian build:

616680767fe519b013f31a8c915719d15aeb45647929a2185e9d7d4b34ed057a  bitcoin-25d39b61095d-osx-unsigned.dmg
19e44ee5454510a7ce77b61ffdfd988f55ec1da8497b0c63803605601ef45ef0  bitcoin-25d39b61095d-osx-unsigned.tar.gz
ab639cd94bcb1ca168312927e1708de84a844be94e9672951d71de1446297f39  bitcoin-25d39b61095d-osx64.tar.gz
574fac8995ab9cb900aeb263e5b920d7ae3d1385523a4dd9cbe58c56184e1761  src/bitcoin-25d39b61095d.tar.gz

build: Replace genisoimage with xorriso

xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are
more maintained, and has the right toggles for us to achieve output
determinism without using blunt tools like faketime.

In this commit, we use xorrisofs from the build environment rather than
building it ourselves using depends. This is not necessary and can be
changed in the future.

From https://wiki.debian.org/genisoimage?action=recall&rev=11 :

> The classical command line interface for production of ISO 9660
> filesystem images is the option set established by program mkisofs.
> For reasons of licensing and other problems with its author, Debian
> ships a fork of mkisofs, called genisoimage, which was split off in
> 2006 and then developed independently.
>
> Meanwhile, genisoimage gets no new features and not even bug fixes. It
> is first choice only if its options -udf or -hfs are needed.
>
> Replacement in most uses cases, especially for bootable ISO 9660
> filesystems, archiving, and backup, is xorrisofs which starts the -as
> mkisofs emulation mode of program xorriso.

0df9819126

build: remove cdrkit package from depends 7587d11ec9

dongcarl force-pushed on Dec 16, 2020

dongcarl commented at 7:27 PM on December 16, 2020: member

Pushed 25d39b61095d7c1ccbc5ec08a9c738aa56ade421 -> 7587d11ec959f15f469bd396d4ad2697729b4ccd: addressed @hebasto's comments, thanks!

6853d97647d9a77e9ddb7e7fdb117917fc64eff35b3b745c99c7f8a9af21dc91  bitcoin-7587d11ec959-osx-unsigned.dmg
e1546285a738ae3a7efea5f743e0e3493c00d3132fd65ff34938783c5ec7ca15  bitcoin-7587d11ec959-osx-unsigned.tar.gz
ad7074ef60772c80330a4fcaeada2ba56e2ce0e66ad29070d8463b3228da5b9e  bitcoin-7587d11ec959-osx64.tar.gz
5fea71337d11f362b086d4424a9254df7d2ec841c7fbac123ac68b71f49ea6f9  src/bitcoin-7587d11ec959.tar.gz

Maintainers: I think this is good to merge.

sipa commented at 7:43 PM on December 16, 2020: member

Concept ACK

laanwj commented at 8:07 PM on December 16, 2020: member

ACK 7587d11ec959f15f469bd396d4ad2697729b4ccd Tested a gitian build:

…
/bin/mkdir -p dist/.background 
/usr/bin/tiffcp -c none dpi36.background.tiff dpi72.background.tiff dist/.background/background.tiff
/home/ubuntu/wrapped/xorrisofs -D -l -V "Bitcoin-Core" -no-pad -r -dir-mode 0755 -o /home/ubuntu/out/bitcoin-7587d11ec959-osx-unsigned.temp.iso dist                                                                 
xorriso 1.4.8 : RockRidge filesystem manipulator, libburnia project.                                      
                                                     
Drive current: -outdev 'stdio:/home/ubuntu/out/bitcoin-7587d11ec959-osx-unsigned.temp.iso'                
Media current: stdio file, overwriteable
Media status : is blank                                                                                   
Media summary: 0 sessions, 0 data blocks, 0 data, 7370m free                                              
xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA 119 rules                              
Added to ISO image: directory '/'='/home/ubuntu/build/bitcoin/distsrc-x86_64-apple-darwin18/dist'
xorriso : UPDATE : 45 files added in 1 seconds                                                            
xorriso : UPDATE : 45 files added in 1 seconds                                                                                                                                                                       
ISO image produced: 14469 sectors                                                                                                                                                                                    
Written to medium : 14469 sectors at LBA 0                                                                                                                                                                           
Writing to 'stdio:/home/ubuntu/out/bitcoin-7587d11ec959-osx-unsigned.temp.iso' completed successfully.    
…

laanwj merged this on Dec 16, 2020

laanwj closed this on Dec 16, 2020

laanwj moved this from the "PRs" to the "Done" column in a project

sidhujag referenced this in commit a409fe578f on Dec 17, 2020

in Makefile.am:139 in 7587d11ec9

 135 | @@ -135,8 +136,12 @@ $(APP_DIST_DIR)/Applications:
 136 |  
 137 |  $(APP_DIST_EXTRAS): $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt
 138 |  
 139 | -$(OSX_DMG): $(APP_DIST_EXTRAS)
 140 | -	$(GENISOIMAGE) -no-cache-inodes -D -l -probe -V "$(OSX_VOLNAME)" -no-pad -r -dir-mode 0755 -apple -o $@ dist
 141 | +.INTERMEDIATE: $(OSX_TEMP_ISO)

hebasto commented at 12:04 PM on December 17, 2020:

This apparently fires a warning:

$ ./autogen.sh 
...
Makefile.am:335: warning: .INTERMEDIATE was already defined in condition !BUILD_DARWIN, which is included in condition TRUE ...
Makefile.am:139: ... '.INTERMEDIATE' previously defined here
...

fanquake referenced this in commit ddbf7a6f50 on Dec 28, 2020

sidhujag referenced this in commit 4a29b4ea73 on Dec 28, 2020

fanquake referenced this in commit 5b41d84b34 on Jan 22, 2021

fanquake referenced this in commit b7e12b350d on Jan 22, 2021

sidhujag referenced this in commit 2a86e38dd8 on Jan 22, 2021

remyers referenced this in commit 8262bc7ed8 on Jan 26, 2021

danben referenced this in commit 0a8bb81ba3 on Feb 1, 2021

laanwj referenced this in commit b4d22654fe on Mar 4, 2021

sidhujag referenced this in commit 1ddff3ed74 on Mar 4, 2021

kittywhiskers referenced this in commit 8e8c61519e on Oct 2, 2021

kittywhiskers referenced this in commit e2b97ab7b0 on Oct 2, 2021

PastaPastaPasta referenced this in commit b5468ef316 on Oct 2, 2021

kittywhiskers referenced this in commit 7f1c225893 on Oct 2, 2021

kittywhiskers referenced this in commit c53ddc93b8 on Oct 2, 2021

PastaPastaPasta referenced this in commit c3955f9f2f on Oct 2, 2021

PastaPastaPasta referenced this in commit 0a3bffbf63 on Oct 2, 2021

kittywhiskers referenced this in commit fd03a23e85 on Oct 5, 2021

PastaPastaPasta referenced this in commit 07b1d52b03 on Oct 5, 2021

kittywhiskers referenced this in commit 696e961336 on Oct 12, 2021

PastaPastaPasta referenced this in commit 72f1dd30d3 on Oct 20, 2021

PastaPastaPasta referenced this in commit 0c79dbc7b6 on Oct 21, 2021

pravblockc referenced this in commit 2e85e08364 on Nov 18, 2021

pravblockc referenced this in commit 10ee8421d7 on Nov 18, 2021

DrahtBot locked this on Feb 15, 2022