build: replace libdmg and genisoimage with xorriso #18151

pull fanquake wants to merge 3 commits into bitcoin:master from fanquake:xorriso_replaces_libdmg_genisoimage changing 10 files +14 −208

fanquake commented at 1:31 pm on February 15, 2020: member
TLDR: Replace two tools that we build and maintain in depends, as part of our macOS deployment, with one that we don’t.

Carl mentioned that xorriso (Debian package) could replace our usage of libdmg-hfsplus and genisoimage. Looks like that’s possible. Have tested gitian builds, and reproducibility looks good. Looking for Concept ACKs.

Also related to #18128, as this eliminates some dependencies. Will post thoughts on using a .zip for distribution there.

TODO:
- xorrisofs vs calling xorriso -as mkisofs
- Compression. Passing zisofs level=9 etc.
build: [WIP] replace libdmg and genisoimage with xorriso 0afed534f9
build: remove libdmg-hfsplus package from depends b3d8d628da
build: remove cdrkit package from depends 581d0e245c
fanquake added the label macOS on Feb 15, 2020
fanquake added the label Build system on Feb 15, 2020
fanquake added the label Needs gitian build on Feb 15, 2020
fanquake requested review from dongcarl on Feb 15, 2020
DrahtBot commented at 6:38 pm on February 15, 2020: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.
dongcarl commented at 6:58 pm on February 15, 2020: member

Should we add native_xorriso to depends?
jonasschnelli commented at 8:42 pm on February 15, 2020: contributor
Nice work! Gitian build works. Tested the DMG https://bitcointools.jonasschnelli.ch/gitian/build/10.

The only difference I observed is, that the new image is not compressed:

Master is around 12 MB (bitcoin-0.19.99-osx-unsigned-2.dmg) where this PR is around 27 (uncompressed).
0-rw-r--r--@ 1 jonasschnelli staff 12494037 15 Feb 21:38 bitcoin-0.19.99-osx-unsigned-2.dmg 1-rw-r--r--@ 1 jonasschnelli staff 27015168 15 Feb 21:36 bitcoin-0.19.99-osx-unsigned.dmg

DrahtBot commented at 6:11 pm on February 16, 2020: member

Gitian builds

File	commit 2a2631fb0dc781504df0f3fe8b42f21cbdb6f20d(master)	commit ff5a90b84660c9acaf79d958a146e6d8bd51a705(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz	`51392e3413128845...`	`deff60fe783e43e7...`
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz	`9c68060f909b176a...`	`6f7b0130323955cb...`
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz	`a4c6bc9d98b02c39...`	`f32fc46496444507...`
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz	`91c276c85c6ea8f0...`	`4643219d4e06c9fe...`
bitcoin-0.19.99-osx-unsigned.dmg	`71c8478f6158f433...`	`e0646d2545ed59d9...`
bitcoin-0.19.99-osx64.tar.gz	`e9f498c9b0694e08...`	`cb8058f79c96afa6...`
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz	`e53b974b60b018a7...`	`57a588c874ae80d1...`
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz	`3ae4b4f4bf44d8ed...`	`6821aa555aa4ba34...`
bitcoin-0.19.99-win64-debug.zip	`88fc928933abadaf...`	`d698a2196041500c...`
bitcoin-0.19.99-win64-setup-unsigned.exe	`78c96ebfdbda69a3...`	`edc68b0d42f434c6...`
bitcoin-0.19.99-win64.zip	`be3e701005cfa682...`	`22c5f0c34591243e...`
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz	`bdbbb809f6569067...`	`0f406d82706569ae...`
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz	`185e77d19a57404c...`	`9d7bfb48554374bd...`
bitcoin-0.19.99.tar.gz	`bd8f4dff76b14fe8...`	`e9ad2d0b24c463aa...`
bitcoin-core-linux-0.20-res.yml	`c3a41885d43ad9c1...`	`bdb62a704c0b378b...`
bitcoin-core-osx-0.20-res.yml	`2ea28d851bd7def3...`	`5088786c9950106b...`
bitcoin-core-win-0.20-res.yml	`db5c46d54aa23a95...`	`b45a5bd753ac2cb5...`
linux-build.log	`67c5b5639d5edc84...`	`77fd68c99f98ae45...`
osx-build.log	`635adff0cbd384f3...`	`8880da5992e6112c...`
win-build.log	`f9061ed05f12c5c0...`	`36988d1d134e8b53...`
bitcoin-core-linux-0.20-res.yml.diff		`51da68c70def5930...`
bitcoin-core-osx-0.20-res.yml.diff		`2c923ed3c4fbcaef...`
bitcoin-core-win-0.20-res.yml.diff		`8ac8ef5190910b1f...`
linux-build.log.diff		`3156db2725d9d2e6...`
osx-build.log.diff		`dcce07030b94b395...`
win-build.log.diff		`f4fd72042720144a...`

DrahtBot removed the label Needs gitian build on Feb 16, 2020
laanwj commented at 5:54 pm on February 26, 2020: member

Concept ACK. Nice. Having to maintain less software as part of the build is good. Travis failure seemed unrelated so restarted.
jonasschnelli commented at 6:41 pm on February 26, 2020: contributor

I’m unsure if xorriso supports HFS+ compression. The libdmg-hfsplus code for compression looks quite extensive https://github.com/planetbeing/libdmg-hfsplus/commit/fc8a21682111a89d1700c4597780d15cae8ddb47 and I haven’t seen a similar implementation in xorrisofs code.

If it doesn’t support HFS+ compression, we might want to discuss if a uncompressed disk image is acceptable (27MB instead of 12).
laanwj commented at 12:38 pm on March 12, 2020: member

That’s a good point. I think that’d be a waste. We might first want to wait (or encourage someone to work on) adding compression there then.
MarcoFalke commented at 12:51 pm on March 12, 2020: member

Even with pruning enabled, the savings of 15 MB are negligible when compared to the minimum disk usage Bitcoin Core requires. So while the savings are nice, it shouldn’t be a requirement?
laanwj added this to the milestone 0.21.0 on Mar 27, 2020
fanquake commented at 12:58 pm on May 25, 2020: member

Rather than doing this, I think switching to a .zip is the better solution, and will result in even less macOS specific code/dependencies for the project to maintain going forward. Going to close this and PR some other changes shortly.
fanquake closed this on May 25, 2020
laanwj referenced this in commit 4acbcfa97d on Dec 16, 2020
MarcoFalke locked this on Feb 15, 2022
fanquake deleted the branch on Sep 29, 2022

Contributors
fanquake DrahtBot dongcarl jonasschnelli laanwj MarcoFalke

Review Requested
dongcarl

Labels
macOS Build system

Milestone
0.21.0

build: replace libdmg and genisoimage with xorriso #18151

TODO:

Conflicts

Gitian builds