contrib: use LIEF for macOS and Windows symbol & security checks

fanquake commented at 3:08 am on April 13, 2021: member

This PR is a proof of concept for using LIEF for the PE and MACHO symbol and security checks. It replaces our current approach of manually parsing the output of objdump & otool. If the consensus is that using LIEF is ok, then I also plan on replacing pixie.py, and using LIEF for all checks. LIEF for Linux is also currently blocked (on the next release, unless we want to build master) on one change for RISC-V that I sent upstream.

LIEF is seemingly well maintained, and is the basis for a number of other tools. It also has some very nice documentation; i.e the Python API for ELF. It also has many builtins we can take advantage of. i.e is_pie, has_nx etc. This means we can consolidate some of our checks. If/when end up using LIEF for lightning then we can consolidate further, and cleanup these scripts. i.e to not parse the binary inside the checks, but once at the start of the script.

Guix builds:

 0# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
 1963a08638c46f9a3d75cd4b0c155d1ca091bbeba27167291adcd3dca03fd4c3d  guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz
 2a3ce927c46b103789a010c41a6ebfafe4548d90ee7d88f2a735c9183b775da5c  guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz
 32503ac8901068805d5e7251fd5cfeb7c1f8ba3528bdfcf3aa1e0c40bfd5c1cbc  guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz
 45798697e58e1788df85aa9e2e4d33fef0456169fcbd2521f13b3b5806ac0d84d  guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz
 54185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2  guix-build-f51237d94d98/output/dist-archive/bitcoin-f51237d94d98.tar.gz
 69b4b8756c5c84295eb6b61b6b32a07a8d07723fb38aaa8f519b6133935061bda  guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz
 7cbd821aa464a9c16f7979dbec1a5e66939e777a567f55f7081499a8d528d42c5  guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz
 8abed530a82e97e3cf621c90a13c0881b0e39ccce2a6f42a3ff80de76e2abc5f7  guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz
 98b6d2bdd8b58ff1f6072bf8693abe3ce773ff3a7d8d2b7218207e69945b9d31b  guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz
10d99cc705032d22ae819975992216899ed960ba25871a05c8789d00b80418511f  guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz
115240ca4f4ef7c62088185224ac319ad9a4a9b40075df10af18d8a6355bca32fb  guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz
12adc16eaee4b51e8615ce8b3be9f6c018698237df4ad6e0886cf0d4ab6bc9e5c4  guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.dmg
13b188af0572ee682d74cc82c7e6e464115205fc130a457cfe19d42ac9ddd267f8  guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.tar.gz
14e764062fde144e6fb5d6dd776c10fc2daa8d775831f7e43247d17a6c6e060c97  guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx64.tar.gz
15dab3d26ac94c669140f7329d14e57ef02b0fe92b8a8f9d96c32a416adea0da0f  guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz
16ca59d4379fbe2b9a52deebeaf88508e0eda4215f28d319aff0781289dd159712  guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz
1752b7c35321a85c4f6c95bf0e687574454b71ede9bec1c9cf17f37c578c888a94  guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win-unsigned.tar.gz
18a543895a00f8ffb3ba50ca68396d52ad5a18dd8efe38730e0049dd70d283a092  guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-debug.zip
19aec050d03c65268a986148500f7341cceb8c5f85287e0e3cde8933ce4b4dee32  guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-setup-unsigned.exe
2057ba33ed6ee8d3a885e342471359301473e83037d5442895beb686921a4c50e9  guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64.zip

Gitian builds:

 0# macOS:
 12f066e852bdd30ac46e5ecdf7619d19d408035c318a3edf0f1893ec2e25efb69  bitcoin-41a1b3d1b130-osx-unsigned.dmg
 28cf8ac4d21740f490262453c330b5f4a5c5b8139dfc1b322efefce3f3b93d1b2  bitcoin-41a1b3d1b130-osx-unsigned.tar.gz
 3cf1b84efdd9d2588a1ce9513580fb56b38bfafe60e18f8adbeedf03521c6c2b2  bitcoin-41a1b3d1b130-osx64.tar.gz
 414995244b0bb3e80e7b79975c9c70fdfb3ee3c04fda3efd5358ce1c4efa3a312  src/bitcoin-41a1b3d1b130.tar.gz
 593881069d5e1dc385c08895a7b035a94eb010325afc2776c99b6aafa21096eb8  bitcoin-core-osx-22-res.yml
 6
 7# Windows:
 84d56dd7713121684b7eaa448679c65df2fd0aa5319bf8d12fb6cfa9f0b005cf7  bitcoin-f51237d94d98-win-unsigned.tar.gz
 94558f4173152b084bcba25aa1a53c605208a70fe20392141b63cefb476528c85  bitcoin-f51237d94d98-win64-debug.zip
10b63feaca010e86d514cfe38d716e3c8a8b8058e4f969b868aaaeb8a8a3d3dc81  bitcoin-f51237d94d98-win64-setup-unsigned.exe
11de7d8586cc91ba391fe911853a99d9fd15fc6f9a60f9b91a0447940173aac67a  bitcoin-f51237d94d98-win64.zip
124185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2  src/bitcoin-f51237d94d98.tar.gz
1345efaca35b5fad0a04dfd06e44f7c00b990aa91c7bf2faea57e020d3491a6cf0  bitcoin-core-win-22-res.yml
14
15# Linux:
16055d646c5f8cf4708008374546176012ff758566a2645a3a01e1a33eab1002fe  bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz
17bfc8b0efc36b0474c88546b12d2723c04b4dc629ae311082025c7e0b8f0d1aa9  bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz
189dfaa5acfffadad8942b32996458013a155d12ed07be76601f232233627b5cb9  bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz
1954eb57905ff8513b9f628707b61aa4659c362fb2f6d17e0ee240b4da3674907d  bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz
20ad98d876616eff578ad8cfd17dfbabe48ed14200823579687d66694bae3d2fe3  bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz
21fe1b421dd1cb6e04d5dc5d341459dc15fa6e15b80906e5d8e0405cf43495e0f7  bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz
229001d95cc7d2722d9d7dd83d9da8e5adf575fddf91b615b76b9bcfece30ecf6f  bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz
239e0650ad2aba70c0fd1608a077e95f335dc1bb4a79eab9b0b56ac87427a4fd4f  bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz
24fbfde0134944d3dbd32991455b0a8abdd334853ab8a4c1a1a4c060d9de071c50  bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz
252fa2cfddce98c44c65305326fc623a7f065129208337503d813a08d51580cb8a  bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz
26b2d6caeee0e3c350a43165c39876ebed8e588958007af0d06996e341c7060683  bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz
27bfdb827e75d43d61462513c9a843620b93c9160d9d246cad13278baaa07f64ea  bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz
284185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2  src/bitcoin-f51237d94d98.tar.gz
2934820a093916fa35b0fd98806a50092f46b20271af7422f43e2a4223ef6f9bb7  bitcoin-core-linux-22-res.yml

fanquake added the label Build system on Apr 13, 2021

fanquake added the label Scripts and tools on Apr 13, 2021

fanquake added the label Needs gitian build on Apr 13, 2021

fanquake added the label Needs Guix build on Apr 13, 2021

hebasto commented at 6:38 am on April 13, 2021: member

Concept ACK.

laanwj commented at 7:33 am on April 13, 2021: member

Concept ACK

If the consensus is that using LIEF is ok, then I also plan on replacing pixie.py

SGTM the only reason i created pixie is to avoid having to install an external python library like this, but if that is what we are going to do, it is better to use it for everything. I think the only reason to prefer the current approach is to have close control over the checks, but there are also advantages to using an external well-maintained library. I do wonder if we should fix a specific version instead of ‘install the latest from PIP’ which sounds suboptimal for security-critical things.

(everything is better than parsing the text output of tools, in any case)

fanquake commented at 12:59 pm on April 13, 2021: member

I do wonder if we should fix a specific version instead of ‘install the latest from PIP’ which sounds suboptimal for security-critical things.

I might be misunderstanding, but at the moment we’ve pinned the installation to version 0.11.4. We could also enhance this by checking hashes during installation.

DrahtBot commented at 2:09 pm on April 13, 2021: member

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#21135 (scripts: check for control flow instrumentation in security-check.py by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

dongcarl commented at 3:11 pm on April 13, 2021: member

Concept ACK, will rebase #20980 on top in the next few days

jarolrod commented at 3:28 pm on April 13, 2021: member

Contributing GUIX hashes, mine match fanquake:

 0find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
 1
 287c3856adb9bac6b5cc2ad440b928a495b84491bfaf89222b07f674b32264237  guix-build-9c5eeb54848f/output/aarch64-linux-gnu/bitcoin-9c5eeb54848f-aarch64-linux-gnu-debug.tar.gz
 3fed2d30720bafb8a80adf73557eae4c1440dba76d35e609234a81b03bd9dc0f9  guix-build-9c5eeb54848f/output/aarch64-linux-gnu/bitcoin-9c5eeb54848f-aarch64-linux-gnu.tar.gz
 4e4763300c5220ea6c3617b89da791f77b538beba3e0f14618aec1e41a3af07ec  guix-build-9c5eeb54848f/output/arm-linux-gnueabihf/bitcoin-9c5eeb54848f-arm-linux-gnueabihf-debug.tar.gz
 5a9ab02f44f8324b328687d5f1bb11e5d3db6010825755386292f31b3ac8db2b1  guix-build-9c5eeb54848f/output/arm-linux-gnueabihf/bitcoin-9c5eeb54848f-arm-linux-gnueabihf.tar.gz
 6ab3b8eaef7f15ae68dbea2a95dffbca1688558b6ac97c6fd1132a00268d39602  guix-build-9c5eeb54848f/output/dist-archive/bitcoin-9c5eeb54848f.tar.gz
 78f4b873a5ebf9c3c5690c242d8050b30879bbe5ed4c5bad2e02208dbbecf12a7  guix-build-9c5eeb54848f/output/powerpc64-linux-gnu/bitcoin-9c5eeb54848f-powerpc64-linux-gnu-debug.tar.gz
 8b3551817a96c6f38ae2839ba0c1172aff270c7d275f6c6e9ee1aa9f037e769d9  guix-build-9c5eeb54848f/output/powerpc64-linux-gnu/bitcoin-9c5eeb54848f-powerpc64-linux-gnu.tar.gz
 9ffb371f95e878ffe15f592b99f870f45ee30c23bc15d69736ef2f00d8b0778bb  guix-build-9c5eeb54848f/output/powerpc64le-linux-gnu/bitcoin-9c5eeb54848f-powerpc64le-linux-gnu-debug.tar.gz
10c983b5973aba7095a1b30d67525a286c64aaba8fa5ea9004e6f35e470e7a257d  guix-build-9c5eeb54848f/output/powerpc64le-linux-gnu/bitcoin-9c5eeb54848f-powerpc64le-linux-gnu.tar.gz
1149f2062871bac870a643412bd73ad3208662a804987022aea38c33fa71b7017b  guix-build-9c5eeb54848f/output/riscv64-linux-gnu/bitcoin-9c5eeb54848f-riscv64-linux-gnu-debug.tar.gz
120767b69334079c99358e8255e4b551d8042a95e31af109c81c7b4a3715b8f152  guix-build-9c5eeb54848f/output/riscv64-linux-gnu/bitcoin-9c5eeb54848f-riscv64-linux-gnu.tar.gz
130e6ed0cefabba736c0c720ecf9e6280a517f1f881daf7e5e6ed273e8b8224501  guix-build-9c5eeb54848f/output/x86_64-apple-darwin18/bitcoin-9c5eeb54848f-osx-unsigned.dmg
14a5042d916d709d032e8671913caf4330127454e5e6471630d495b085150027ec  guix-build-9c5eeb54848f/output/x86_64-apple-darwin18/bitcoin-9c5eeb54848f-osx-unsigned.tar.gz
1534f86071eafd69797166b95e7a53a25118fa26bbab8eda57f6be04e20068ea06  guix-build-9c5eeb54848f/output/x86_64-apple-darwin18/bitcoin-9c5eeb54848f-osx64.tar.gz
1600561ab6b2855e2e5c6751dc9eea59f056f76fb6b862911e3a17d21cc6600e8b  guix-build-9c5eeb54848f/output/x86_64-linux-gnu/bitcoin-9c5eeb54848f-x86_64-linux-gnu-debug.tar.gz
171358fcdaf5887b6f8c47561f8e6c3299b85d9b74f843b818622ebb1308625d47  guix-build-9c5eeb54848f/output/x86_64-linux-gnu/bitcoin-9c5eeb54848f-x86_64-linux-gnu.tar.gz
1893a522386b3c845ee7b278e1c49fff71541988478573e01a5dfbbd013dacb9d8  guix-build-9c5eeb54848f/output/x86_64-w64-mingw32/bitcoin-9c5eeb54848f-win-unsigned.tar.gz
19c1af15e81cb9b468846f20a365b0cbdf284fd4d492a7d01ab18b37eb9a7dcdcd  guix-build-9c5eeb54848f/output/x86_64-w64-mingw32/bitcoin-9c5eeb54848f-win64-debug.zip
20cfca49fa7b950b8ca8a7ee8834b165d7caaa98541b4299ef54ea8db2def35032  guix-build-9c5eeb54848f/output/x86_64-w64-mingw32/bitcoin-9c5eeb54848f-win64-setup-unsigned.exe
2160409a138d7e9f9e27ac8349000342f75d6a1489a79450a44b0d8d0142f8c39d  guix-build-9c5eeb54848f/output/x86_64-w64-mingw32/bitcoin-9c5eeb54848f-win64.zip

laanwj commented at 10:42 am on April 14, 2021: member

I might be misunderstanding, but at the moment we’ve pinned the installation to version 0.11.4.

That’s good. Yes, I see now!

We could also enhance this by checking hashes during installation.

Would be even better. TIL that pip has this built-in!

DrahtBot commented at 2:26 pm on April 14, 2021: member

Gitian builds

File	commit a1f0b8b62eb851c837a3618583b7c2fd4d12006c(master)	commit 195f6221a9887ffbd30a79851c618312816c84e0(master and this pull)
*-aarch64-linux-gnu-debug.tar.gz	`f696795ff1a65765...`	`bf9c7b63dfe13e09...`
*-aarch64-linux-gnu.tar.gz	`97a0a23136d3c17e...`	`146fc1948786a888...`
*-arm-linux-gnueabihf-debug.tar.gz	`b41962bd47a4372f...`	`43139d6adf1f3f68...`
*-arm-linux-gnueabihf.tar.gz	`b8ffb7e37b9a23eb...`	`fa8002ca6ee555f1...`
*-osx-unsigned.dmg	`402d56525eafe6ba...`	`8505a8aa28df1aa2...`
*-osx64.tar.gz	`4bc0c6db97ae051d...`	`235286b371addde7...`
*-powerpc64-linux-gnu-debug.tar.gz	`2ca3e858c79f6ce9...`	`60af00f2b2c4e947...`
*-powerpc64-linux-gnu.tar.gz	`69cc301decf65451...`	`d5cc56afa710d876...`
*-powerpc64le-linux-gnu-debug.tar.gz	`34db07a0a6ff9e76...`	`5a068019b78169e2...`
*-powerpc64le-linux-gnu.tar.gz	`aed7277874860c23...`	`a90134268b705978...`
*-riscv64-linux-gnu-debug.tar.gz	`861e4e60648b9433...`	`01fb9eeed41edd8c...`
*-riscv64-linux-gnu.tar.gz	`ba3ecbdc5832c154...`	`65a8c80ba246f64b...`
*-win64-debug.zip	`f977018dbdaea330...`	`b9012a802744a47b...`
*-win64-setup-unsigned.exe	`a06146d042b6dcae...`	`d69d08c1ce75ed8f...`
*-win64.zip	`5dfb0eb0ba2e4802...`	`d869df07b42e2276...`
*-x86_64-linux-gnu-debug.tar.gz	`28c29d860663e17f...`	`f0fe8fe48f5bf207...`
*-x86_64-linux-gnu.tar.gz	`9339e90ebb113618...`	`f30b530600b2711d...`
*.tar.gz	`2be2b8be4279237d...`	`c37fb5807166b635...`
bitcoin-core-linux-22-res.yml	`aac37904a0c9bd45...`	`4d83e6f66cfcd30f...`
bitcoin-core-osx-22-res.yml	`03f7c9a49a5731c0...`	`7dca479f4e966a26...`
bitcoin-core-win-22-res.yml	`ace7252156c5da27...`	`75528ef0e497a1aa...`
linux-build.log	`7ac5c6d618981a55...`	`a003a1250dc9420b...`
osx-build.log	`b01aced429650ee3...`	`134f418db7d000f2...`
win-build.log	`3e2c47cecc5ddb30...`	`65380aa40b15c3d0...`
bitcoin-core-linux-22-res.yml.diff		`762d6cfcfcb5c6f7...`
bitcoin-core-osx-22-res.yml.diff		`0d37f10bb80f316a...`
bitcoin-core-win-22-res.yml.diff		`2567ba8bd9497ff3...`
linux-build.log.diff		`4b47e745c90bfe0e...`
osx-build.log.diff		`0d0092935ce84e4b...`
win-build.log.diff		`d509b7a9e735efd9...`

DrahtBot removed the label Needs gitian build on Apr 14, 2021

practicalswift commented at 8:17 pm on April 15, 2021: contributor

Concept ACK

LIEF is battle tested and very well maintained. Great choice!

dongcarl commented at 7:55 pm on April 28, 2021: member

Rebased #20980 on top of this, everything seems to work fine.

fanquake removed the label Needs Guix build on May 1, 2021

fanquake force-pushed on May 1, 2021

fanquake commented at 8:53 am on May 1, 2021: member

Rebased. Addressed installing LIEF in the CI. Updated Guix and gitian hashes will be added to the PR description.

fanquake force-pushed on May 1, 2021

DrahtBot commented at 9:32 am on May 3, 2021: member

🕵️ @achow101 @sipa have been requested to review this pull request as specified in the REVIEWERS file.

laanwj commented at 10:49 am on May 3, 2021: member

Code review ACK f51237d94d9858749e53c4f70c9a8ee3c8a17c30

DrahtBot added the label Needs rebase on May 4, 2021

DrahtBot commented at 12:29 pm on May 4, 2021: member

🐙 This pull request conflicts with the target branch and needs rebase.

Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a “draft”.

gitian: install LIEF in gitian container 465967b5ef

guix: install LIEF in Guix container

Co-authored-by: Carl Dong <contact@carldong.me>

2e7a9f7ade

contrib: use f strings in security-check.py 7e7eae7aa8

contrib: use LIEF for macOS security checks a25b2e965c

contrib: use LIEF for PE security checks 8e1f40dd9a

contrib: add PE PIE check to security checks 0f5d77c8e4

contrib: use f strings in symbol-check.py a632cbcee5

contrib: use LIEF in macOS symbol checks e93ac26b85

contrib: use LIEF in PE symbol checks 2aa1631822

contrib: consolidate PIE and NX security checks 955140b326

test: install lief in CI 7fc5e865b9

fanquake force-pushed on May 4, 2021

fanquake removed the label Needs rebase on May 4, 2021

laanwj commented at 11:22 am on May 5, 2021: member

re-ACK 7fc5e865b93af59364e9c8bf75ec68b4decc7e5d

laanwj merged this on May 5, 2021

laanwj closed this on May 5, 2021

sidhujag referenced this in commit bf4550dd0b on May 5, 2021

fanquake deleted the branch on May 6, 2021

gwillen referenced this in commit cc047c66f0 on Jun 1, 2022

DrahtBot locked this on Aug 16, 2022

contrib: use LIEF for macOS and Windows symbol & security checks #21664

Conflicts

Gitian builds