macOS App Notarization & Stapling

fanquake commented at 8:39 am on April 9, 2019: member

The Apple notary service is an automated system that scans your software for malicious content, checks for code-signing issues, and returns the results to you quickly. If there are no issues, the notary service generates a ticket for you to staple to your software; the notary service also publishes that ticket online where Gatekeeper can find it.

Beginning in macOS 10.14.5, all new or updated kernel extensions and all software from developers new to distributing with Developer ID must be notarized in order to run. In a future version of macOS, notarization will be required by default for all software.

The currently available version of macOS is 10.14.4.

Apple’s notary service requires you to adopt the following protections:

Enable code-signing for all of the executables you distribute.
Enable the Hardened Runtime capability for your executable targets, as described in Enable hardened runtime.
Use a Developer ID application, kernel extension, or installer certificate for your code-signing signature.
Include a secure timestamp with your code-signing signature. (The Xcode distribution workflow includes a secure timestamp by default. For custom workflows, include the --timestamp option when running the codesign tool.)
Don’t include the com.apple.security.get-task-allow entitlement with the value set to any variation of true.
Link against the macOS 10.9 or later SDK.

More info available here: https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution

fanquake added the label macOS on Apr 9, 2019

fanquake added the label Build system on Apr 9, 2019

maflcko added this to the milestone 0.19.0 on Apr 9, 2019

maflcko commented at 1:46 pm on April 9, 2019: member

@jonasschnelli @theuni

flack commented at 9:14 pm on April 10, 2019: contributor

If I’m reading this right, it only applies to “software from developers new to distributing with Developer ID”, i.e. since you already shipped signed Mac binaries, you should be exempt for the time being. Or were the binaries signed without a Developer ID somehow?

fanquake commented at 0:47 am on September 10, 2019: member

Moving this to v0.20.0 as we shouldn’t have to solve anything here for v0.19.0.

fanquake removed this from the milestone 0.19.0 on Sep 10, 2019

fanquake added this to the milestone 0.20.0 on Sep 10, 2019

moneyball commented at 10:18 pm on October 30, 2019: contributor

I see this message with v0.19 RC3 on MacOS Catalina.

Sjors commented at 11:15 am on October 31, 2019: member

Same issue (macOS 10.15 Catalina).

Note that when copying the DMG from my Gitian machine it doesn’t show the same warning. Only when I download it. Files are identical, but presumably macOS tracks this stuff somewhere.

moneyball commented at 4:25 pm on October 31, 2019: contributor

I’m on 10.15 too. I wonder what the difference is?

On Thu, Oct 31, 2019 at 4:18 AM Sjors Provoost notifications@github.com wrote:

On 10.15 I don’t get this message when I update my existing installation from the signed DMG, nor when I delete it.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/15774?email_source=notifications&email_token=AACPUAYZQAGRSGET36JZK3DQRK5JNA5CNFSM4HEPXQW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECXLWDQ#issuecomment-548322062, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACPUA6SMQ3O5E4I237DB63QRK5JNANCNFSM4HEPXQWQ .

Sjors commented at 5:27 pm on October 31, 2019: member

@moneyball none. I updated my comment to reflect that it matters if you get the file as a download or via SSH from a Gitian machine.

Fuzzbawls commented at 0:58 am on November 11, 2019: contributor

If I’m reading this right, it only applies to “software from developers new to distributing with Developer ID”, i.e. since you already shipped signed Mac binaries, you should be exempt for the time being. Or were the binaries signed without a Developer ID somehow?

This is how I too initially understood the documentation. However, as others have pointed out, this does not seem to be the case. With my own project that has distributed code signed binaries for a while now, those same code signed binaries, when downloaded, are actually blocked from running on Catalina with the message shown in @moneyball ’s screenshot.

Notarization can be done either after-the-fact (post-release) for existing code signed binaries, or during the release process’ final steps using altool (see https://developer.apple.com/documentation/xcode/notarizing_your_app_before_distribution/customizing_the_notarization_workflow for command examples starting at the “Upload Your App to the Notarization Service” section). This would need to be performed by the code signer.

If all goes well, the binary package will then be recognized by GateKeeper, providing that an active internet connection is available. It is possible to “staple” the notarization receipt to the binary package so that an internet connection is not required, but that would change it’s sha256sum, which would no longer match the hashes from gitian’s output.

Note: There is an intermittent bug in Apple’s notarization system that may block the binary from being notarized properly. Details can be found on Apple’s Developer Forums: https://forums.developer.apple.com/thread/118465

Sjors commented at 11:39 am on November 14, 2019: member

If all goes well, the binary package will then be recognized by GateKeeper, providing that an active internet connection is available.

Which implies giving Apple an overview of all Bitcoin Core macOS user, the exact version they installed and when. Unless GateKeeper downloads a list of hashes, rather than uploading the hash.

It is possible to “staple” the notarization receipt to the binary package so that an internet connection is not required, but that would change it’s sha256sum, which would no longer match the hashes from gitian’s output.

This seems worth some additional hassle for the privacy benefit. During a recent IRC meeting the idea came up to make a tool that strips the extra information from the package and then checks the resulting checksum.

NicolasDorier commented at 6:53 am on December 9, 2019: contributor

I had to make sure my app was notarized and “stapled”. For those interested, here is what I needed to do to automate the task on travis:

https://github.com/btcpayserver/BTCPayServer.Vault/blob/master/Build/travis/applesign.sh

Basically, I needed to:

extract the unsigned dmg,
sign the app,
rebundle the app as a dmg,
sign the dmg,
notarize the dmg,
wait that it succeed
Apply the staple to the dmg

Note that applying the staple to the dmg is optional. However, if you don’t apply it, gatekeeper will need internet connection to verify the dmg.

I think that I should also staple the .App folder itself and not only the dmg if I wanted gatekeeper to be 100% happy when offline. (yes, you can staple the .App folder… I have no idea why it works, where the staple get saved?) The documentation is very bad, so I don’t really know…

If you want to also staple the .App I think you need a two steps notarization: First the .App folder as a tar file, then the dmg after it has been built.

Take a good note about the parameters I passed to codesign, this took a while to get right.

molnard commented at 2:48 pm on February 12, 2020: none

Hi there, I have just finished signing a dotnet core application for distributing app outside the Mac App Store. There are several recommendations by apple and as @NicolasDorier mentioned the documentation to do it outside XCode is very bad quality.

Take a look at my code: https://github.com/molnard/WalletWasabi/blob/4f499aab964d7c9298a0f041633fb20927d3a18f/WalletWasabi.Packager/MacSignTools.cs

The whole PR: https://github.com/zkSNACKs/WalletWasabi/pull/3041

Oh, just one more thing - I had to move our packager from linux to mac because there is no way to make it work otherwise.

jonasschnelli commented at 4:34 pm on February 12, 2020: contributor

I looked a bit into this issue and I think a possible approach could be:

During the manual macOS code signing process via detached-sig-create.sh (currently done by me), we could also notarize the .app bundle (and/or the bitcoin-qt binary).
We “staple” the tickets from apple notarisation service directly onto the .app bundle (kinda pack the signature for the notarization into the application bundle so no network connection is required)
We create the dmg with gitian (pulling in the detached signature & notarization). This is when the gitian people build the win/osx signed builds.
We manually notarize the dmg file (2nd manual step, executed by the macOS code signing key holder)

(requires testing in order to make sure this process works)

Downsides:

A 2nd manual step through macOS code signing maintainer

Alternatives:

Create directly the dmg in 1st manual code signing step (kinda defeats the purpose of gitian,… so a bad alternative)

Risks:

Since the final .dmg notarization is a manual one man show step, the question pops up if this could be exploited. I guess it can’t. It only tells Apples Gatekeeper server that the gitian built .dmg file was notarized with someone holding the macOS code signing keys. Manually notarizing a malicious .dmg by a maintainer would still require to distribute that malicious .dmg somehow (also codesign).

Anyways, our binary distribution security is based on a completely different layer (deterministic builds with multiple signatures). The os protection with code-signing and notarization is nice but provides much less trust/security IMO.

Ping @theuni @dongcarl.

Sjors commented at 4:38 pm on February 12, 2020: member

In practice, given that the macOS code signing maintainer is also a Gitian builder, they can do both manual steps in a single session?

jonasschnelli commented at 5:37 pm on February 12, 2020: contributor

For test purposes, I notarized the 0.19.1rc1 dmg file (RC1! not 2!). If anyone can test if the 0.19.1rc1 gives them a notarization warning ("…apple can not check the app for malicious code"-error should be gone).

Sjors commented at 5:49 pm on February 12, 2020: member

I just downloaded it (note: it won’t check the notary stuff if you obtain it via SSH or build yourself) and didn’t get a notarization warning. Just the usual “you downloaded from bitcoincore.org, are you sure”.

jonasschnelli commented at 6:44 pm on February 12, 2020: contributor

Great. So just notarizing the .dmg during the release process would work. Additionally, we could staple the notarization ticket onto the app bundle. But that would be non trivial via our gitian build process. “Stapling” would allow detecting the notarization even if no internet connection is available.

I propose that we start with the “we just notarize the dmg”-approach and see if there is demand/need for the “stapling”-approach.

luke-jr commented at 3:15 pm on February 21, 2020: member

How do users verify the end DMG is generated deterministically by gitian?

jonasschnelli commented at 5:46 pm on February 21, 2020: contributor

@luke-jr it’s still the same process (detached signature). Just an additional flag. The actual notarization binds to the hash of the deterministic built dmg.

luke-jr commented at 6:42 pm on February 21, 2020: member

So the notarization is a separate file? How does the end user get that?

jonasschnelli commented at 7:14 pm on February 21, 2020: contributor

@luke-jr From my understanding it works like this (simplified):

We build the dmg in the same was as now
The macOS codesign key holder notarizes the .dmg file:
- During notarization, it uploads the .dmg file to apples server and checks its content
- If the application has all required flags set, etc., it will approve the notarization and stores its hash, etc. on the Apple servers (notarization ticket)
Once a user starts an application the first time, it checks it’s installers hash (the .dmg file) against Apples notarization server and if the hash has an approved notarization ticket, the app launches.

So it needs online connectivity the first time the app launches. Though user can disable the notarization check in the macOS security settings.

It’s mandatory since 10.14.5 (means the default security setting only allows notarized apps to launch).

There is an option to add the notarization ticket directly to the app to allow offline users to check the notarization (Apple calls that stapling). But it would complicated our build process. Can be done later.

Sjors commented at 7:23 pm on February 21, 2020: member

Once a user starts an application the first time, it checks it’s installers hash (the .dmg file) against Apples notarization server and if the hash has an approved notarization ticket, the app launches.

This part has me worried. Does Apple use a (provable) anonimzation technique here? Or is this giving them a list of IP address of Bitcoin Core users? Stapling the notarisation onto the DMG seems better, though it makes the process more complicated.

jonasschnelli commented at 8:30 am on February 22, 2020: contributor

@Sjors: this is indeed a good point. Though I don’t know if we can even prevent the ugly online check when we staple the notarization ticket to the application. Maybe it only checks it when the user is offline. But I agree, if we decide to notarize (and I guess we have to), stapling the ticket to the app is the preferred approach.

I just checked what stapling actually does. It adds a file CodeResources to the application bundle (all other files are untouched). We could add that CodeResources file to the detach signature git and “apply” it during gitian-osx-signer process.

The process would look like:

the macOS code signer needs to run the .dmg creation step (a manual gitian-osx-signer.yml) after he ran the detached-sig-create.sh
… then, he needs to notarize the created .dmg file
… staple the notarization ticket to the .app folder (creates the CodeResources file)
… commit the detached signatures & the CodeResources file to https://github.com/bitcoin-core/bitcoin-detached-sigs
… all gitian builders pull the signature & the CodeResource file and apply it to the .app folder and create the dmg

Downside:

More work for the macOS code signer (acceptable IMO)
Notarization happens before we have proof of determinism and the possibility to check the hashes of other gitian builds (we already have that problem with code signing)
No current reliable proof if this would not break determinism (I think it wouldn’t)

Sjors commented at 11:01 am on February 23, 2020: member

This sounds like a reasonable approach to me.

fanquake removed this from the milestone 0.20.0 on Apr 7, 2020

fanquake added this to the milestone Future on Apr 7, 2020

hebasto commented at 11:56 am on May 24, 2020: member

FWIW, https://sigpipe.macromates.com/2020/macos-catalina-slow-by-design/ “Spawning a new Process” subheader.

maflcko removed this from the milestone Future on Jan 26, 2022

maflcko added this to the milestone 23.0 on Jan 26, 2022

fanquake removed this from the milestone 23.0 on Mar 1, 2022

fanquake added this to the milestone 24.0 on Mar 1, 2022

fanquake commented at 1:43 pm on March 1, 2022: member

At this point this isn’t going to happen for 23.0. Moving this to 24.0

prusnak commented at 3:55 pm on March 16, 2022: contributor

It seems that Xcode 13 ships with a new tool called notarytool making the whole process easier: https://scriptingosx.com/2021/07/notarize-a-command-line-tool-with-notarytool/

Also the process is supposed to be 10x faster than the old method via altool - see https://github.com/electron/electron-notarize/pull/82

prusnak commented at 9:51 pm on August 8, 2022: contributor

might be relevant:

luke-jr commented at 1:43 am on August 9, 2022: member

But it still phones home and tells Apple you installed Bitcoin Core, right?

fanquake removed this from the milestone 24.0 on Sep 13, 2022

fanquake added this to the milestone Future on Sep 13, 2022

fanquake commented at 2:38 pm on September 13, 2022: member

Moving this to future. We can revisit depending on what happens during testing of release candidates for 24.x.

gruve-p commented at 6:31 pm on October 2, 2022: contributor

I see this message with v0.19 RC3 on MacOS Catalina.

I am getting this error message when opening the rc1 24.x

Sjors commented at 9:14 am on October 3, 2022: member

I’m not sure if the release candidates get notarized (and when)?

fanquake commented at 9:20 am on October 3, 2022: member

I’m not sure if the release candidates get notarized (and when)?

Notarization is not occuring at any point, for any binaries.

Sjors commented at 9:25 am on October 3, 2022: member

Ok, in that case the error message is as expected. You have to right click the first time and then hold option when you click Open. Needless to say this is an annoying anti open source move by Apple…

gruve-p commented at 11:32 am on October 3, 2022: contributor

Ok, in that case the error message is as expected. You have to right click the first time and then hold option when you click Open. Needless to say this is an annoying anti open source move by Apple…

This does not work when a macbook is mdm’ed (corporate and student macbooks). Can we get the rc1 dmg notarized and skip it on the release binaries?

maflcko referenced this in commit 6d40a1a7e7 on Dec 21, 2022

AdmiralOrange commented at 10:24 am on February 16, 2023: none

Guys - the user experience is horrible. I download Bitcoin Core from the official webpage - Norton kills my download as it falsely recognizes it as a virus, if I temporarily disable Norton Apple prevents me from opening the downloaded program cause the signing key is outdated and BTC Core is not enabled for multi-key-signing and still depending on the outdated key of the departed Wladimir van der Laan, if I manage to get a valid signing key and work around Apple preventions Norton kills the running software. This is the future of money? So what is the right way to run BTC Core?

willcl-ark commented at 3:04 pm on April 23, 2023: contributor

The policy reads to me that notarization will always phone home, even in the case of having an externally-stapled ticket, unless a connection is not available in which case the stapled ticket is locally verified. Therefore it seems to me that we have two options available:

Notarize the binary and somehow communicate to users that if they don’t want Apple to know that they are running Bitcoin Core they must start it up with no internet connection (under the assumption that Gatekeeper only verifies first run).
Do not notarize the binary and include instructions similar to those seen on this page (copied below) in the MacOS DMG/zip archive, which imply that it is (permanently) saved as an exception.

Personally I’d tend towards the latter option.

This does not work when a macbook is mdm’ed (corporate and student macbooks). Can we get the rc1 dmg notarized and skip it on the release binaries?

I don’t know if option 2 works on an “mdm’ed” macbook though…

If you want to open an app that hasn’t been notarized or is from an unidentified developer

Running software that hasn’t been signed and notarized may expose your computer and personal information to malware that can harm your Mac or compromise your privacy. If you’re certain that an app you want to install is from a trustworthy source and hasn’t been tampered with, you can temporarily override your Mac security settings to open it.

If you still want to open an app for which the developer cannot be verified, open System Preferences.*

Go to Security & Privacy. Click the Open Anyway button in the General pane to confirm your intent to open or install the app.

The warning prompt reappears, and if you’re absolutely sure you want to open the app anyway, you can click Open.

The app is now saved as an exception to your security settings, and you can open it in the future by double-clicking it, just as you can any authorized app.

bitcoin deleted a comment on Apr 24, 2023

knst referenced this in commit 2170604f33 on Nov 20, 2023

knst referenced this in commit 0417209266 on Nov 20, 2023

knst referenced this in commit cca5a467de on Nov 20, 2023

PastaPastaPasta referenced this in commit d7bdf8e424 on Nov 24, 2023

maaku commented at 0:56 am on December 21, 2023: contributor

I’m a bit mystified that this issue is over 4 years old. Near as I can tell, the release manager just needs to pas --hardened-runtime to the signing command, and the resulting binaries will be notorizable. I’ve personally tested this on v26.

achow101 commented at 1:59 am on December 21, 2023: member

I’m a bit mystified that this issue is over 4 years old. Near as I can tell, the release manager just needs to pas --hardened-runtime to the signing command, and the resulting binaries will be notorizable. I’ve personally tested this on v26.

I don’t think it’s technically difficult, although it will probably need additional tooling and procedural changes to work in the Guix build system.

I believe the current main objection is a philosophical one to the fact that notarizing means that installing the application will result in Macs phoning home to Apple, regardless of how the notarization is applied. This essentially hands a list of everyone who has installed Bitcoin Core on a Mac over to Apple, which some people are uncomfortable with us doing that.

maaku commented at 3:45 am on December 21, 2023: contributor

Not if the notarization is stapled to the binary, no. That should prevent phoning home. I have not used Wireshark to verify this, but that’s what the documentation claims.

jonasschnelli commented at 7:21 am on December 21, 2023: contributor

@maaku My tests in 2020 (https://github.com/bitcoin/bitcoin/pull/18187#issuecomment-592453829) showed me, that regardless of stapling, macOS does an onlinecheck of the notarization (if the user has internet connection).

Not sure if that has changed. Might be worth to redo #18187 (comment).

That unnecessary “calling-apple” was the reason to not do notarization back in 2020.

maaku commented at 7:52 am on December 21, 2023: contributor

@jonasschnelli Sounds like bug 57278824, which was fixed back in 2020 just a few months after you ran your test:

There’s one further wrinkle here: Once Brigitte removed the com.apple.security.cs.disable-library-validation entitlement from their app the problem didn’t go away! And that’s the result of a historical bug in Gatekeeper (r. 57278824). Prior to 10.15.4 Gatekeeper does not recognise the implied library validation that you get when you enabled the hardened runtime. So, even though you have library validation enabled via the hardened runtime, Gatekeeper still runs these extra checks )-:

https://developer.apple.com/forums/thread/695026

Also, I can’t verify what steps you used to test the notary because your link to your own website is down, and not captured by archive.org.

fanquake commented at 10:19 am on December 21, 2023: member

Not if the notarization is stapled to the binary, no. That should prevent phoning home. I have not used Wireshark to verify this, but that’s what the documentation claims.

Can you link to this documentation you’re talking about? Or is that the developer thread above?

Reading https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/customizing_the_notarization_workflow#3087720, it’s not clear that no phone home happens if there is a ticket present, just that, if there is one, it can be used as a fallback when there is no network connection, but I assume Gatekeeper will always attempt an online check first:

Notarization produces a ticket that tells Gatekeeper that your app is notarized. After notarization completes, the next time any user attempts to run your app on macOS 10.14 or later, Gatekeeper finds the ticket online. This includes users who downloaded your app before notarization.

You should also attach the ticket to your software using the stapler tool, so that future distributions include the ticket. This ensures that Gatekeeper can find the ticket even when a network connection isn’t available.

gruve-p commented at 10:46 am on December 21, 2023: contributor

Might be relevant:

SomberNight commented at 4:32 pm on December 21, 2023: contributor

Does the OS not phone home if the application is not notarized? If so, how can it distinguish notarized+not_stapled vs not_notarized applications?

maaku commented at 11:19 pm on December 22, 2023: contributor

If so, how can it distinguish notarized+not_stapled vs not_notarized applications?

You’ve hit the nail on the head: notarized+not_stapled !notarized apps are bit-for-bit identical. It phones home for all applications being run, except those with stapled notary (maybe, this is disputed above).

fanquake closed this on Jan 11, 2024

fanquake reopened this on Jan 11, 2024

fanquake renamed this:
~~macOS App Notarization~~
macOS App Notarization & Stapling
on Jan 11, 2024

achow101 commented at 9:26 pm on January 18, 2024: member

Did some tests on MacOS 13.5.1. There is no difference in phone home behavior for no notarization at all, notarization without stapling, and stapled notarization. Tested with Bitcoin Core 26.0, Wireshark (notarized but not stapled), and Firefox (notarized and stapled), and did packet capture for each with Wireshark. They all resulted in communication with api.apple-cloudkit.com, which, AFAIK, is the domain for the notarization check.

To replicate this experiment, the command spctl -a -vvv -t install <path to app> can be used to do a notarization check that will always phone home, and stapler validate -v <path to app> can be used to check whether an app has a stapled notarization (note that this command also phones home, I believe to check that the stapled notarization matches).

Given that MacOS is just phoning home all the time, I think we should move ahead with notarizing our MacOS releases to improve the installation experience. Although stapling sounds like a good idea, at this point, I sincerely doubt that Apple will change their tooling to stop phoning home for stapled apps, but maybe we should do so in the hopes that they change their minds.

willcl-ark commented at 11:25 am on January 19, 2024: contributor

Given that MacOS is just phoning home all the time, I think we should move ahead with notarizing our MacOS releases to improve the installation experience. Although stapling sounds like a good idea, at this point, I sincerely doubt that Apple will change their tooling to stop phoning home for stapled apps, but maybe we should do so in the hopes that they change their minds.

In this case notarizing MacOS releases seems the most reasonable course of action to me too. Of course we’d like the additional privacy of not phoning home, if it were feasible. But in the case it’s unavoidable then improving the installation experience for (all) MacOS users makes sense.

I am curious though, what happens if the user does this:

runs sudo spctl --master-disable
installs bitcoin core
runs bitcoin core
runs sudo spctl --master-enable

Does the app phone home during that sequence? If not, does it phone home the next time Bitcoin Core is started (after Gatekeeper was re-enabled)?

macOS App Notarization & Stapling #15774

If you want to open an app that hasn’t been notarized or is from an unidentified developer