How to handle notarizing apps and kexts that are generated programmatically?

My company has an automatic release process for our driver and associated apps. The driver is designed to work with an arbitrarily large number of different hardware configurations, and each time we make a release, our release process automatically does the following:


1. Generates a new Info.plist file for the kext containing the PID / VID combinations that the driver supports and then signs the kext using our "Mac Developer ID Application (With Kext)" certificate

2. Assembles together a combination of utility apps, launch daemons, and launch agents to bundle with the kext. Some of these apps may be modified during this process and then resigned using our "Mac Developer ID Application (With Kext)" certificate.

3. Builds a pkg containing the generated kext and apps and signs it using our "Mac Developer ID Installer" certificate

4. Assembles the pkg into a dmg


Some notes about this:


- Our kext is codeless, and its only purpose is to stop the default macOS HID drivers from taking control of the hardware devices we support.

- Our installers currently support macOS 10.7 and later, and it's important we continue supporting these early versions of macOS.

- The installer currently chooses which kext to install automatically at runtime, but other threads in this forum suggest that the installer will fail notarization if an unsigned kext is bundled into it, which concerns me.


At this point, I'm not sure how to integrate notarization into this process, and I have many questions.


Firstly, this process is not supposed to take longer than a few minutes. So if at all possible it's quite important that I avoid having to automatically send out each release we make for notarization. Is that possible?


Do we need to individually notarize each version of the kext? Meaning for every unique Info.plist we create, bundle into a kext and then sign, does that version of the kext have to be notarized? Or does it even need to be notarized at all since it's codeless?


What about the apps, launch daemons, and launch agents that are bundled into the installer? Do they need to be notarized? If so, how should we handle apps that are modified during the release process?


Lastly, does the installer itself need to be notarized? If so, will bundling unsigned kexts cause notarization to fail?


I'm worried that this notarization process is going to remove our ability to release software the way we need to, since we support a lot of different customers each with different versions of our apps and driver. Notarization seems to be intended for apps and kexts that are built and distributed in a more typical fashion. I would greatly appreciate any assistance that anyone can provide!

Replies

First of all, this is a fairly complex question. Had you asked it last October when you were first informed of this requirement, it would have been much more manageable. When Apple says they are going to be changing something and you need to make updates to your processes, you need to do that before your software breaks. Speaking as a user, I really don't like it when vendors wait until their software breaks to make updates. That gives me a strong incentive to find new vendors.


10.14.5 will probably not be released for a few more weeks. Maybe 2 weeks? Maybe 3? That should still give you plenty of time to make these changes to your processes. Considering that 10.7 isn't going to be updated anytime soon, that OS version should not be your priority at this time. If nothing else, find the person who was responsible for reading that Apple e-mail six months ago and have them do the notarization manually until your scripts are updated.


Beyond that, no one on a web support forum is going to be able to give you much help. Few people write kernel extensions. No one else probably uses your build and distribution system. I have seen a few other people who are in a similar situation. My suggestion for them would probably work for you too. Change your installer to be a stub that downloads the appropriate extensions and installs them.

We did receive the notice concerning notarization back in October and considered it carefully, and the notice gave us no indication as to when specifically notarization would become necessary. It did not say macOS 10.14.5 would require kexts to be signed. In fact the only reason it suggested that notarization was important was because it would "give users even more confidence in your software" which is a fairly vapid reason, especially given our product and userbase. Given the onerous amount of work that will probably be involved for us to start notarizing our software, we decided not to pursue it at that time given that we had other more important priorities. I guessed (incorrectly of course) that the requirements for software might change in macOS 10.15, and waited for further announcements from Apple.


edit: I just made use of the Wayback machine to confirm the previous information Apple gave us: "In macOS 10.14, notarizing your software is optional, but recommended. In a future version of macOS, notarization will be required." The strongly implies that we had until, at the earliest, the release of macOS 10.15 before anything needed to be notarized since them saying "macOS 10.14" implies all bugfix releases of 10.14 as well. So not only did they not announce that 10.14.5 would require signed kernel extensions until very recently, but they implied we'd have more time than that. Quite distressingly too, they decided not to grandfather in already installed but non-notarized kexts, unlike when kext signing became required. This means that our software will breakon anyone's system that updates to 10.14.5! That is very bad for us plus any other developer that must utilize kernel extensions, and demonstrates some unpleasant thoughtlessness from Apple.


edit 2: I read from a third party source the kexts installed before March 11, 2019 will continue to work, so it's not quite as bad as I at first thought. However, why is a crucial piece of information like that not included in Apple's own documentation?


In any case you're probably right that I won't receive much help here. Unless it's enough for the time being to just notarize our signed kext and package that into the installer, I'll probably resort to using a technical support incident to receive further help directly from Apple.


I will go on record as saying though that I'm not happy about the security process and restrictions Apple has been introducing now and for the last several major versions of macOS. All of them strike me as a solution looking for a problem, and they're solutions that introduce a lot of problems on their own. (Don't get me started on how much trouble the restrictions on the Accessibility API and Apple Events has caused us, given that the security mechanisms in place for them are half broken.) The more macOS starts to get locked down and restricted similarly to iOS, the less useful it will be for ourselves and our customers.

If you haven't already done so, I strongly encourage you to install the 10.14.5 beta and test your software. That way, you will know for sure what will happen.


People on these forums will be happy to help, but there are limits to what we can do. As I mentioned before, no one knows anything about your processes or scripts, so that's off the table. Clearly, you are going to have issues with 10.14.5, so that needs to be your priority.


Does your current installer work at all on 10.14.5? Can you do a manual notarization of a recent build such that it does work? Normally I suggest using a VM for installation issues. That isn't going to work in your case. You will need a clean install of 10.14.5 for testing. You can use APFS snapshots to quickly go back to a clean slate.


Get something ready that will work for 10.14.5 customers. Develop a manual notarization process that can generate a new, valid installer. Once you have that in place, you will be able to update your automated scripts.

My question was mainly about trying to understand what Apple's requirements are for notarized software more than how I integrate it into my company's specific release system. Apple says that your software should be notarized, mention several different ways to do it (notarize an app, notarize a kext, notarize an installer, notarize all of them) but don't make it clear what the requirements for 10.14.5 are or what the future requirements will be.


For example, it's not necessary for us to sign all of our apps because when they're installed via an installer they aren't flagged with the com.apple.quarantine attribute and therefore don't run afoul of Gatekeeper. Will that continue to be the case when Apple adds in the notarization requirement for all apps? If so, then of course I have lots of work to do. But if not then there's no work for me to do. But it'd be nice if they were clear about that! One of the reasons why we don't attend to these issues until Apple is about to release an update with new requirements is that we have no way of knowing what the full requirements are until we have Apple's software to test with.


In any case, I've just spent the afternoon doing some tests with a clean 10.14.5 beta 2 installation. The previous version of the installer does not work, but if I manually build a new one with just the kext notarized it works again, so fortunately I have a relatively easy solution that will work for the time being.


The other good news is that notarizing the kext only takes a few minutes, and only has to be done once per unique kext. So most releases will not require sending the kext off to Apple to be notarized, and the ones that do won't take all that long.


Probably the biggest downside to this will be having to contact all of our clients and sending them updated installers. I'm sure we'll get a few support requests from users that will update to 10.14.5 and see the software break, but we should have an updated installer well before then.


Also I apologize if I was agitated earlier, but it's been a rather frustrating day!

I just went throught this process for the last three days of discovery. Our apps has many components. THe core is built in Xcode, but we have always used an extensive packaging script that does the final bundle assembly, MACH-O symbol corrections, meta file stripping, recursive code signing etc. This had to be every much overhauled driven by notarization errors. Since we use some JIT and plugins, I also had to work through entitlement definitions I never had to worry about before. Hardened code must be used, which can be forced via code signign option -o runtime.

The system is quite insistent that all code signing is consistent throughout the bundle. You cannot mix different IDs and options.

Every coded component within an installer package needs to be signed. This includes Helper apps and bundle apps in Frameworks etc. The requirement is complete and exhaustive.

We also have a KEXT in our application and already had to deal with earlier variants of the OS requiring different signing approaches. With this latest we decided to limit us to Sierra or higher, which simplified building the installer.

Notarizing the installer pkg will look into the components and check for complete compliance and consistency. You do that again with the DMG.


Makes sense to see, what they are looking for. But it is all rather painful, when you have more than a single app normally assembled via an automate process in a tool chain.

Other folks have commented in general, so I’m going to try to tackle your specific questions.

Firstly, this process is not supposed to take longer than a few minutes. So if at all possible it's quite important that I avoid having to automatically send out each release we make for notarization. Is that possible?

I’m not sure what you’re getting at here. If you don’t want a specific release notarised, don’t send it to the notarisation service. That’s totally obvious, so you must be anticipating some other problem. Can you explain that?

Do we need to individually notarize each version of the kext?

Yes.

Meaning for every unique Info.plist we create, bundle into a kext and then sign, does that version of the kext have to be notarized?

Yes. The

Info.plist
, along with everything else inside the bundle, is sealed by the code signature. Changing the
Info.plist
requires a new code signature, which requires a new notarisation.

Or does it even need to be notarized at all since it's codeless?

Whether the KEXT is codeless or not does not matter.

What about the apps, launch daemons, and launch agents that are bundled into the installer? Do they need to be notarized?

Apps, launchd daemons, and launchd agents are all code signed. Only apps can be notarised, but you can then notarise your installer after bundled all three of these things inside it.

If so, how should we handle apps that are modified during the release process?

This is essentially the same as your KEXT question. Modifying the app will require it to be re-signed, and that will require re-notarisation.

Lastly, does the installer itself need to be notarized?

Yes. There’s basically two types of installers out there:

  • Apple installer packages (

    .pkg
    ) — These can be notarised directly.
  • Custom installers — From the system perspective these are just apps, and can be notarised as such. You’ll want to first notarise the stuff that gets installed, and then notarise the installer (as per the advice in the Important box in Customizing the Notarization Workflow).

If so, will bundling unsigned kexts cause notarization to fail?

The latest version of Notarizing Your App Before Distribution mentions that there’s some affordance for notarising unsigned stuff, but I’m not yet up to speed on the details so I can’t comment. Honestly, I don’t see the need for this for new products, where your KEXT is always going to be signed.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

If I were skilled with ASCII art, I would include one of me shugging my shoulders with my hands up. I can't speak for Apple's future plans. When Apple announced this "future requirement" I submitted a support request to ask specifically when this requirement would be implemented. Apple responded after only a couple of days but said they did not have that information. To me, that was a Red Flag telling me to Do This Now.


I strongly suggest that you sign all of your executables, even if it is not "required". It is very easy to do and there is no legitimate reason not to do so. If you don't sign your software then you are putting yourself at risk of piracy and exploitation by malware. Piracy is widespread and verifying your own app signature is a good, initial barrier to at least make the pirates do some code-level editing. So far, most malware is pretty simplistic, but they are getting better. I have only seen them masquerade as ubiquitous software like Apple and Adobe. But when developers release unsigned software and fail to verify their own signatures, they become like malware typhoid marys. They become an outwardly legitimate host in which malware can hide.


At this time, once software makes it past Gatekeeper (hence the name), it is home free. But it is not logical to assume that this current behaviour will never change. The security industry is very critical of Apple's Gatekeeper strategy. So far, Gatekeeper has proven effective due to the relative unsophistication of Mac malware. But there is no area of Mac software growing as fast or as dynamically as the security industry - both the white hats and the black hats. Gatekeeper's days are numbered. Apple will be forced to start random signature checks at some point.

From the first post to the last, this info has helped me greatly. Thank you.

Few more questions please:


  • If I notarize a pkg, does the notary service handle notarizing all internal tools and frameworks?
  • When I staple the .pkg, does all internal tools and frameworks also get stapled?
  • If I notarize (upload via xcrun altool) a compressed .zip containing my kext and .app, do I staple the items inside the .zip or can I staple the original items prior to compressing so long as their precisly the same hash?


I'm simply wondering if using a .zip file for sending items to the notary is a better choice than sending the pkg installer. That way I can staple the kext and .app individually. I mention this because after hardening the .app and kext, (Developer ID Cert with Kext), notarizing, stapling, the app crashes on launch with an error, 'code signature invalid'.


Any insight will be greatly appreciated.

Thank you!

If I notarize a pkg, does the notary service handle notarizing all internal tools and frameworks?

Yes. You can confirm this by looking at the returned ticket. Check out this thread, where I posted a notarisation example. As you can see, the

ticketContents
property of the log tells you exactly what got notarised, including the path to each item and the cdhash of that item.

When I staple the

.pkg
, does all internal tools and frameworks also get stapled?

I don’t know, and I don’t have time to check right now, but it doesn’t matter because when the user installs the package the system learns about the notarisation from the ticket stapled to the package. So unless you plan to notarise the package, then extract stuff from it, then ship that stuff independently of the package, it makes no difference.

If I notarize (upload via

xcrun altool
) a compressed
.zip
containing my kext and
.app
, do I staple the items inside the
.zip
or can I staple the original items prior to compressing so long as their precisly the same hash?

I don’t understand this. You can’t staple to a

.zip
, so if you notarise a
.zip
and you plan to ship a
.zip
you would have to notarise, then unpack, then staple, then repack. But are you actually planning to ship a
.zip
? You’re earlier questions are about
.pkg
, which a) makes more sense in this context (someone is going to have to install that KEXT!), and b) does support direct stapling.

I'm simply wondering if using a

.zip
file for sending items to the notary is a better choice than sending the
.pkg
installer.

In your case, I don’t think it is.

I mention this because after hardening the

.app
and kext, (Developer ID Cert with Kext), notarizing, stapling, the app crashes on launch with an error, 'code signature invalid'.

Is this when you install it from the

.pkg
? Or after doing the whole unzip-staple-rezip thing?

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Referring to my original statement, "...crash on launch with an error, 'code signature invalid'."


eskimo said, "Is this when you install it from the

.pkg
? Or after doing the whole unzip-staple-rezip thing?"

This only happens when I install it from the .pkg. I've not yet tried zip.


A little more informaiton regarding the crash on launch, after I enable hardened runtime, sign kext and .app, launch the .app before creating the pkg, the app crashes. Natually installing the app via the pkg then launching also crashes.


I am signing the kext and .app with the same Deverloper ID Certificate (kext). That should work, correct?

This only happens when I install it from the .pkg. I've not yet tried zip.

Hmmm.

after I enable hardened runtime, sign kext and

.app
, launch the
.app
before creating the
.pkg
, the app crashes.

Hmmm.

I am signing the kext and

.app
with the same Deverloper ID Certificate (kext). That should work, correct?

Yes. A KEXT-enabled Developer ID certificate is also valid for other Developer ID purposes.

What does this crash look like? Normally, when an app crashes due to a code signing problem the crash report contains strong hints as to what went wrong. However, enabling the hardened runtime enables a bunch of extra security features, so it’s possible that this isn’t a code signing problem but an actual runtime problem. Regardless, the crash report is the place to start.

If you are having problems interpreting the crash report, feel free to post it here. Please:

  • Symbolicate it (at least for your symbols)

  • Post the entire thing, not just an excerpt

  • Put it in a code block (using the

    <>
    icon) to make it easier to read

DevForums may require that a moderator approve the post, but I can’t take care of that.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Another question please:


For an app distributed outside the app store via pkg, do I have to re-notarize and re-staple for each new release, no matter how small the change?

Any change to your app bundle would invalidate the signature and, certainly, the notarization.

For an app distributed outside the app store via pkg, do I have to re-notarize and re-staple for each new release, no matter how small the change?

Yes. A notarisation ticket effectively whitelists a set of code signatures, and even a small change will change the code signature.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

On this page, in the Stapler section near the bottom of the page, it states,


"While you can notarize a ZIP archive, you can’t staple to it directly. Instead, run

stapler
against each individual item that you originally added to the archive. Then create a new ZIP file containing the stapled items for distribution. Although tickets are created for standalone binaries, it’s not currently possible to staple tickets to them.
"


Does that also apply to a flat PKG installer? I was under the impression that simply stapling the notarized .PKG would also take care of the .app bundle located inside the Payload as well as other items packaged inside (kext for example).


Or, will I need to:

  1. unpack the .PKG
  2. unpack the Payload
  3. staple the .app bundle
  4. staple the kext
  5. re-pack the Payload
  6. re-pack the PKG?
  7. re-notarize the PKG (because the hash changed)
  8. then finally, staple the PKG

Please tell me this is not the case.


I notoarized and stapled our PKG. Tested it on a machine running 10.14.4, and shortly after a successful installation, we see the "System Extension Blocked" dialog, with a choice of "Open Security Preferences" or "OK". Is this the correct behavior, or is this a product of not notarizing and stapling the internals of the PKG? And by behavior, I mean is the user required to click, "Allow" on the Security section, or are they not even supposed to be challanged by installing a properly signed kext, notarized and stapled?


[EDIT]

This thread (the orginal thread you referred me in my first post), which I think (according to the accepte answer) means that I do not have to manually notarize and staple the internals of the pkg installer. However, other posts suggest as of only two days ago, that I may indeed need to unpack to manually notarize and staple pkg internals. This one too.


Please advise.

Thank you kindly