7 Replies
      Latest reply on Apr 12, 2019 7:56 AM by john daniel
      guygizmo Level 1 Level 1 (0 points)

        My company has an automatic release process for our driver and associated apps. The driver is designed to work with an arbitrarily large number of different hardware configurations, and each time we make a release, our release process automatically does the following:

         

        1. Generates a new Info.plist file for the kext containing the PID / VID combinations that the driver supports and then signs the kext using our "Mac Developer ID Application (With Kext)" certificate

        2. Assembles together a combination of utility apps, launch daemons, and launch agents to bundle with the kext. Some of these apps may be modified during this process and then resigned using our "Mac Developer ID Application (With Kext)" certificate.

        3. Builds a pkg containing the generated kext and apps and signs it using our "Mac Developer ID Installer" certificate

        4. Assembles the pkg into a dmg

         

        Some notes about this:

         

        - Our kext is codeless, and its only purpose is to stop the default macOS HID drivers from taking control of the hardware devices we support.

        - Our installers currently support macOS 10.7 and later, and it's important we continue supporting these early versions of macOS.

        - The installer currently chooses which kext to install automatically at runtime, but other threads in this forum suggest that the installer will fail notarization if an unsigned kext is bundled into it, which concerns me.

         

        At this point, I'm not sure how to integrate notarization into this process, and I have many questions.

         

        Firstly, this process is not supposed to take longer than a few minutes. So if at all possible it's quite important that I avoid having to automatically send out each release we make for notarization. Is that possible?

         

        Do we need to individually notarize each version of the kext? Meaning for every unique Info.plist we create, bundle into a kext and then sign, does that version of the kext have to be notarized? Or does it even need to be notarized at all since it's codeless?

         

        What about the apps, launch daemons, and launch agents that are bundled into the installer? Do they need to be notarized? If so, how should we handle apps that are modified during the release process?

         

        Lastly, does the installer itself need to be notarized? If so, will bundling unsigned kexts cause notarization to fail?

         

        I'm worried that this notarization process is going to remove our ability to release software the way we need to, since we support a lot of different customers each with different versions of our apps and driver. Notarization seems to be intended for apps and kexts that are built and distributed in a more typical fashion. I would greatly appreciate any assistance that anyone can provide!

        • Re: How to handle notarizing apps and kexts that are generated programmatically?
          john daniel Level 3 Level 3 (350 points)

          First of all, this is a fairly complex question. Had you asked it last October when you were first informed of this requirement, it would have been much more manageable. When Apple says they are going to be changing something and you need to make updates to your processes, you need to do that before your software breaks. Speaking as a user, I really don't like it when vendors wait until their software breaks to make updates. That gives me a strong incentive to find new vendors.

           

          10.14.5 will probably not be released for a few more weeks. Maybe 2 weeks? Maybe 3? That should still give you plenty of time to make these changes to your processes. Considering that 10.7 isn't going to be updated anytime soon, that OS version should not be your priority at this time. If nothing else, find the person who was responsible for reading that Apple e-mail six months ago and have them do the notarization manually until your scripts are updated.

           

          Beyond that, no one on a web support forum is going to be able to give you much help. Few people write kernel extensions. No one else probably uses your build and distribution system. I have seen a few other people who are in a similar situation. My suggestion for them would probably work for you too. Change your installer to be a stub that downloads the appropriate extensions and installs them.

            • Re: How to handle notarizing apps and kexts that are generated programmatically?
              guygizmo Level 1 Level 1 (0 points)

              We did receive the notice concerning notarization back in October and considered it carefully, and the notice gave us no indication as to when specifically notarization would become necessary. It did not say macOS 10.14.5 would require kexts to be signed. In fact the only reason it suggested that notarization was important was because it would "give users even more confidence in your software" which is a fairly vapid reason, especially given our product and userbase. Given the onerous amount of work that will probably be involved for us to start notarizing our software, we decided not to pursue it at that time given that we had other more important priorities. I guessed (incorrectly of course) that the requirements for software might change in macOS 10.15, and waited for further announcements from Apple.

               

              edit: I just made use of the Wayback machine to confirm the previous information Apple gave us: "In macOS 10.14, notarizing your software is optional, but recommended. In a future version of macOS, notarization will be required." The strongly implies that we had until, at the earliest, the release of macOS 10.15 before anything needed to be notarized since them saying "macOS 10.14" implies all bugfix releases of 10.14 as well. So not only did they not announce that 10.14.5 would require signed kernel extensions until very recently, but they implied we'd have more time than that. Quite distressingly too, they decided not to grandfather in already installed but non-notarized kexts, unlike when kext signing became required. This means that our software will break on anyone's system that updates to 10.14.5! That is very bad for us plus any other developer that must utilize kernel extensions, and demonstrates some unpleasant thoughtlessness from Apple.

               

              edit 2: I read from a third party source the kexts installed before March 11, 2019 will continue to work, so it's not quite as bad as I at first thought. However, why is a crucial piece of information like that not included in Apple's own documentation?

               

              In any case you're probably right that I won't receive much help here. Unless it's enough for the time being to just notarize our signed kext and package that into the installer, I'll probably resort to using a technical support incident to receive further help directly from Apple.

               

              I will go on record as saying though that I'm not happy about the security process and restrictions Apple has been introducing now and for the last several major versions of macOS. All of them strike me as a solution looking for a problem, and they're solutions that introduce a lot of problems on their own. (Don't get me started on how much trouble the restrictions on the Accessibility API and Apple Events has caused us, given that the security mechanisms in place for them are half broken.) The more macOS starts to get locked down and restricted similarly to iOS, the less useful it will be for ourselves and our customers.

                • Re: How to handle notarizing apps and kexts that are generated programmatically?
                  john daniel Level 3 Level 3 (350 points)

                  If you haven't already done so, I strongly encourage you to install the 10.14.5 beta and test your software. That way, you will know for sure what will happen.

                   

                  People on these forums will be happy to help, but there are limits to what we can do. As I mentioned before, no one knows anything about your processes or scripts, so that's off the table. Clearly, you are going to have issues with 10.14.5, so that needs to be your priority.

                   

                  Does your current installer work at all on 10.14.5? Can you do a manual notarization of a recent build such that it does work? Normally I suggest using a VM for installation issues. That isn't going to work in your case. You will need a clean install of 10.14.5 for testing. You can use APFS snapshots to quickly go back to a clean slate.

                   

                  Get something ready that will work for 10.14.5 customers. Develop a manual notarization process that can generate a new, valid installer. Once you have that in place, you will be able to update your automated scripts.

                    • Re: How to handle notarizing apps and kexts that are generated programmatically?
                      guygizmo Level 1 Level 1 (0 points)

                      My question was mainly about trying to understand what Apple's requirements are for notarized software more than how I integrate it into my company's specific release system. Apple says that your software should be notarized, mention several different ways to do it (notarize an app, notarize a kext, notarize an installer, notarize all of them) but don't make it clear what the requirements for 10.14.5 are or what the future requirements will be.

                       

                      For example, it's not necessary for us to sign all of our apps because when they're installed via an installer they aren't flagged with the com.apple.quarantine attribute and therefore don't run afoul of Gatekeeper. Will that continue to be the case when Apple adds in the notarization requirement for all apps? If so, then of course I have lots of work to do. But if not then there's no work for me to do. But it'd be nice if they were clear about that! One of the reasons why we don't attend to these issues until Apple is about to release an update with new requirements is that we have no way of knowing what the full requirements are until we have Apple's software to test with.

                       

                      In any case, I've just spent the afternoon doing some tests with a clean 10.14.5 beta 2 installation. The previous version of the installer does not work, but if I manually build a new one with just the kext notarized it works again, so fortunately I have a relatively easy solution that will work for the time being.

                       

                      The other good news is that notarizing the kext only takes a few minutes, and only has to be done once per unique kext. So most releases will not require sending the kext off to Apple to be notarized, and the ones that do won't take all that long.

                       

                      Probably the biggest downside to this will be having to contact all of our clients and sending them updated installers. I'm sure we'll get a few support requests from users that will update to 10.14.5 and see the software break, but we should have an updated installer well before then.

                       

                      Also I apologize if I was agitated earlier, but it's been a rather frustrating day!

                        • Re: How to handle notarizing apps and kexts that are generated programmatically?
                          HaraldQI Level 1 Level 1 (0 points)

                          I just went throught this process for the last three days of discovery. Our apps has many components. THe core is built in Xcode, but we have always used an extensive packaging script that does the final bundle assembly, MACH-O symbol corrections, meta file stripping, recursive code signing etc. This had to be every much overhauled driven by notarization errors. Since we use some JIT and plugins, I also had to work through entitlement definitions I never had to worry about before. Hardened code must be used, which can be forced via code signign option -o runtime.

                          The system is quite insistent that all code signing is consistent throughout the bundle. You cannot mix different IDs and options.

                          Every coded component within an installer package needs to be signed. This includes Helper apps and bundle apps in Frameworks etc. The requirement is complete and exhaustive.

                          We also have a KEXT in our application and already had to deal with earlier variants of the OS requiring different signing approaches. With this latest we decided to limit us to Sierra or higher, which simplified building the installer.

                          Notarizing the installer pkg will look into the components and check for complete compliance and consistency. You do that again with the DMG.

                           

                          Makes sense to see, what they are looking for. But it is all rather painful, when you have more than a single app normally assembled via an automate process in a tool chain.

                          • Re: How to handle notarizing apps and kexts that are generated programmatically?
                            john daniel Level 3 Level 3 (350 points)

                            If I were skilled with ASCII art, I would include one of me shugging my shoulders with my hands up. I can't speak for Apple's future plans. When Apple announced this "future requirement" I submitted a support request to ask specifically when this requirement would be implemented. Apple responded after only a couple of days but said they did not have that information. To me, that was a Red Flag telling me to Do This Now.

                             

                            I strongly suggest that you sign all of your executables, even if it is not "required". It is very easy to do and there is no legitimate reason not to do so. If you don't sign your software then you are putting yourself at risk of piracy and exploitation by malware. Piracy is widespread and verifying your own app signature is a good, initial barrier to at least make the pirates do some code-level editing. So far, most malware is pretty simplistic, but they are getting better. I have only seen them masquerade as ubiquitous software like Apple and Adobe. But when developers release unsigned software and fail to verify their own signatures, they become like malware typhoid marys. They become an outwardly legitimate host in which malware can hide.

                             

                            At this time, once software makes it past Gatekeeper (hence the name), it is home free. But it is not logical to assume that this current behaviour will never change. The security industry is very critical of Apple's Gatekeeper strategy. So far, Gatekeeper has proven effective due to the relative unsophistication of Mac malware. But there is no area of Mac software growing as fast or as dynamically as the security industry - both the white hats and the black hats. Gatekeeper's days are numbered. Apple will be forced to start random signature checks at some point.

                    • Re: How to handle notarizing apps and kexts that are generated programmatically?
                      eskimo Apple Staff Apple Staff (11,035 points)

                      Other folks have commented in general, so I’m going to try to tackle your specific questions.


                      Firstly, this process is not supposed to take longer than a few minutes. So if at all possible it's quite important that I avoid having to automatically send out each release we make for notarization. Is that possible?

                      I’m not sure what you’re getting at here.  If you don’t want a specific release notarised, don’t send it to the notarisation service.  That’s totally obvious, so you must be anticipating some other problem.  Can you explain that?


                      Do we need to individually notarize each version of the kext?

                      Yes.

                      Meaning for every unique Info.plist we create, bundle into a kext and then sign, does that version of the kext have to be notarized?

                      Yes.  The Info.plist, along with everything else inside the bundle, is sealed by the code signature.  Changing the Info.plist requires a new code signature, which requires a new notarisation.

                      Or does it even need to be notarized at all since it's codeless?

                      Whether the KEXT is codeless or not does not matter.


                      What about the apps, launch daemons, and launch agents that are bundled into the installer? Do they need to be notarized?

                      Apps, launchd daemons, and launchd agents are all code signed.  Only apps can be notarised, but you can then notarise your installer after bundled all three of these things inside it.

                      If so, how should we handle apps that are modified during the release process?

                      This is essentially the same as your KEXT question.  Modifying the app will require it to be re-signed, and that will require re-notarisation.


                      Lastly, does the installer itself need to be notarized?

                      Yes.  There’s basically two types of installers out there:

                      • Apple installer packages (.pkg) — These can be notarised directly.

                      • Custom installers — From the system perspective these are just apps, and can be notarised as such.  You’ll want to first notarise the stuff that gets installed, and then notarise the installer (as per the advice in the Important box in Customizing the Notarization Workflow).

                      If so, will bundling unsigned kexts cause notarization to fail?

                      The latest version of Notarizing Your App Before Distribution mentions that there’s some affordance for notarising unsigned stuff, but I’m not yet up to speed on the details so I can’t comment.  Honestly, I don’t see the need for this for new products, where your KEXT is always going to be signed.

                      Share and Enjoy

                      Quinn “The Eskimo!”
                      Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                      let myEmail = "eskimo" + "1" + "@apple.com"