14 Replies
      Latest reply on May 13, 2019 8:46 AM by evitcejbo
      evitcejbo Level 1 Level 1 (0 points)

        Hello !

         

        I'm trying to automatize notarization of a complex product composed of a bunch of components in preparation of macOS 10.14.5

         

        The product is composed of a set of daemons, an app, and a kext. Everything use hardened runtime.

         

        We build everything via a big Xcode workspace and xcodebuild command. To notarize, I create a zip archive of the resulting xcarchive, then I upload this zip archive to Apple server with altool --notarize-app.

         

        It eventually finishes with success, and I receive the e-mail from Apple saying that my Mac software is ready to be distributed.

         

        To be safe if users are offline, I want to staple the notarization ticket to my binaries.

         

        I skip the daemons, because it seems it's not possible, for now, to attach a ticket to single Mach-O binaries (Error 73)

         

        I staple the ticket to the kext : it works fine (and it's loaded even if the 10.14.5 machine is offline).

         

        But when I try to staple the ticket to the app, I have this error :

        Processing: /xxx/MyApplication.app
        CloudKit query for MyApplication.app (2/[sha256]) failed due to "record not found".
        Could not find base64 encoded ticket in response for 2/[sha256]
        The staple and validate action failed! Error 65.

         

        Should be noted that MyApplication.app and MyApplication.app/Contents/MacOS/MyApplication are in the final Apple logs (as all other Mach-O files in the archive). And [sha256] is a real valid sha256 value (just redacted there).

         

        Should be noted too that if I zip directly MyApplication.app and upload this zip on Apple server with altool, then everything works as expected (I'm able to staple the notarization ticket to the application). So it seems to fail only when I notarize xcarchive, and only for .app bundles. I would still want to not do that, as each upload and notarization take time, so if it can be done in a single big step, it would be better.

         

        And should be noted, finally, that MyApplication.app is inside a .bundle, while the kext is not (myarchive.xcarchive/Products/Library/TheKext.kextmyarchive.xcarchive/Products/Library/MyProduct/MyProduct.bundle/Contents/MacOS/MyApplication.app)

         

        Any idea of what is happening there ?

         

        Can it be related to https://forums.developer.apple.com/thread/115657 ?

         

        Thank you !

        • Re: Notarization: stapler error 65
          Setag Apple Staff Apple Staff (0 points)

          The error means that the bundle that's being stapled doesn't have a notarization ticket for the given cdhash.

           

          If you're looking at the developer_log.json file for your successful notarization (altool --notarization-info <UUID>), there's a section for ticketContents. This shows the cdhashes for everything that was successfully notarized and can be stapled. Confirm that the cdhash for MyApplication.app (codesign -dvvv MyApplication.app) matches the cdhash for MyApplication.app in the ticketContents. Any changes to the code signature (including exporting in the organizer) will create a new cdhash.

           

          Although it is recommended that your export your app instead of submitting the raw xcarchive.

           

          Information about automating the export process can be found in Customizing the Notarization Workflow. There's also an entire script that can be used in a single big step to export everything and notarize it as part of an Archive phase post-action in Customizing the Xcode Archive Process.

           

          An Example:

          /usr/bin/xcodebuild -exportArchive -archivePath "$ARCHIVE_PATH" -exportOptionsPlist "$SRCROOT/ExportOptions.plist" -exportPath "$EXPORT_PATH"

           

          If this doesnt help, I can give more information for your specific issue with the Request UUID altool returned to you.

            • Re: Notarization: stapler error 65
              sstanley Level 1 Level 1 (20 points)

              The document 'Customizing the Notarization Workflow' says this:

               

              While you can notarize a ZIP archive, you can’t staple to it directly. Instead, run stapleragainst each individual item that you originally added to the archive.

               

              So I can successfully notarize a .zip file, but stapling fails every time.

               

              CloudKit query for TestingApp.app (2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02) failed due to "record not found".

              Could not find base64 encoded ticket in response for 2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02

              The staple and validate action failed! Error 65.

               

              It seems people are having success using .dmg instead of .zip.

               

              Is the documentation wrong?

                • Re: Notarization: stapler error 65
                  eskimo Apple Staff Apple Staff (11,475 points)

                  It seems people are having success using .dmg instead of .zip.

                  Is the documentation wrong?

                  No, you can notarise a .zip just fine.  Here’s an example of me doing just that.  First I created my .zip:

                  $ /usr/bin/ditto -c -k --keepParent TestXXX.app TestXXX.zip

                  Then I ran altool to submit it:

                  $ xcrun altool --notarize-app -u eskimo1 -p "@keychain:altool-eskimo1" --primary-bundle-id com.example.apple-samplecode.TestXXX --file TestXXX.zip
                  2019-04-23 15:42:17.640 altool[73935:15163224] No errors uploading 'TestXXX.zip'.
                  RequestUUID = fee0fef0-be86-4646-84a3-XXXXXXXXXXXX

                  When it was done I ran altool to get the result:

                  $ xcrun altool --notarization-info fee0fef0-be86-4646-84a3-XXXXXXXXXXXX -u eskimo1 -p "@keychain:altool-eskimo1"
                  2019-04-23 15:45:09.560 altool[74113:15168593] No errors getting notarization info.
                  
                     RequestUUID: fee0fef0-be86-4646-84a3-XXXXXXXXXXXX
                            Date: 2019-04-23 14:42:18 +0000
                          Status: success
                      LogFileURL: https://osxapps-ssl.itunes.apple.com/itunes-assets/…
                     Status Code: 0
                  Status Message: Package Approved

                  Note the LogFileURL property.  Fetching that reveals the following:

                  $ curl https://osxapps-ssl.itunes.apple.com/itunes-assets/…
                  {
                    "logFormatVersion": 1,
                    "jobId": "fee0fef0-be86-4646-84a3-XXXXXXXXXXXX",
                    "status": "Accepted",
                    "statusSummary": "Ready for distribution",
                    "statusCode": 0,
                    "archiveFilename": "TestXXX.zip",
                    "uploadDate": "2019-04-23T14:42:18Z",
                    "sha256": "1e1661336138c9fbfb1da56daa4715c3883c47d54aed1925df70694af7a2d7b8",
                    "ticketContents": [
                      {
                        "path": "TestXXX.zip/TestXXX.app",
                        "digestAlgorithm": "SHA-256",
                        "cdhash": "e113e957d4c1368e88a3a3548f2b742eee4b3d85",
                        "arch": "x86_64"
                      },
                      {
                        "path": "TestXXX.zip/TestXXX.app/Contents/MacOS/TestXXX",
                        "digestAlgorithm": "SHA-256",
                        "cdhash": "e113e957d4c1368e88a3a3548f2b742eee4b3d85",
                        "arch": "x86_64"
                      }
                    ],
                    "issues": null

                  Look at the ticketContents > xxx > cdhash values.  These should match the equivalent (SHA-256) value in the code signature:

                  $ codesign -d -vvv TestXXX.app
                  …
                  CandidateCDHash sha1=8888aa4bbf524d047835cb27e44970fbcbd27abe
                  CandidateCDHash sha256=e113e957d4c1368e88a3a3548f2b742eee4b3d85
                  …
                  CDHash=e113e957d4c1368e88a3a3548f2b742eee4b3d85
                  …

                  Note In this context, cdhash means code directory hash, and it’s the part of the code signature that uniquely identifiers this code.

                  Earlier Setag wrote:

                  The error means that the bundle that's being stapled doesn't have a notarization ticket for the given cdhash.

                  When you staple a ticket to an item, the stapler tool gets the cdhash from the item and attempts to find a matching ticket.  If it can’t find such a ticket, you get this error 65.  This typically means that the code you’re trying to staple the ticket on to doesn’t match the code that you notarised.

                  Share and Enjoy

                  Quinn “The Eskimo!”
                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                  let myEmail = "eskimo" + "1" + "@apple.com"

                    • Re: Notarization: stapler error 65
                      sstanley Level 1 Level 1 (20 points)

                      > No, you can notarise a .zip just fine.

                       

                      Yes, notarising is not the problem -- stapling is the problem.

                       

                      Here's my log:

                      {
                        "logFormatVersion": 1,
                        "jobId": "9624e80f-b08a-4ca6-84c2-954f6796a5ee",
                        "status": "Accepted",
                        "statusSummary": "Ready for distribution",
                        "statusCode": 0,
                        "archiveFilename": "TestingApp.zip",
                        "uploadDate": "2019-04-23T12:05:52Z",
                        "sha256": "6ce0361e6fd9318c89ea79558ec0b250d8711e994d2d11e7089ef229694ba488",
                        "ticketContents": [
                          {
                            "path": "TestingApp.zip/TestingApp.app",
                            "digestAlgorithm": "SHA-256",
                            "cdhash": "84ce1594c511d31a0abf41544284e0fc42b86c70",
                            "arch": "i386"
                          },
                          {
                            "path": "TestingApp.zip/TestingApp.app",
                            "digestAlgorithm": "SHA-256",
                            "cdhash": "56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02",
                            "arch": "x86_64"
                          },
                          {
                            "path": "TestingApp.zip/TestingApp.app/Contents/MacOS/applet",
                            "digestAlgorithm": "SHA-256",
                            "cdhash": "84ce1594c511d31a0abf41544284e0fc42b86c70",
                            "arch": "i386"
                          },
                          {
                            "path": "TestingApp.zip/TestingApp.app/Contents/MacOS/applet",
                            "digestAlgorithm": "SHA-256",
                            "cdhash": "56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02",
                            "arch": "x86_64"
                          }
                        ],
                        "issues": null
                      }

                       

                      And here's the signature stuff:

                       

                      CandidateCDHash sha1=e81934dfb5eb04e9321a9dbac2d7502a38f1b90c
                      CandidateCDHash sha256=56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02
                      ...
                      CDHash=56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02

                       

                      It all matches. But when I try to staple to the app it fails. The docs say "run stapleragainst each individual item that you originally added to the archive", so I try to staple to the signed app:

                       

                      /usr/bin/xcrun stapler staple -v /Users/shane/Desktop/Testing\ notary/TestingApp.app

                       

                      And the result is:

                       

                      CloudKit query for TestingApp.app (2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02) failed due to "record not found".
                      Could not find base64 encoded ticket in response for 2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02
                      The staple and validate action failed! Error 65.

                       

                      The JSON Response is:

                       

                      JSON Response is: {
                          records =     (
                                      {
                                  reason = "record not found";
                                  recordName = "2/2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02";
                                  serverErrorCode = "NOT_FOUND";
                              }
                          );
                      }

                       

                      Would you mind trying to staple your test app, to see what response you get there?

                        • Re: Notarization: stapler error 65
                          eskimo Apple Staff Apple Staff (11,475 points)

                          Would you mind trying to staple your test app, to see what response you get there?

                          I did a bunch of stapling yesterday and it all worked fine.

                          "path": "TestingApp.zip/TestingApp.app/Contents/MacOS/applet",

                          OK, this is makes it clear that you’re dealing with an AppleScript app (it should have been obvious based on your comments on the other thread, but I missed that).  I tried running through exactly the same process with an AppleScript app and I see the same problem you do.

                          Given that this is specific to AppleScript apps, I’m going to respond over on that other thread.

                          Share and Enjoy

                          Quinn “The Eskimo!”
                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                          let myEmail = "eskimo" + "1" + "@apple.com"

                    • Re: Notarization: stapler error 65
                      evitcejbo Level 1 Level 1 (0 points)

                      Hi Setag !

                       

                      Thank you for your answer, and sorry for the late answer (day off…)

                       

                      So, yes, when I say

                       

                      Should be noted that MyApplication.app and MyApplication.app/Contents/MacOS/MyApplication are in the final Apple log (as all other Mach-O files in the archive). And [sha256] is a real valid sha256 value (just redacted there).

                       

                      I mean they are in the result JSON, in the "ticketContents" section.

                       

                      I just checked the cdhash itself (as suggested by eskimo) : it seems valid for my app (as for the kext), I have the same cdhash in the JSON result and codesign.

                       

                      I hear your point about archives, but it better fit the whole product building for now, so if we can stay on this, it would be nice.

                       

                      About

                       

                      If this doesnt help, I can give more information for your specific issue with the Request UUID altool returned to you.

                       

                      Is it possible to give it to you by PM / e-mail ? I would like to keep this log result somehow private. Perhaps it worth to open a bug ticket.

                        • Re: Notarization: stapler error 65
                          evitcejbo Level 1 Level 1 (0 points)

                          I created a ticket : 50294732

                            • Re: Notarization: stapler error 65
                              eskimo Apple Staff Apple Staff (11,475 points)

                              I had a look at your bug (r. 50294732) and it seems like you’re submitting an entire .xcarchive.  That’s most unusual.  Typically folks notarise their distribution package.  For an app that can be a .zip, a .dmg, or a .pkg.  For a complex program like yours, which contains command-line tools, KEXTs, and so on, you’d normally bundle that up into a .pkg and notarise that.

                              Clearly you don’t distribute an .xcarchive to customers?  How do they get your product?

                              Is it possible to give it to you by PM / e-mail ?

                              If you want one-on-one help with this, you should open a DTS tech support incident.  The folks helper out here, be they Apple or non-Apple, are mostly volunteers.

                              Share and Enjoy

                              Quinn “The Eskimo!”
                              Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                              let myEmail = "eskimo" + "1" + "@apple.com"

                                • Re: Notarization: stapler error 65
                                  evitcejbo Level 1 Level 1 (0 points)

                                  Thank for your answer (and thank you for your time) !

                                   

                                  We indeed distribute the software via a .pkg, and so my first plan was indeed to just notarize this pkg, but I ended up in this situation :

                                  - We produce the .xcarchive

                                  - We create a pkg from files to be installed from this archive

                                  - We sign this pkg, and send it to Apple server for notarization

                                  - Apple notarize everything inside

                                  - I staple the notarization ticket to the pkg

                                  -> If I install the resulting package on a machine which is offline in 10.14.5, then the kext doesn't load, as the notarization ticket is not stapled to it and macOS is not able to contact Apple servers to verify. Stapling the notarization ticket to the pkg doesn't staple notarization tickets to the content of the pkg (I tested). And this is a problem for us.

                                   

                                  If I try to modify the pkg to staple the notarization ticket to each element (by using pkgutil --expand etc.), then it modify my pkg, and invalidate the signature, so I need to resign, and notarize again so I can staple a notarization ticket to the pkg itself (it doesn't seem necessary for now to staple the notarization ticket to the pkg, but I guess it will at some time, so I would like to be a bit proactive).

                                   

                                  So it was simpler to notarize the .xcarchive, staple the ticket to elements inside, create the pkg, sign it, notarize it (and notarize its content again, but it's an involuntary side effect in this scheme), and staple the ticket to the pkg itself. This way everything is 1/ notarized 2/ stapled.

                                   

                                  That being said, even if I notarize only the pkg itself, I'm not able to staple notarization ticket to the application neither (I have updated the ticket). I get the exact same error (stapler error 65).

                                   

                                  But yes, I will probably open a DTS. I have the feeling we are on a specific complex situation.

                                    • Re: Notarization: stapler error 65
                                      eskimo Apple Staff Apple Staff (11,475 points)

                                      I have the feeling we are on a specific complex situation.

                                      Indeed.

                                      But yes, I will probably open a DTS.

                                      Thanks.

                                      Share and Enjoy

                                      Quinn “The Eskimo!”
                                      Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                      let myEmail = "eskimo" + "1" + "@apple.com"

                                      • Re: Notarization: stapler error 65
                                        haikeeba Apple Staff Apple Staff (10 points)

                                        DTS is discussing this with you separately, but I wanted to provide some info for people who stumble across this thread:

                                         

                                        • We recommend you notarize only your final software distribution. If you ship a dmg to your customers, then just notarize the dmg and staple a ticket to it. Same thing for a pkg. There's no need to notarize internal components separately in most cases. The main execption is if you have a custom installer format (not .pkg), you may need to notarize the internals first and then the outer installer exe after building it. You'll see this if you try to upload your custom installer and the developer log shows some files were missing from the ticket contents since they couldn't be unpacked.
                                        • You can notarize using a zip file, but it does mean you need to separately staple each of the things in the zip since there is no ticket generated for the zip file itself (it's not something that can be code-signed).

                                         

                                        Separately, we have fixed bug 50294732 yesterday that caused this developer's submission to not have all tickets created.

                                          • Re: Notarization: stapler error 65
                                            evitcejbo Level 1 Level 1 (0 points)

                                            Hi haikeeba !

                                             

                                            Thank you for your message.

                                             

                                            So I thought my problem was solved because I fixed a bogus CFBundleExecutable on my "master" bundle, but then I guess it's just because you corrected the problem.

                                             

                                            ---

                                             

                                            "Same thing for a pkg. There's no need to notarize internal components separately in most cases."

                                             

                                            We are a bit moving away from the original subject, but my first guess was "if I notarize my pkg and staple the notarization ticket to the pkg, then the installer will automatically validate / staple notarization ticket to inner elements, like my kext, when installing".

                                             

                                            So it's what I tried to do in the first place : I notarized only the pkg, and then stapled the notarization ticket to the pkg. But then the kext was not loadable when installing on an offline machine (it was working fine when online).

                                             

                                            So I decided to :

                                            - Notarize the elements to be installed.

                                            - Staple notarization tickets to them.

                                            - Create a pkg with this stapled elements.

                                            - Notarize the final pkg (this step is not really necessary in this scenario, but my idea was that it will be necessary, soon or later, to have notarized pkg files to be able to install them - some proactivity).

                                             

                                            This scenario actually fixed the problem (kext could be loaded even when installed on an offline machine).

                                             

                                            If it's supposed to work without stapling notarization tickets to inner elements of the pkg (only the pkg itself), then I guess there is a bug somewhere ? I see there is something about this in the 10.14.5b5 release note (related to ticket 50205533). I guess it's related to my scenario ? When I tested it, the resulting pkg was not downloaded (so no quarantine flag).

                                             

                                            I will try to use the suggested workarounds (or wait for a fix on macOS, if it's planned to fix it).

                                             

                                            ---

                                             

                                            I agree on the zip, it was just a facility, in our build process, to do the double notarization to fix the offline problem.

                                              • Re: Notarization: stapler error 65
                                                eskimo Apple Staff Apple Staff (11,475 points)

                                                I guess it's related to my scenario ?

                                                That’s correct.

                                                Share and Enjoy

                                                Quinn “The Eskimo!”
                                                Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                                let myEmail = "eskimo" + "1" + "@apple.com"

                                                  • Re: Notarization: stapler error 65
                                                    evitcejbo Level 1 Level 1 (0 points)

                                                    And just to update this thread: I tested the suggested workaround, i.e. set this in my preinstall

                                                     

                                                    if [[ `/usr/bin/sw_vers -productVersion` == 10.14.5 ]]; then
                                                        /usr/sbin/spctl -a -vvv -t install "$PACKAGE_PATH"; fi

                                                     

                                                    and it actually works (I can install my kext via an unquarantined pkg on an offline machine), so it's great.