Notarization: stapler error 65

Code Signing General
This thread has been locked by a moderator; it no longer accepts new replies.
evitcejbo OP
Created Apr ’19
Replies 17
Boosts 0
Views 17k
Participants 7

Hello !


I'm trying to automatize notarization of a complex product composed of a bunch of components in preparation of macOS 10.14.5


The product is composed of a set of daemons, an app, and a kext. Everything use hardened runtime.


We build everything via a big Xcode workspace and xcodebuild command. To notarize, I create a zip archive of the resulting xcarchive, then I upload this zip archive to Apple server with altool --notarize-app.


It eventually finishes with success, and I receive the e-mail from Apple saying that my Mac software is ready to be distributed.


To be safe if users are offline, I want to staple the notarization ticket to my binaries.


I skip the daemons, because it seems it's not possible, for now, to attach a ticket to single Mach-O binaries (Error 73)


I staple the ticket to the kext : it works fine (and it's loaded even if the 10.14.5 machine is offline).


But when I try to staple the ticket to the app, I have this error :

Processing: /***/MyApplication.app
CloudKit query for MyApplication.app (2/[sha256]) failed due to "record not found".
Could not find base64 encoded ticket in response for 2/[sha256]
The staple and validate action failed! Error 65.


Should be noted that MyApplication.app and MyApplication.app/Contents/MacOS/MyApplication are in the final Apple logs (as all other Mach-O files in the archive). And [sha256] is a real valid sha256 value (just redacted there).


Should be noted too that if I zip directly MyApplication.app and upload this zip on Apple server with altool, then everything works as expected (I'm able to staple the notarization ticket to the application). So it seems to fail only when I notarize xcarchive, and only for .app bundles. I would still want to not do that, as each upload and notarization take time, so if it can be done in a single big step, it would be better.


And should be noted, finally, that MyApplication.app is inside a .bundle, while the kext is not (myarchive.xcarchive/Products/Library/TheKext.kext — myarchive.xcarchive/Products/Library/MyProduct/MyProduct.bundle/Contents/MacOS/MyApplication.app)


Any idea of what is happening there ?


Can it be related to https://forums.developer.apple.com/thread/115657 ?


Thank you !

Answered by DTS Engineer in 405735022

Plus I'm doing exactly the same operations I've been doing successfully for several MONTHS.

It’s hard to be sure without looking at your case in more detail, but it’s likely that this was previously working because, since Sep 2019, the notary service has been treating errors as warnings. A few months later we announced that, in Feb 2020, it would resume treating errors as errors. If you notarised a product with an unsigned component a few months back and then tried to notarise exactly the same file again today, you’d see this problem.

I’m happy to help you investigate this. There’s two ways we can move forward here:

Finally, I’ve locked this thread and I want to explain why. The error that kicked off this thread, 

stapler
failing with error 65, has a very specific cause: 
stapler
has extracted the cdhash from the item you are trying to staple too, passed that to Apple’s server’s to look up its notarisation ticket, and has got a response saying that no such ticket exists. For example, to quote dev\@eptar.hu’s most recent post, the item they’re trying to staple has a cdhash of b4d5dbccccfdd15f5c4bd6c043a3684f7053f1fb but there’s no ticket that covers that cdhash.

As to why there’s no ticket for that cdhash, that’s a complex question. Assuming you actually submitted the item for notarisation, you can use the technique from my 23 Apr 2019 to determine exactly which cdhashes are included in your ticket. There’s two possibilities here:

  • The cdhash is actually in the ticket (A).

  • The cdhash isn’t in the ticket (B).

In case A, something wacky is happening. The only time I’ve seen this happen is where the developer was notarising the same product multiple times, and renaming the outermost app in between. This caused the notary service to replace the existing correct ticket with a new ticket that covered less than the original. If you see this problem — where the output from 

--notarization-info
shows that a ticket was issued by 
stapler
is unable to find it — you should definitely file a bug with the details. And if you want help with that, I recommend that you open a DTS tech support incident.

Case B is far more common. There have been a few cases where this was triggered by a bug in the notary service — for example, over on this thread it turned out that the notary service wasn’t handling universal binaries properly — but most of the time this is caused by a problem with the way the submitted product was formatted.

Regardless, my experience is that there’s no single cause of this error, so having everyone who encounters this error pile on to this thread just confuses matters, and that’s why I’ve locked it. If you see this problem, please start a new thread here in Distribution > Mac Apps and we can pick up things there.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
Boost
Replies  17
Boosts  0
Views  17k
Participants  7
Security Engineer OP
Apple
Apr ’19

The error means that the bundle that's being stapled doesn't have a notarization ticket for the given cdhash.


If you're looking at the developer_log.json file for your successful notarization (altool --notarization-info <UUID>), there's a section for ticketContents. This shows the cdhashes for everything that was successfully notarized and can be stapled. Confirm that the cdhash for MyApplication.app (codesign -dvvv MyApplication.app) matches the cdhash for MyApplication.app in the ticketContents. Any changes to the code signature (including exporting in the organizer) will create a new cdhash.


Although it is recommended that your export your app instead of submitting the raw xcarchive.


Information about automating the export process can be found in Customizing the Notarization Workflow. There's also an entire script that can be used in a single big step to export everything and notarize it as part of an Archive phase post-action in Customizing the Xcode Archive Process.


An Example:

/usr/bin/xcodebuild -exportArchive -archivePath "$ARCHIVE_PATH" -exportOptionsPlist "$SRCROOT/ExportOptions.plist" -exportPath "$EXPORT_PATH"


If this doesnt help, I can give more information for your specific issue with the Request UUID altool returned to you.

0
sstanley OP
Apr ’19

The document 'Customizing the Notarization Workflow' says this:


While you can notarize a ZIP archive, you can’t staple to it directly. Instead, run 

stapler
against each individual item that you originally added to the archive.


So I can successfully notarize a .zip file, but stapling fails every time.


CloudKit query for TestingApp.app (2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02) failed due to "record not found".

Could not find base64 encoded ticket in response for 2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02

The staple and validate action failed! Error 65.


It seems people are having success using .dmg instead of .zip.


Is the documentation wrong?

0
DTS Engineer OP
Apple
Apr ’19

It seems people are having success using 

.dmg
instead of 
.zip
.

Is the documentation wrong?

No, you can notarise a 

.zip
just fine. Here’s an example of me doing just that. First I created my 
.zip
: 
$ /usr/bin/ditto -c -k --keepParent TestXXX.app TestXXX.zip

Then I ran 

altool
to submit it: 
$ xcrun altool --notarize-app -u eskimo1 -p "@keychain:altool-eskimo1" --primary-bundle-id com.example.apple-samplecode.TestXXX --file TestXXX.zip
2019-04-23 15:42:17.640 altool[73935:15163224] No errors uploading 'TestXXX.zip'.
RequestUUID = fee0fef0-be86-4646-84a3-XXXXXXXXXXXX

When it was done I ran 

altool
to get the result: 
$ xcrun altool --notarization-info fee0fef0-be86-4646-84a3-XXXXXXXXXXXX -u eskimo1 -p "@keychain:altool-eskimo1"
2019-04-23 15:45:09.560 altool[74113:15168593] No errors getting notarization info.

   RequestUUID: fee0fef0-be86-4646-84a3-XXXXXXXXXXXX
          Date: 2019-04-23 14:42:18 +0000
        Status: success
    LogFileURL: https://osxapps-ssl.itunes.apple.com/itunes-assets/…
   Status Code: 0
Status Message: Package Approved

Note the 

LogFileURL
property. Fetching that reveals the following: 
$ curl https://osxapps-ssl.itunes.apple.com/itunes-assets/…
{
  "logFormatVersion": 1,
  "jobId": "fee0fef0-be86-4646-84a3-XXXXXXXXXXXX",
  "status": "Accepted",
  "statusSummary": "Ready for distribution",
  "statusCode": 0,
  "archiveFilename": "TestXXX.zip",
  "uploadDate": "2019-04-23T14:42:18Z",
  "sha256": "1e1661336138c9fbfb1da56daa4715c3883c47d54aed1925df70694af7a2d7b8",
  "ticketContents": [
    {
      "path": "TestXXX.zip/TestXXX.app",
      "digestAlgorithm": "SHA-256",
      "cdhash": "e113e957d4c1368e88a3a3548f2b742eee4b3d85",
      "arch": "x86_64"
    },
    {
      "path": "TestXXX.zip/TestXXX.app/Contents/MacOS/TestXXX",
      "digestAlgorithm": "SHA-256",
      "cdhash": "e113e957d4c1368e88a3a3548f2b742eee4b3d85",
      "arch": "x86_64"
    }
  ],
  "issues": null

Look at the 

ticketContents
> *** > 
cdhash
values. These should match the equivalent (SHA-256) value in the code signature: 
$ codesign -d -vvv TestXXX.app
…
CandidateCDHash sha1=8888aa4bbf524d047835cb27e44970fbcbd27abe
CandidateCDHash sha256=e113e957d4c1368e88a3a3548f2b742eee4b3d85
…
CDHash=e113e957d4c1368e88a3a3548f2b742eee4b3d85
…

Note In this context, cdhash means code directory hash, and it’s the part of the code signature that uniquely identifiers this code.

Earlier Setag wrote:

The error means that the bundle that's being stapled doesn't have a notarization ticket for the given cdhash.

When you staple a ticket to an item, the 

stapler
tool gets the cdhash from the item and attempts to find a matching ticket. If it can’t find such a ticket, you get this error 65. This typically means that the code you’re trying to staple the ticket on to doesn’t match the code that you notarised.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
0
sstanley OP
Apr ’19

> No, you can notarise a 

.zip
just fine.


Yes, notarising is not the problem -- stapling is the problem.


Here's my log:

{
  "logFormatVersion": 1,
  "jobId": "9624e80f-b08a-4ca6-84c2-954f6796a5ee",
  "status": "Accepted",
  "statusSummary": "Ready for distribution",
  "statusCode": 0,
  "archiveFilename": "TestingApp.zip",
  "uploadDate": "2019-04-23T12:05:52Z",
  "sha256": "6ce0361e6fd9318c89ea79558ec0b250d8711e994d2d11e7089ef229694ba488",
  "ticketContents": [
    {
      "path": "TestingApp.zip/TestingApp.app",
      "digestAlgorithm": "SHA-256",
      "cdhash": "84ce1594c511d31a0abf41544284e0fc42b86c70",
      "arch": "i386"
    },
    {
      "path": "TestingApp.zip/TestingApp.app",
      "digestAlgorithm": "SHA-256",
      "cdhash": "56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02",
      "arch": "x86_64"
    },
    {
      "path": "TestingApp.zip/TestingApp.app/Contents/MacOS/applet",
      "digestAlgorithm": "SHA-256",
      "cdhash": "84ce1594c511d31a0abf41544284e0fc42b86c70",
      "arch": "i386"
    },
    {
      "path": "TestingApp.zip/TestingApp.app/Contents/MacOS/applet",
      "digestAlgorithm": "SHA-256",
      "cdhash": "56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02",
      "arch": "x86_64"
    }
  ],
  "issues": null
}


And here's the signature stuff:


CandidateCDHash sha1=e81934dfb5eb04e9321a9dbac2d7502a38f1b90c
CandidateCDHash sha256=56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02
...
CDHash=56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02


It all matches. But when I try to staple to the app it fails. The docs say "run 

stapler
against each individual item that you originally added to the archive", so I try to staple to the signed app:


/usr/bin/xcrun stapler staple -v /Users/shane/Desktop/Testing\ notary/TestingApp.app


And the result is:


CloudKit query for TestingApp.app (2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02) failed due to "record not found".
Could not find base64 encoded ticket in response for 2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02
The staple and validate action failed! Error 65.


The JSON Response is:


JSON Response is: {
    records =     (
                {
            reason = "record not found";
            recordName = "2/2/56e0d4edb22fcebfdfcf2835a0c5d3b616cbdd02";
            serverErrorCode = "NOT_FOUND";
        }
    );
}


Would you mind trying to staple your test app, to see what response you get there?

0
DTS Engineer OP
Apple
Apr ’19

Would you mind trying to staple your test app, to see what response you get there?

I did a bunch of stapling yesterday and it all worked fine.

"path": "TestingApp.zip/TestingApp.app/Contents/MacOS/applet",

OK, this is makes it clear that you’re dealing with an AppleScript app (it should have been obvious based on your comments on the other thread, but I missed that). I tried running through exactly the same process with an AppleScript app and I see the same problem you do.

Given that this is specific to AppleScript apps, I’m going to respond over on that other thread.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
0
evitcejbo OP
Apr ’19

Hi Setag !


Thank you for your answer, and sorry for the late answer (day off…)


So, yes, when I say


Should be noted that MyApplication.app and MyApplication.app/Contents/MacOS/MyApplication are in the final Apple log (as all other Mach-O files in the archive). And [sha256] is a real valid sha256 value (just redacted there).


I mean they are in the result JSON, in the "ticketContents" section.


I just checked the cdhash itself (as suggested by eskimo) : it seems valid for my app (as for the kext), I have the same cdhash in the JSON result and codesign.


I hear your point about archives, but it better fit the whole product building for now, so if we can stay on this, it would be nice.


About


If this doesnt help, I can give more information for your specific issue with the Request UUID altool returned to you.


Is it possible to give it to you by PM / e-mail ? I would like to keep this log result somehow private. Perhaps it worth to open a bug ticket.

0
evitcejbo OP
Apr ’19

I created a ticket : 50294732

0
DTS Engineer OP
Apple
Apr ’19

I had a look at your bug (r. 50294732) and it seems like you’re submitting an entire 

.xcarchive
. That’s most unusual. Typically folks notarise their distribution package. For an app that can be a 
.zip
, a 
.dmg
, or a 
.pkg
. For a complex program like yours, which contains command-line tools, KEXTs, and so on, you’d normally bundle that up into a 
.pkg
and notarise that.

Clearly you don’t distribute an 

.xcarchive
to customers? How do they get your product?

Is it possible to give it to you by PM / e-mail ?

If you want one-on-one help with this, you should open a DTS tech support incident. The folks helper out here, be they Apple or non-Apple, are mostly volunteers.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
0
evitcejbo OP
Apr ’19

Thank for your answer (and thank you for your time) !


We indeed distribute the software via a .pkg, and so my first plan was indeed to just notarize this pkg, but I ended up in this situation :

- We produce the .xcarchive

- We create a pkg from files to be installed from this archive

- We sign this pkg, and send it to Apple server for notarization

- Apple notarize everything inside

- I staple the notarization ticket to the pkg

-> If I install the resulting package on a machine which is offline in 10.14.5, then the kext doesn't load, as the notarization ticket is not stapled to it and macOS is not able to contact Apple servers to verify. Stapling the notarization ticket to the pkg doesn't staple notarization tickets to the content of the pkg (I tested). And this is a problem for us.


If I try to modify the pkg to staple the notarization ticket to each element (by using pkgutil --expand etc.), then it modify my pkg, and invalidate the signature, so I need to resign, and notarize again so I can staple a notarization ticket to the pkg itself (it doesn't seem necessary for now to staple the notarization ticket to the pkg, but I guess it will at some time, so I would like to be a bit proactive).


So it was simpler to notarize the .xcarchive, staple the ticket to elements inside, create the pkg, sign it, notarize it (and notarize its content again, but it's an involuntary side effect in this scheme), and staple the ticket to the pkg itself. This way everything is 1/ notarized 2/ stapled.


That being said, even if I notarize only the pkg itself, I'm not able to staple notarization ticket to the application neither (I have updated the ticket). I get the exact same error (stapler error 65).


But yes, I will probably open a DTS. I have the feeling we are on a specific complex situation.

0
DTS Engineer OP
Apple
May ’19

I have the feeling we are on a specific complex situation.

Indeed.

But yes, I will probably open a DTS.

Thanks.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
0
Engineer OP
Apple
May ’19

DTS is discussing this with you separately, but I wanted to provide some info for people who stumble across this thread:


  • We recommend you notarize only your final software distribution. If you ship a dmg to your customers, then just notarize the dmg and staple a ticket to it. Same thing for a pkg. There's no need to notarize internal components separately in most cases. The main execption is if you have a custom installer format (not .pkg), you may need to notarize the internals first and then the outer installer exe after building it. You'll see this if you try to upload your custom installer and the developer log shows some files were missing from the ticket contents since they couldn't be unpacked.
  • You can notarize using a zip file, but it does mean you need to separately staple each of the things in the zip since there is no ticket generated for the zip file itself (it's not something that can be code-signed).


Separately, we have fixed bug 50294732 yesterday that caused this developer's submission to not have all tickets created.

0
evitcejbo OP
May ’19

Hi haikeeba !


Thank you for your message.


So I thought my problem was solved because I fixed a bogus CFBundleExecutable on my "master" bundle, but then I guess it's just because you corrected the problem.


---


"Same thing for a pkg. There's no need to notarize internal components separately in most cases."


We are a bit moving away from the original subject, but my first guess was "if I notarize my pkg and staple the notarization ticket to the pkg, then the installer will automatically validate / staple notarization ticket to inner elements, like my kext, when installing".


So it's what I tried to do in the first place : I notarized only the pkg, and then stapled the notarization ticket to the pkg. But then the kext was not loadable when installing on an offline machine (it was working fine when online).


So I decided to :

- Notarize the elements to be installed.

- Staple notarization tickets to them.

- Create a pkg with this stapled elements.

- Notarize the final pkg (this step is not really necessary in this scenario, but my idea was that it will be necessary, soon or later, to have notarized pkg files to be able to install them - some proactivity).


This scenario actually fixed the problem (kext could be loaded even when installed on an offline machine).


If it's supposed to work without stapling notarization tickets to inner elements of the pkg (only the pkg itself), then I guess there is a bug somewhere ? I see there is something about this in the 10.14.5b5 release note (related to ticket 50205533). I guess it's related to my scenario ? When I tested it, the resulting pkg was not downloaded (so no quarantine flag).


I will try to use the suggested workarounds (or wait for a fix on macOS, if it's planned to fix it).


---


I agree on the zip, it was just a facility, in our build process, to do the double notarization to fix the offline problem.

0
DTS Engineer OP
Apple
May ’19

I guess it's related to my scenario ?

That’s correct.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
0
evitcejbo OP
May ’19

And just to update this thread: I tested the suggested workaround, i.e. set this in my preinstall


if [[ `/usr/bin/sw_vers -productVersion` == 10.14.5 ]]; then
    /usr/sbin/spctl -a -vvv -t install "$PACKAGE_PATH"; fi


and it actually works (I can install my kext via an unquarantined pkg on an offline machine), so it's great.

0
dev@eptar.hu OP
Feb ’20

Dear All!
A still have this issue with the error message about "The staple and validate action failed! Error 65."

However i was working fine before, Couple of days ago its not working.


- MacOS % stapler staple -v <FileName>

Processing: /Users/work/<FileName>/root/builds/<FileName>/Release/<FileName>.apx/Contents/MacOS/<FileName>

Properties are {

NSURLIsDirectoryKey = 0;

NSURLIsPackageKey = 0;

NSURLIsSymbolicLinkKey = 0;

NSURLLocalizedTypeDescriptionKey = "Unix executable";

NSURLTypeIdentifierKey = "public.unix-executable";

"_NSURLIsApplicationKey" = 0;

}

Props are {

cdhash = {length = 20, bytes = 0xb4d5dbccccfdd15f5c4bd6c043a3684f7053f1fb};

digestAlgorithm = 2;

flags = 65536;

signingId = "hu.<FileName>.<FileName>";

teamId = 6T96**9J7Z;

}

JSON Data is {

records = (

{

recordName = "2/2/b4d5dbccccfdd15f5c4bd6c043a3684f7053f1fb";

}

);

}

Headers: {

"Content-Type" = "application/json";

}

Domain is api.apple-cloudkit.com

Response is <NSHTTPURLResponse: 0x7f9b195105b0> { URL: https://api.apple-cloudkit.com/database/1/com.apple.gk.ticket-delivery/production/public/records/lookup } { Status Code: 200, Headers {

"Apple-Originating-System" = (

UnknownOriginatingSystem

);

Connection = (

"keep-alive"

);

"Content-Encoding" = (

gzip

);

"Content-Type" = (

"application/json; charset=UTF-8"

);

Date = (

"Tue, 04 Feb 2020 12:42:59 GMT"

);

Server = (

"AppleHttpServer/216f8733b0a9"

);

"Strict-Transport-Security" = (

"max-age=31536000; includeSubDomains;"

);

"Transfer-Encoding" = (

Identity

);

Via = (

"xrail:st13p00ic-zteu25223601.me.com:8301:19C194:grp60",

"631194250daa17e24277dea86cf30319:3557634d1cd1133d35d0d04a3dd3a943:Berlin"

);

"X-Apple-CloudKit-Version" = (

"1.0"

);

"X-Apple-Request-UUID" = (

"1c9d9f61-836e-45d7-b82b-28984077f709"

);

"X-Responding-Instance" = (

"ckdatabasews:16302501:st42p63ic-ztfb18191001:8201:1925B100:3e698439c07205310d897cffdce886c291b89b1e"

);

"access-control-expose-headers" = (

"X-Apple-Request-UUID, X-Responding-Instance",

Via

);

"apple-seq" = (

0

);

"apple-tk" = (

false

);

} }

Size of data is 165

JSON Response is: {

records = (

{

reason = "record not found";

recordName = "2/2/b4d5dbccccfdd15f5c4bd6c043a3684f7053f1fb";

serverErrorCode = "NOT_FOUND";

}

);

}

CloudKit query for <FileName> (2/b4d5dbccccfdd15f5c4bd6c043a3684f7053f1fb) failed due to "record not found".

Could not find base64 encoded ticket in response for 2/b4d5dbccccfdd15f5c4bd6c043a3684f7053f1fb

The staple and validate action failed! Error 65.


Can you please help me?

0
Notarization: stapler error 65
First post date Last post date
 
 
Q