stapling DMG fails with error 68: Certificate authority pinning mismatch.

We have a DMG for our Mac desktop app that has notarized OK, but on stapling we get the error below. The DMG and its contained app are signed (prior to Catalina this was sufficient.) The .app folder is directly constructed in our build process (not using XCode or similar); the .dmg is by DMGCanvas. The app only contains the UI; the libs and command-line tools are in a sibling folder, laid out much as they are on our other *nix builds. (When installed, everything is placed in a dedicated folder inside /Applications to keep it all in one place.)


Searching for the error "Certificate authority pinning mismatch" almost entirely links to cssmapple.h, which implies not many other people have run into this?


OS: 10.15.1

Xcode: 11.1

xcrun: 48



Notarization:

Request Identifier: 71c0468a-2a58-46ae-b699-22462e8593b0


Stapling:

Properties are {
    NSURLIsDirectoryKey = 0;
    NSURLIsPackageKey = 0;
    NSURLIsSymbolicLinkKey = 0;
    NSURLLocalizedTypeDescriptionKey = "Disk Image";
    NSURLTypeIdentifierKey = "com.apple.disk-image-udif";
    "_NSURLIsApplicationKey" = 0;
}
Codesign offset 0xcee4caf length: 9556
Stored Codesign length: 9556 number of blobs: 3
Total Length: 9556 Found blobs: 3
Props are {
    cdhash = {length = 20, bytes = 0xfb512617c5c078595f7a2ab6f74c73d7fa00a73c};
    digestAlgorithm = 2;
    flags = 0;
    secureTimestamp = "2019-09-12 15:10:53 +0000";
    signingId = "FICO Xpress 8.7.0 for Mac Installer";
    teamId = KL84GEX7ZW;
}
JSON Data is {
    records =     (
                {
            recordName = "2/2/fb512617c5c078595f7a2ab6f74c73d7fa00a73c";
        }
    );
}
Headers: {
    "Content-Type" = "application/json";
}
Domain is api.apple-cloudkit.com
Certificate trust evaluation did not return expected result. (5)  [leaf AnchorApple ChainLength IntermediateMarkerOid]
Certificate trust evaluation for api.apple-cloudkit.com did not return expected result. Certificate authority pinning mismatch.
Certificate trust evaluation did not return expected result. (5)  [leaf AnchorApple ChainLength IntermediateMarkerOid]
Certificate trust evaluation for api.apple-cloudkit.com did not return expected result. Certificate authority pinning mismatch.
Could not establish secure connection to api.apple-cloudkit.com
Response is (null)
error is Error Domain=NSURLErrorDomain Code=-999 "cancelled" UserInfo={NSErrorFailingURLStringKey=https://api.apple-cloudkit.com/database/1/com.apple.gk.ticket-delivery/production/public/records/lookup, NSLocalizedDescription=cancelled, NSErrorFailingURLKey=https://api.apple-cloudkit.com/database/1/com.apple.gk.ticket-delivery/production/public/records/lookup}
Size of data is 0
CloudKit's response is inconsistent with expections: (null)
The staple and validate action failed! Error 68.
Answered by DTS Engineer in 391844022

This doesn’t look like an issue with the image or the ticket, but rather with contacting CloudKit in the first place. Hence this line:

Could not establish secure connection to api.apple-cloudkit.com

I suspect that some sort of middlebox in your network environment is monkeying with CloudKit’s TLS connection [1]. I recommend that you try stapling in a different environment, purely as a diagnostic test. Stapling doesn’t require credentials, so you could just take the disk image home and try stapling there.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

[1] Search the ’net for “TLS inspection” or “TLS interception”.

Accepted Answer

This doesn’t look like an issue with the image or the ticket, but rather with contacting CloudKit in the first place. Hence this line:

Could not establish secure connection to api.apple-cloudkit.com

I suspect that some sort of middlebox in your network environment is monkeying with CloudKit’s TLS connection [1]. I recommend that you try stapling in a different environment, purely as a diagnostic test. Stapling doesn’t require credentials, so you could just take the disk image home and try stapling there.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

[1] Search the ’net for “TLS inspection” or “TLS interception”.

new uuid:

40580f0b-d316-4df6-aac7-c6e2ccb549d4


The stapling worked once I adjust our network settings. However:

  • when I try to run any of our executables from the command-line (which is valid mode of operation for the users) I get a pop-up saying "the developer cannot be verified", and "macOS cannot verify that this app is free from malware". Despite the executable and its .dylibs all being signed, and, it having beein installed from a notarized & stapled .dmg. Could this be because they are in the .dmg but outside the .app?
  • Launching the app results in a crash. If I run the executable in its Contents/MacOS folder, I get an error saying a nested framework is not signed before it aborts with error code 6.
    1. I tried going on a codesigning spree (guided by what was listed in the issues in the JSON from a previous notarization process), and with that version of the installer I got this message instead
  • [18116:775:1101/121703.384099:ERROR:component_loader.cc(162)] Failed to parse extension manifest.
    [1101/121703.506416:WARNING:system_snapshot_mac.cc(42)] sysctlbyname kern.nx: No such file or directory (2)
    [1101/121703.517707:WARNING:crash_report_exception_handler.cc(239)] UniversalExceptionRaise: (os/kern) failure (5)
    Killed: 9


    Edited to add: i've determined that if i just codesign the framework, I get the kern.nx crash listed above

    Launching the app results in a crash.

    If you run the app from the Finder, what does the resulting crash report look like? Please post a copy here.

    Share and Enjoy

    Quinn “The Eskimo!”
    Apple Developer Relations, Developer Technical Support, Core OS/Hardware

    let myEmail = "eskimo" + "1" + "@apple.com"

    To the version with the unsigned

    .framework
    …, we get this from running the executable from the console:

    Right. This is expected. Notarisation requires the hardened runtime, which requires library validation, which prevents your app from loading unsigned code (or code signed by a different team). There is a way to disable library validation but it doesn’t make sense in this case. With the framework embedded within your app, you should simply sign it.

    Which brings us to…

    In contrast, the version with the signed .framework … quits silently when run from Finder, but from console gives us

    Those log messages don’t look like anything from Apple. A ’net search suggests that the errors are coming from an embedded copy of Chromium. I can’t really help you with third-party code. You have a couple of choices:

    • You can trace into this third-party code to see what Apple API is failing. That might give you a clue as to what’s going wrong. If not, post info about the Apple API here and I’ll see what I can see.

    • You can raise this via the support channel for the third-party coed you’re using.

    Share and Enjoy

    Quinn “The Eskimo!”
    Apple Developer Relations, Developer Technical Support, Core OS/Hardware

    let myEmail = "eskimo" + "1" + "@apple.com"

    Thanks, that makes sense.


    Any idea why the standalone executables (such as bin/optimizer) still get the pop-up saying the developer can't be identified when (a) they are signed and (b) they were in the .dmg that got notarized, and according to the notarization log JSON they were listed as being in the contents?


    If the .framework is signed, is it still necessary to sign its component executables and dylibs?

    Any idea why the standalone executables (such as

    bin/optimizer
    ) still get the pop-up saying the developer can't be identified when (a) they are signed and (b) they were in the .dmg that got notarized, and according to the notarization log JSON they were listed as being in the contents?

    That alert means that the Gatekeeper check failed for this program. Gatekeeper can fail for many different reasons, so it’s hard to speculate as to what went wrong. In most cases there are some hints emitted to the system log, but they are generally pretty darned cryptic. Search for contemporaneous messages containing “syspolicyd” or “xprotect”.

    If the .framework is signed, is it still necessary to sign its component executables and dylibs?

    Every Mach-O image must be signed. And if you have auxiliary Mach-O image nested within a bundled component, you must sign from the inside out. For example, if you have a helper tool nested inside a framework, you must sign the helper tool first and then the framework.

    Share and Enjoy

    Quinn “The Eskimo!”
    Apple Developer Relations, Developer Technical Support, Core OS/Hardware

    let myEmail = "eskimo" + "1" + "@apple.com"

    Fixed the crash issue when I applied the following entitlements to everything (files, apps, frameworks,...)


    <key>com.apple.security.automation.apple-events</key>

    <true/>

    <key>com.apple.security.cs.allow-dyld-environment-variables</key>

    <true/>

    <key>com.apple.security.cs.allow-jit</key>

    <true/>

    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>

    <true/>

    <key>com.apple.security.cs.debugger</key>

    <true/>

    <key>com.apple.security.cs.disable-executable-page-protection</key>

    <true/>

    <key>com.apple.security.cs.disable-library-validation</key>

    <true/>

    @jperks - When you say you tweaked the network settings, can you give us some details on what changes were required? It would be of great help if you can share the details.

    Note: I'm encountering similar issue and trying to find a solution for the same. Appreciate any inputs on this, thanks!

    Could not establish secure connection to api.apple-cloudkit.com
    

    @ra_pa - We faced this same issue, and we finally managed to resolve it, I'll post what solved it for us mainly because this post is the first result when you try and Google the error, so hopefully it helps others in the future.

    The error we were receiving when we ran the staple command:

    Domain is api.apple-cloudkit.com
    Certificate trust evaluation did not return expected result. (5)  [leaf AnchorApple ChainLength IntermediateMarkerOid LeafMarkersProdAndQA]
    Certificate trust evaluation for api.apple-cloudkit.com did not return expected result. Missing required extension.
    Certificate trust evaluation did not return expected result. (5)  [leaf AnchorApple ChainLength IntermediateMarkerOid LeafMarkersProdAndQA]
    Certificate trust evaluation for api.apple-cloudkit.com did not return expected result. Missing required extension.
    Could not establish secure connection to api.apple-cloudkit.com
    Response is (null)
    error is Error Domain=NSURLErrorDomain Code=-999 "cancelled" UserInfo={NSErrorFailingURLStringKey=https:&#x2F;&#x2F;api.apple-cloudkit.com&#x2F;database&#x2F;1&#x2F;com.apple.gk.ticket-delivery&#x2F;production&#x2F;public&#x2F;records&#x2F;lookup, NSErrorFailingURLKey=https:&#x2F;&#x2F;api.apple-cloudkit.com&#x2F;database&#x2F;1&#x2F;com.apple.gk.ticket-delivery&#x2F;production&#x2F;public&#x2F;records&#x2F;lookup, _NSURLErrorRelatedURLSessionTaskErrorKey=(
        "LocalDataTask "
    ), _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask , NSLocalizedDescription=cancelled}
    Size of data is 0
    CloudKit&#039;s response is inconsistent with expections: (null)
    The staple and validate action failed! Error 68.
    

    While we could do a CURL POST to the https://api.apple-cloudkit.com/database/1/com.apple.gk.ticket-delivery/production/public/records/lookup URL and get the expected result, the staple process would fail with the certificate trust error. Within our corporate network, all SSL traffic is decrypted for inspection, so we came to the conclusion there must be a decryption issue due to certificate pinning.

    The solution was to exclude the api.apple-cloudkit.com domain from this decryption process. I'm assuming the stapler process is strict in the SSL certificate validation check.

    Anyway hope this helps anyone else facing the same problem!

    stapling DMG fails with error 68: Certificate authority pinning mismatch.
     
     
    Q