Application stays stuck at Verifying on 10.14

I’ve seen problems like this before and they were triggered by a bug in Gatekeeper’s malware scanning engine. Does the failure occur on 10.14 specifically? Or on a fully patch 10.14.6?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hi Quinn, this is a fully patched 10.14.6. Thanks

this is a fully patched 10.14.6.

OK. That has the fix for the specific issue I dealt with.

As to how you debug this, that’s tricky. Some general hints:

• Test on a ‘fresh’ machine. It’s not uncommon for machines to accumulate cruft that causes problems like this to happen on that machine only. To rule that out, I always test this stuff on a VM so that I can restore to a fresh snapshot between tests.

• Monitor the system log. Look for entries with keywords like gk (for Gatekeeper), xprotect (an internal component that’s part of Gatekeeper), and yara (likewise).

• Make sure to monitor Info and Debug level entries. I typically do this by:

  1. Setting up Console to show those entries (via the Action menu) and then filtering them out based on a text query that never matches (typically I use QQQQQQQQQQ).

  2. Run the test.

  3. Run sudo log collect to snapshot the log.

  I can then open the log in Console and run various queries without being continually bombarded by new log entries.

Still, this is kind of a long shot. If none of the above pans out I recommend that you open a DTS tech support incident so I can allocate more time to help you investigate this.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Have you figured out a solution or workaround? We are stuck in the same situation. The problem occurs when we are moving from intel-only to universal binaries, without making changes to the notarization process (while I'm not sure if we moved the Xcode Version from 12 to 13 at this time).

Any help is appreciated.

Greetings Brigitte

We are stuck in the same situation. The problem occurs when we are moving from intel-only to universal binaries

I’d like to clarify this. The issue raised by perrstep only occurs on macOS 10.14, but Apple silicon requires macOS 11. Are you saying that the new, universal version of your app works fine on Apple silicon and Intel 10.15, but hangs on Intel running 10.14?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Exactly. If I remove the quarantine flag or download the dmg via curl (which ends with the same result: no quarantine flag) on 10.14 the app starts fine. If the verification starts, the verification stucks after some time.

Greetings Brigitte

Exactly.

Interesting. As I mentioned above, this is rare but I’ve seen it before.

I filled a DTS. Follow-up: 788776198

Excellent! That should land in my queue later today.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

No, I haven't figured out the issue yet.

No, I haven't figured out the issue yet.

Shoulda opened that TSI (-:

Seriously though, I’ve made a note to update this thread if Brigitte’s TSI turns up anything that’s generally useful.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I’ve made a note to update this thread if Brigitte’s TSI turns up anything that’s generally useful.

I just received word from Brigitte that the workaround I suggested has resolved their issue, so I thought I’d post a summary here.

A while back I added this snippet to the hardened runtime docs:

Important

Because library validation is such an important security-hardening feature, Gatekeeper runs extra security checks on programs that have it disabled. If your program is blocked by Gatekeeper, check whether you’ve unnecessarily disabled library validation.

This is really important. Do not disable library validation as a matter of course. I see a lot of folks do this as a form of cargo cult programming. They’re having problems with the hardened runtime, notarisation, or Gatekeeper, and they try to resolve these by adding all the entitlements that disable the hardened runtime security checks. In many cases this is bad, but only bad because it reduces the security in their app. However, in the case of library validation it’s extra bad, because it actually makes it harder to pass Gatekeeper.

IMPORTANT As far as I’m aware, the only good reason to disable library validation is because your app needs to load plug-ins from other third-party developers. If you’re app doesn’t do that, make sure to leave library validation enabled.

The normal symptom of this problem is that Gatekeeper unnecessarily rejects your app. That’s bad, but it’s not the end of the world. You look in the docs, ask on DevForums, open a DTS TSI, randomly search the ’net, or whatever, you find this tidbit, you re-enable the hardened runtime and you’re done.

In this case, however, the symptoms were weird. That’s the result of four factors:

• When library validation is disabled, Gatekeeper does a bunch of extra checks on the app (these checks are done by an internal subsystem called XprotectService).

• The app being launched was weirdly structured.

• The app being launched was universal, that is, it contains a Intel and Apple silicon slice.

• In macOS 10.14.x the implementation of those checks has a bug that’s triggered by the app’s structure. This causes the checks to run pathologically slowly. Like really really slowly. Like they might not finish by the end of the universe slowly )-:

If you remove any of these factors you don’t hit this problem.

There’s one further wrinkle here: Once Brigitte removed the com.apple.security.cs.disable-library-validation entitlement from their app the problem didn’t go away! And that’s the result of a historical bug in Gatekeeper (r. 57278824). Prior to 10.15.4 Gatekeeper does not recognise the implied library validation that you get when you enabled the hardened runtime. So, even though you have library validation enabled via the hardened runtime, Gatekeeper still runs these extra checks )-:

Fortunately there’s a good workaround: Explicitly enable library validation via its code signing flag. So, where you might previously sign your code like this:

% codesign -s … -o runtime …other stuff…

you would instead do this:

% codesign -s … -o runtime,library …other stuff…

This is a no-op as far as the runtime is concerned; you are enabling the same library validation in two different ways. However, it works around this problem because Gatekeeper recognises this flag, realises that you have library validation enabled, and disables its extra checks.

So, to recap:

• If you’re having problems passing Gatekeeper, check that you haven’t disabled library validation unnecessarily.

• If that fixes the problem in general, you’re done.

• If you’re still having problems, do they only show up on 10.14.x [1]?

• If so, set the library code signing flag.

If that doesn’t fix the problem, please start a new thread here on DevForums and we can discuss your issue (although be prepared for me to bounce you into DTS because debugging this stuff is hard).

[1] Technically this also affects 10.15.x, x < 4, but IMO you shouldn’t be testing on such systems because you want to encourage your 10.15 users to update to the latest version of 10.15, that is, 10.15.7.

Finally, upthread I wrote:

I’ve seen problems like this before and they were triggered by a bug in Gatekeeper’s malware scanning engine.

I thought it worth noting that this was a red herring. This is an entirely separate problem than the one I investigated previously.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"