JIT, disable library validation, and Rosetta 2

Question

wojtekmach OP

Created Oct ’21

Replies 6

Boosts 0

Views 1.7k

Participants 2

Hello,

On a x86_64 host I'm building an executable that uses JIT (asmjit) and dynamically loads a library. When running on arm64, running that executable segfaults:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000058
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [2107]

I'm using the following entitlements:

<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.allow-dyld-environment-variables</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>

and that's how I sign it:

codesign \
   --sign "-" \
   --verbose=4 \
   --options=runtime \
   --entitlements=erts/etc/darwin/Entitlements.plist \
   release/*/erts-*/bin/*

I noticed that when JIT is disabled, everything works fine.

When JIT is enabled, but there's no disable-library-validation entitlement, there's no segfault but loading the library fails:

dlopen(/Users/wojtek/otp/lib/crypto-5.0.3/priv/lib/crypto.so, 2): no suitable image found.  Did find:
        /Users/wojtek/otp/lib/crypto-5.0.3/priv/lib/crypto.so: code signature in (/Users/wojtek/otp/lib/crypto-5.0.3/priv/lib/crypto.so) not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
        /Users/wojtek/otp/lib/crypto-5.0.3/priv/lib/crypto.so: stat() failed with errno=35

If I code sign the library:

codesign \
   --sign "-" \
   --verbose=4 \
   --options=runtime \
   release/*/lib/crypto-*/priv/lib/crypto.so

then the error becomes:

dlopen(/Users/wojtek/otp/lib/crypto-5.0.3/priv/lib/crypto.so, 2): no suitable image found.  Did find:
        /Users/wojtek/otp/lib/crypto-5.0.3/priv/lib/crypto.so: code signature in (/Users/wojtek/otp/lib/crypto-5.0.3/priv/lib/crypto.so) not valid for use in process using Library Validation: mapped file has no Team ID and is not a platform binary (signed with custom identity or adhoc?)
        /Users/wojtek/otp/lib/crypto-5.0.3/priv/lib/crypto.so: stat() failed with errno=35

In terms of reproducing this issue, unfortunately I wasn't able to create a minimal program that does it. You should see the error when following https://github.com/erlang/otp/pull/5036#issuecomment-933397076 though.

Any guidance would be very appreciated.

Answered by DTS Engineer in 690105022

The library validation errors you’re seeing are only relevant if you have the hardened runtime enabled. Just for investigative purposes, disable the hardened runtime (and remove those entitlements because they’re pointless if the hardened runtime is disabled) and see if you continue to experience the problem.

If so, please post a full crash report. See Posting a Crash Report for some advice on how to do that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Boost

Answer 1

DTS Engineer OP

Apple

Oct ’21

Accepted Answer

The library validation errors you’re seeing are only relevant if you have the hardened runtime enabled. Just for investigative purposes, disable the hardened runtime (and remove those entitlements because they’re pointless if the hardened runtime is disabled) and see if you continue to experience the problem.

If so, please post a full crash report. See Posting a Crash Report for some advice on how to do that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 2

wojtekmach OP

Oct ’21

Hi, thank you for your answer!

By removing hardened runtime, you meant to remove the --option runtime, right? I did that and also removed the --entitlement option. Here are the code signing invocation I have at the moment:

# ad-hoc identity
identity="-"

codesign \
   --sign $identity \
   --verbose=4 \
   release/*/erts-*/bin/*

I'm still getting the segfault:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000058
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [3924]

beam.smp_2021-10-05-122607_macbookair.crash

please find the attached full report below. Apologies if I didn't include all the relevant information.

0

Answer 3

wojtekmach OP

Oct ’21

I've accidentally marked the answer as 'accepted', not sure how to revert it. Sorry for the noise.

0

Answer 4

wojtekmach OP

Oct ’21

Oh, If you wanted me to have just the allow-jit entitlement but no --options runtime, that segfaults too:

codesign \
   --sign $identity \
   --verbose=4 \
   --entitlements=erts/etc/darwin/Entitlements.plist \
   release/*/erts-*/bin/*

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>com.apple.security.cs.allow-jit</key>
    <true/>
  </dict>
</plist>

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000058
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [4991]

beam.smp_2021-10-05-125400_macbookair.crash

0

Answer 5

DTS Engineer OP

Apple

Oct ’21

I've accidentally marked the answer as 'accepted', not sure how to revert it.

Yeah, I don’t think that’s possible right now )-: but I’m happy to ignore it though (-:

By removing hardened runtime, you meant to remove the --option runtime, right?

Correct.

I did that and also removed the --entitlement option.

Good. These hardened runtime exception entitlement only make sense if the hardened runtime is enabled. Having them there with the hardened runtime disabled is not bad, but it’s pointless and potentially confusing.

please find the attached full report below.

Thanks!

That log provides evidence that Rosetta is having a hard time with your JITed code. Consider the backtrace of your crashing thread:

Thread 7 Crashed:: 3_scheduler
0   ???         0x000000014c1aa53c ???
1   beam.smp    0x000000010315cba2 erts_call_nif_early + 66

Note the ??? in the second column, indicating that this PC address doesn’t fall into one of the Mach-O images listed in the Binary Images section. There’s two possibilities with that:

Your code has jumped off into oblivion, which has then crashed.
Your code has called JITed code, which has then crashed.

Is erts_call_nif_early your entry point into the JITed code?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0

Answer 6

wojtekmach OP

Oct ’21

Yeah we have strong suspicion that our JIT code is indeed the culprit after all, I will update this thread accordingly once I know more. Thank you for your support.

0