I just found out about the com.apple.security.cs.allow-dyld-environment-variables entitlement. This seems to be the answer to my needs, but I'm worried about notarization with this entitlement being used.
Post
Replies
Boosts
Views
Activity
Thanks for the confirmation, @eskimo. Appreciated.
Having said that, I have some questions. You wrote:
We have a language (Allegro Common Lisp) which has a module for making SSL connections.
Can you explain more about the relationship between this language and your product. Is your product the language itself? Or are you using the language to build your actual product?
The language is our product (google "Allegro Common Lisp" if you are curious). The language has a networking layer and one of the features is an SSL socket stream. Using OpenSSL (or LibreSSL) is how we provide SSL socket streams.
We dynamically load the OpenSSL libraries because we want to get the latest, secure libraries. On Linux, we use the system libraries in standard places and loading it is trivial. On macOS, it's been a real hardship on our users (and us). Hopefully the entitlements above will make things much easier going forward.
Right. My specific concern is setting DYLD_LIBRARY_PATH. Is that the approach you recommend to your customers? Or something else?
Yes, we tell them to use DYLD_LIBRARY_PATH or put the OpenSSL libraries into the application directory, if they aren't in an environment where security is a concern.
We have this problem, too. Monterey, fully up to date as of today. Mini M1. Merely copying binaries can trigger it, where the originals work and the copies do not.
There's a good answer here: https://apple.stackexchange.com/questions/258623/how-to-fix-killed-9-error-in-mac-os
The key is, when copying binaries, to remove the existing destination before the copy. Or, just move if you can.
This works for us.
I've been getting this on multiple machines the last few days. Seems kinda random. Am I the only one?
I do have ping access to timestamp.apple.com.
This is the command we are using:
codesign -vvv -s ... --ignore-resources --timestamp foo.dylib
For others reading this thread, it was IPv6 connections to timestamp.apple.com were unreliable. If you don't need IPv6, then you can set it to "link-local only", which will force codesign to use IPv4.
Thanks to @eskimo for the assist.
@Etresoft thanks for the reply.
How do applications patch themselves and still not run afoul of Gatekeeper, then? Are they just complete reinstalls?
Allegro Common Lisp is 35+ years old. We have a customers with a huge amount of history. We have been using our patching system for decades. Sounds like Apple is saying we must throw that away, because the patches do include binary executables (shared libraries), though they are code signed.
Moving everything that changes outside of Contents/Resources will mean either it's user specific, or we'll need to use /Library/Application Support/, which will require admin privs. Also, it's quite a lot of redesign for us to move everything out.
As for the "not an app" ... I mean, Apple notarized our app and didn't complain then. Seems like I should have gotten errors or warnings for that.
[quote='809525022, Etresoft, /thread/766491?answerId=809525022#809525022, /profile/Etresoft']
In theory, it's possible to modify parts of the application bundle and then supply an updated signature that matches the new bundle. But I'm not sure if anyone has ever managed to do that successfully. It's a high-risk, high-effort solution when storage and bandwidth are cheap.
[/quote]
Can you explain more about this? It seems we might have to go that route, otherwise, it's the redesign of a 40 year old application.
I will mention this first happened on a customer machine. We did extensive debugging over zoom and found the "lldb hang" issue there.
Then, I was able to reproduce in Parallels on my 14.7 machine. I ended up upgrading my machine to 15.1, and the problem happened immediately.
At this point, my customers running 15.1 are all down. At least, the ones using HTTPS for outgoing network connections.
I want to also point out we do not use Xcode to build our product (which is 40 years old, at this point). We use the Command Line Tools for Xcode. The version used to compile the downloadable example above is 2395 (as printed by xcode-select --version).
@DTS Engineer
I'm not asking for OpenSSL support. I'm asking for help on why lldb hangs when I try to step into an SSL library function.
I also want to be clear: we can load the OpenSSL libraries fine. They work fine in macOS < 15. They do not work in macOS 15. lldb hangs in macOS 15. It does not hang in < 15.
This all feels like some sort of security measure in macOS 15.
I followed your "Posting a Crash Report" link and it seems to apply to everything except macOS. I don't see any "your app crashed" dialog, so I don't know if there is information in the system which would help diagnose the problem.
@Etresoft @DTS Engineer
Dynamic loading is a basic feature of our language. It's been there 40 years and it worked fine until macOS 15. The OpenSSL libraries are no different, I assume, than any other libraries.
Let's take a step back. We use public, documented APIs for dynamic loading of shared libraries. Is Apple saying we should not use them?
It's not a trivial thing to use the "canonical approach", either.
I keep coming back to two things:
It worked before macOS 15.
lldb hangs when stepping into an OpenSSL library function.
In the spirit of trying to find the problem, I spent hours making a statically linked version. That is, our glue library aclissl3.dylib now statically includes all the OpenSSL functions it needs, from the locally installed libraries. This is all based on OpenSSL 3.0.
I replaced the aclissl3.dylib in the product. Loaded fine. Using it, however, gives me a SEGV:
$ ./alisp
International Allegro CL Free Express Edition
11.0 [64-bit macOS (Apple Silicon)]
CL-USER(1): (progn (require :ssl)(require :aserve))
; Fast loading
; /Applications/AllegroCL64express.app/Contents/Resources/code/SSL.003
;;; Installing ssl patch, version 3.
...
; Foreign loading sys:aclssl3.dylib.
CL-USER(2): (trace excl::rand_add)
(EXCL::RAND_ADD)
CL-USER(3): (net.aserve.client:do-http-request "https://franz.com")
0[3]: (EXCL::RAND_ADD #<Vector @ #x307892d62> 40 40.0d0)
Error: Received signal number 11 (Segmentation fault: 11)
[condition type: SYNCHRONOUS-OPERATING-SYSTEM-SIGNAL]
Restart actions (select using :continue):
0: Return to Top Level (an "abort" restart).
1: Abort entirely from this (lisp) process.
[1] CL-USER(4):
It literally dies on the first call to the first function we call in that library. But, I can't debug it because lldb hangs:
@max[git:master]$ lldb alisp
lldb alisp
(lldb) target create "alisp"
Current executable set to '/Applications/AllegroCL64express.app/Contents/Resources/alisp' (arm64).
(lldb) br s -n RAND_add
br s -n RAND_add
Breakpoint 1: where = libcrypto.46.dylib`RAND_add, address = 0x000000025a7241bc
(lldb) br s -n catch_exception_raise_state
br s -n catch_exception_raise_state
Breakpoint 2: where = alisp`catch_exception_raise_state, address = 0x0000000100003380
(lldb) run
run
Process 49169 launched: '/Applications/AllegroCL64express.app/Contents/Resources/alisp' (arm64)
1 location added to breakpoint 2
International Allegro CL Free Express Edition
11.0 [64-bit macOS (Apple Silicon)]
Copyright (C) 1985-2023, Franz Inc., Lafayette, CA, USA. All Rights Reserved.
This development copy of Allegro CL is licensed to:
Allegro CL 11.0 Express user
;; Optimization settings: safety 1, space 1, speed 1, debug 2,
;; compilation-speed 1.
;; For a complete description of all compiler switches given the
;; current optimization settings evaluate (EXPLAIN-COMPILER-SETTINGS).
CL-USER(1): (progn (require :ssl)(require :aserve))
(progn (require :ssl)(require :aserve))
; Fast loading
; /Applications/AllegroCL64express.app/Contents/Resources/code/SSL.003
;;; Installing ssl patch, version 3.
; Fast loading
; /Applications/AllegroCL64express.app/Contents/Resources/code/OSI.fasl
; Fast loading
; /Applications/AllegroCL64express.app/Contents/Resources/code/fileutil.001
;;; Installing fileutil patch, version 1.
; Fast loading from bundle code/acldns.fasl.
; Fast loading from bundle code/iodefs.fasl.
; Fast loading from bundle code/iordefs.fasl.
; Fast loading from bundle code/efmacs.fasl.
1 location added to breakpoint 1
; Foreign loading sys:aclssl3.dylib.
; Fast loading
; /Applications/AllegroCL64express.app/Contents/Resources/code/ASERVE.001
;;; Installing aserve patch, version 1.
; Fast loading from bundle code/autozoom.fasl.
; Fast loading from bundle zlib/inflate.fasl.
; Fast loading from bundle zlib/deflate.fasl.
; Foreign loading libz.1.dylib.
; Fast loading from bundle code/streamp.fasl.
; Autoloading for class BROADCAST-STREAM:
; Fast loading from bundle code/streamc.fasl.
; Fast loading from bundle code/efft-utf-8s-base.fasl.
; Fast loading from bundle code/efft-utf8-base.fasl.
; Fast loading from bundle code/efft-void.fasl.
; Fast loading from bundle code/efft-latin1-base.fasl.
T
CL-USER(2): (net.aserve.client:do-http-request "https://franz.com")
(net.aserve.client:do-http-request "https://franz.com")
Process 49169 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.2
frame #0: 0x00000001025e9cc0 aclssl3.dylib`RAND_add
aclssl3.dylib`RAND_add:
-> 0x1025e9cc0 <+0>: stp d9, d8, [sp, #-0x30]!
0x1025e9cc4 <+4>: stp x20, x19, [sp, #0x10]
0x1025e9cc8 <+8>: stp x29, x30, [sp, #0x20]
0x1025e9ccc <+12>: add x29, sp, #0x20
(lldb) si
This constant "you're doing it wrong" is really frustrating because we've been doing it this way since the first version of macOS. It worked before the upgrade. I'm asking Apple to help us and tell us why.
@Etresoft I'm not sure what language your app is written in, but ours is written in Common Lisp. Using Xcode isn't an option.
With my last post, I showed that breaking the dependency on the Homebrew OpenSSL library doesn't do anything to change the errors, which don't happen in macOS < 15. It doesn't do anything to change the lldb hang when single stepping into a function in the SSL library, now statically bound into my own shared library.
Making a standalone example with Xcode doesn't get me any closer to solving this problem for my customers, which are down at the moment. Downgrading to 14.7.1 isn't something they want to do, so they're waiting for me. It's just that I can't debug the problem. I've tried a lot of things over the last 10 days, and nothing has been effective. If the experiment today was successful, I would have embarked on the big task of making a static version, but that didn't work, so I'm back to square 1.
I opened a DTS ticket and was told to post here. I'm hoping @DTS Engineer will escalate this to someone that can tell me why lldb doesn't work for this library. I suspect it's because macOS knows it's SSL and it's preventing lldb from snooping. Only Apple knows that, though.
@Etresoft
Most likely, the problem is related to your patching system. I understand that this is a 40 year-old project that was working fine a couple of months ago. That's what I was talking about regarding "risk". Other people might use the term "technical debt". You can pay that debt in instalments over time, or you can wait until the bank calls in the loan.
This is absolutely false. No patching has been done in the example I uploaded and included above.
I built a NEW app bundle with all patches and notarized it. The problem still happens.