Outgoing SSL connections fail on macOS 15, work fine on earlier versions

I will mention this first happened on a customer machine. We did extensive debugging over zoom and found the "lldb hang" issue there.

Then, I was able to reproduce in Parallels on my 14.7 machine. I ended up upgrading my machine to 15.1, and the problem happened immediately.

At this point, my customers running 15.1 are all down. At least, the ones using HTTPS for outgoing network connections.

I want to also point out we do not use Xcode to build our product (which is 40 years old, at this point). We use the Command Line Tools for Xcode. The version used to compile the downloadable example above is 2395 (as printed by xcode-select --version).

DTS doesn’t support OpenSSL. From our perspective it’s just like any other open source library. For definitive advice on how to use it, you should look to the library’s support channel.

However, there’s a bunch of history here, so I want to reply here to make sure you get on the right path.

Way back in the day, Apple included OpenSSL in the macOS SDK. That changed in macOS 10.7 (circa 2011) where we removed the OpenSSL headers and libraries from the SDK. The canonical way to use OpenSSL in your product is to build it from source and embed a copy in your app, just like other open source libraries.

Note We still include the OpenSSL runtime libraries in macOS, but those are present purely to maintain compatibility with existing binaries.

If you decide to embed OpenSSL in your app then I recommend that you give it an rpath-relative install name. See the following for more details on that:

• Dynamic Library Identification DevForums post, and specifically the Dynamic Library Standard Setup for Apps post it links to

• Embedding nonstandard code structures in a bundle documentation, which has a specific example of how to set this up using install_name_tool

If you adopt this strategy and things still fail, please post a crash report of the failure. Again, we don’t support OpenSSL, but the crash report might be enough for me to offer you further hints.

See Posting a Crash Report for info on how to post a crash report.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@DTS Engineer

I'm not asking for OpenSSL support. I'm asking for help on why lldb hangs when I try to step into an SSL library function.

I also want to be clear: we can load the OpenSSL libraries fine. They work fine in macOS < 15. They do not work in macOS 15. lldb hangs in macOS 15. It does not hang in < 15.

This all feels like some sort of security measure in macOS 15.

I followed your "Posting a Crash Report" link and it seems to apply to everything except macOS. I don't see any "your app crashed" dialog, so I don't know if there is information in the system which would help diagnose the problem.

we can load the OpenSSL libraries fine. They work fine in macOS < 15. They do not work in macOS 15. lldb hangs in macOS 15. It does not hang in < 15.

You are talking about loading the dynamic library at runtime. That is always going to be more challenging than embedding a framework into a static app.

The only advice I can give is to stick with one, canonical approach. That means forget about Homebrew and focus on the official version that you've built yourself. Much of what you've posted is only relevant to the Homebrew build. I realize your project is very complicated already. You don't need additional complications.

@Etresoft @DTS Engineer

Dynamic loading is a basic feature of our language. It's been there 40 years and it worked fine until macOS 15. The OpenSSL libraries are no different, I assume, than any other libraries.

Let's take a step back. We use public, documented APIs for dynamic loading of shared libraries. Is Apple saying we should not use them?

It's not a trivial thing to use the "canonical approach", either.

I keep coming back to two things:

1. It worked before macOS 15.
2. lldb hangs when stepping into an OpenSSL library function.

In the spirit of trying to find the problem, I spent hours making a statically linked version. That is, our glue library aclissl3.dylib now statically includes all the OpenSSL functions it needs, from the locally installed libraries. This is all based on OpenSSL 3.0.

I replaced the aclissl3.dylib in the product. Loaded fine. Using it, however, gives me a SEGV:

$ ./alisp
International Allegro CL Free Express Edition
11.0 [64-bit macOS (Apple Silicon)]
CL-USER(1): (progn (require :ssl)(require :aserve))
; Fast loading
;    /Applications/AllegroCL64express.app/Contents/Resources/code/SSL.003
;;; Installing ssl patch, version 3.
...
;   Foreign loading sys:aclssl3.dylib.
CL-USER(2): (trace excl::rand_add)
(EXCL::RAND_ADD)
CL-USER(3): (net.aserve.client:do-http-request "https://franz.com")
 0[3]: (EXCL::RAND_ADD #<Vector @ #x307892d62> 40 40.0d0)
Error: Received signal number 11 (Segmentation fault: 11)
  [condition type: SYNCHRONOUS-OPERATING-SYSTEM-SIGNAL]

Restart actions (select using :continue):
 0: Return to Top Level (an "abort" restart).
 1: Abort entirely from this (lisp) process.
[1] CL-USER(4): 

It literally dies on the first call to the first function we call in that library. But, I can't debug it because lldb hangs:

@max[git:master]$ lldb alisp
lldb alisp
(lldb) target create "alisp"
Current executable set to '/Applications/AllegroCL64express.app/Contents/Resources/alisp' (arm64).
(lldb) br s -n RAND_add
br s -n RAND_add
Breakpoint 1: where = libcrypto.46.dylib`RAND_add, address = 0x000000025a7241bc
(lldb) br s -n catch_exception_raise_state
br s -n catch_exception_raise_state
Breakpoint 2: where = alisp`catch_exception_raise_state, address = 0x0000000100003380
(lldb) run
run
Process 49169 launched: '/Applications/AllegroCL64express.app/Contents/Resources/alisp' (arm64)
1 location added to breakpoint 2
International Allegro CL Free Express Edition
11.0 [64-bit macOS (Apple Silicon)]
Copyright (C) 1985-2023, Franz Inc., Lafayette, CA, USA.  All Rights Reserved.

This development copy of Allegro CL is licensed to:
   Allegro CL 11.0 Express user

;; Optimization settings: safety 1, space 1, speed 1, debug 2,
;; compilation-speed 1.
;; For a complete description of all compiler switches given the
;; current optimization settings evaluate (EXPLAIN-COMPILER-SETTINGS).
CL-USER(1): (progn (require :ssl)(require :aserve))
(progn (require :ssl)(require :aserve))
; Fast loading
;    /Applications/AllegroCL64express.app/Contents/Resources/code/SSL.003
;;; Installing ssl patch, version 3.
;   Fast loading
;      /Applications/AllegroCL64express.app/Contents/Resources/code/OSI.fasl
;     Fast loading
;        /Applications/AllegroCL64express.app/Contents/Resources/code/fileutil.001
;;; Installing fileutil patch, version 1.
;     Fast loading from bundle code/acldns.fasl.
;     Fast loading from bundle code/iodefs.fasl.
;       Fast loading from bundle code/iordefs.fasl.
;         Fast loading from bundle code/efmacs.fasl.
1 location added to breakpoint 1
;   Foreign loading sys:aclssl3.dylib.
; Fast loading
;    /Applications/AllegroCL64express.app/Contents/Resources/code/ASERVE.001
;;; Installing aserve patch, version 1.
;   Fast loading from bundle code/autozoom.fasl.
;   Fast loading from bundle zlib/inflate.fasl.
;   Fast loading from bundle zlib/deflate.fasl.
;     Foreign loading libz.1.dylib.
;   Fast loading from bundle code/streamp.fasl.
; Autoloading for class BROADCAST-STREAM:
;   Fast loading from bundle code/streamc.fasl.
;     Fast loading from bundle code/efft-utf-8s-base.fasl.
;     Fast loading from bundle code/efft-utf8-base.fasl.
;     Fast loading from bundle code/efft-void.fasl.
;     Fast loading from bundle code/efft-latin1-base.fasl.
T
CL-USER(2): (net.aserve.client:do-http-request "https://franz.com")
(net.aserve.client:do-http-request "https://franz.com")
Process 49169 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.2
    frame #0: 0x00000001025e9cc0 aclssl3.dylib`RAND_add
aclssl3.dylib`RAND_add:
->  0x1025e9cc0 <+0>:  stp    d9, d8, [sp, #-0x30]!
    0x1025e9cc4 <+4>:  stp    x20, x19, [sp, #0x10]
    0x1025e9cc8 <+8>:  stp    x29, x30, [sp, #0x20]
    0x1025e9ccc <+12>: add    x29, sp, #0x20
(lldb) si

This constant "you're doing it wrong" is really frustrating because we've been doing it this way since the first version of macOS. It worked before the upgrade. I'm asking Apple to help us and tell us why.

Dynamic loading is a basic feature of our language. It's been there 40 years and it worked fine until macOS 15. The OpenSSL libraries are no different, I assume, than any other libraries.

Just to clarify, I was referring to the old-school practice of setting something like DYLD_LIBRARY_PATH and loading something you find there that's hopefully compatible. You should be able to still do that, but it's going to put you more at risk of various low-level "gotchas" that may not have existed 40 years ago. And when it breaks, the pool of people who know anything about that gets smaller and smaller every year. I'm not Apple, so I can't speak to the level of support that Apple can offer in that situation.

I can tell you that in the more canonical case, you are at less risk of random, low-level breakages. If Sequoia breaks something in the canonical case, then it's on Apple to fix it, not you. And if you are loading libraries in a standard fashion, there is a far greater pool of people on forums, in OpenSSL, etc. that will be able to help.

This canonical case is still dynamic library loading. It's just that the framework is embedded inside the app. I understand that you aren't using Xcode. But you could use Xcode to build a simple demo app, with an embedded version of OpenSSL as a framework, and confirm that it works (and debugs) properly in that case. Then you can review the details of how Xcode builds that framework, links it to your app, constructs the app bundle, and adjust your build scripts to match.

Otherwise, you're just spinning your wheels trying to come up with yet another way to link OpenSSL. That's a moving target.

The project I'm working with isn't 40 years old. It's only 24 years old. But I can tell you that once I came up with a system to convert all of my dependencies into frameworks, and constructed the project correctly in Xcode, it made my life so much easier. I have all the power of Xcode, and I can effortlessly step into any library any time I want. It was a fair amount of work to get to this point. But I still see other developers using this project struggle with all the different build systems on many different platforms. That's an entire category of problems that Xcode has eliminated for me.

@Etresoft I'm not sure what language your app is written in, but ours is written in Common Lisp. Using Xcode isn't an option.

With my last post, I showed that breaking the dependency on the Homebrew OpenSSL library doesn't do anything to change the errors, which don't happen in macOS < 15. It doesn't do anything to change the lldb hang when single stepping into a function in the SSL library, now statically bound into my own shared library.

Making a standalone example with Xcode doesn't get me any closer to solving this problem for my customers, which are down at the moment. Downgrading to 14.7.1 isn't something they want to do, so they're waiting for me. It's just that I can't debug the problem. I've tried a lot of things over the last 10 days, and nothing has been effective. If the experiment today was successful, I would have embarked on the big task of making a static version, but that didn't work, so I'm back to square 1.

I opened a DTS ticket and was told to post here. I'm hoping @DTS Engineer will escalate this to someone that can tell me why lldb doesn't work for this library. I suspect it's because macOS knows it's SSL and it's preventing lldb from snooping. Only Apple knows that, though.

I'm not sure what language your app is written in, but ours is written in Common Lisp. Using Xcode isn't an option.

Xcode is just an IDE. It can support any language for which you have a compiler or interpreter.

I'm recommending this test because it's guaranteed to work. You can build a demo app from the template using OpenSSL and see if it works on Sequoia. If it does, then your theory about Apple having blocked it would be disproven.

Most likely, the problem is related to your patching system. I understand that this is a 40 year-old project that was working fine a couple of months ago. That's what I was talking about regarding "risk". Other people might use the term "technical debt". You can pay that debt in instalments over time, or you can wait until the bank calls in the loan.

@Etresoft

Most likely, the problem is related to your patching system. I understand that this is a 40 year-old project that was working fine a couple of months ago. That's what I was talking about regarding "risk". Other people might use the term "technical debt". You can pay that debt in instalments over time, or you can wait until the bank calls in the loan.

This is absolutely false. No patching has been done in the example I uploaded and included above.

I built a NEW app bundle with all patches and notarized it. The problem still happens.

This is absolutely false. No patching has been done in the example I uploaded and included above.

I built a NEW app bundle with all patches and notarized it. The problem still happens.

Is this the version that prints out ";;; Installing ssl patch, version 3."?

It's easy enough to build OpenSSL, create a demo program, and verify that it works. If it's not your patching system, then it's something else unusual that you're doing.

The system applies the same code signing and library validation checks regardless of whether you import the library or load the library dynamically. I agree with Etresoft that importing the library is the better option, but if you can’t change that then it’s not a showstopper.

As to why LLDB is hanging, I don’t have an easy answer to that. I suspect it’s some sort of code signing or library validation issue. I’m disinclined to chase that because:

• LLDB isn’t a great tool for debugging code signing and library validation issues. It has enough on its plate being a debugger.

• Once you work out what’s going wrong with the library loading, it’s likely that LLDB will just start working again.

Anyway, just to get us on a firm footing, I decide to run a test:

1. On macOS 14.7.1 using Xcode 16.1, I created a new macOS app project.

2. I downloaded the disk image from your first post and extracted the OpenSSL libraries.

3. I modified them to be rpath-relative, per the docs I referenced above. I’ve put the exact commands at the end of this post.

4. I added them to my project.

5. I added a custom build phase to copy the libraries to Contents/Frameworks, making sure to enable Code Sign on Copy.

6. I added @executable_path/../Frameworks to the Runpath Search Paths build setting. This is necessary to allow libssl.1.1.dylib to find libcrypto.1.1.dylib.

7. I wrote code to load and call the library.

- (IBAction)testAction:(id)sender {
    #pragma unused(sender)
    if (self.libSSL == NULL) {
        NSURL * frameworksURL = [[NSBundle mainBundle] privateFrameworksURL];
        NSURL * libSSLURL = [frameworksURL URLByAppendingPathComponent:@"libssl.1.1.dylib"];
        self.libSSL = dlopen(libSSLURL.path.UTF8String, RTLD_LAZY);
        if (self.libSSL == NULL) {
            NSLog(@"QQQ did not load library, error: %s", dlerror());
            __builtin_trap();
        }
    }
    typedef void * (*TLSMethodProc)(void);
    TLSMethodProc tlsMethod = dlsym(self.libSSL, "TLS_method");
    if (tlsMethod == NULL) {
        NSLog(@"QQQ did not load symbol, error: %s", dlerror());
        __builtin_trap();
    }
    void * result = tlsMethod();
    NSLog(@"QQQ success %p", result);
}

I tested this in two ways:

• From Xcode I chose Product > Run and tested it on macOS 14.7.1 in the debugger.

• I copied the app to macOS 15.1 and ran it from the Finder.

In both cases it worked as expected, logging QQQ success ….

So this overall technique works. Which raises the question of why it’s not working with your setup. And that’s hard to answer without knowing exactly what your code is doing when it loads and runs OpenSSL.

I’ll send you a copy of my test project via the DTS support case you opened.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

% install_name_tool -id '@rpath/libcrypto.1.1.dylib' 'libcrypto.1.1.dylib'
…
% install_name_tool -id '@rpath/libssl.1.1.dylib' 'libssl.1.1.dylib'
…
% install_name_tool -change '@executable_path/libcrypto.1.1.dylib' '@rpath/libcrypto.1.1.dylib' 'libssl.1.1.dylib'
…
% codesign -s - -f libcrypto.1.1.dylib
libcrypto.1.1.dylib: replacing existing signature
% codesign -s - -f libssl.1.1.dylib   
libssl.1.1.dylib: replacing existing signature

@DTS Engineer

"Once you work out what’s going wrong with the library loading, it’s likely that LLDB will just start working again."

The library loads fine, but that's not even important anymore. The test I uploaded does STATIC linking of OpenSSL. Even that doesn't single step into any SSL function.

I don't see how your example helps find the problem with my application. I've ALREADY made changes to the downloadable demo to bring it more in line with what was discussed here.

Originally is dynamically loaded OpenSSL libraries. Now the libraries are statically linked. This means you don't even need to set DYLD_LIBRARY_PATH to run the example and get the SEGV.

@DTS Engineer it's simple enough to run this in macOS 14.7.1 and 15.1:

https://franz.com/ftp/pri/layer/acl11.0express-macos-arm64.dmg

cd /Applications/AllegroCL64express.app/Contents/Resources
env ACL_OPENSSL_VERSION=30 DYLD_LIBRARY_PATH="$(brew --prefix openssl@3.0)/lib:$DYLD_LIBRARY_PATH" ./alisp
(progn (require :ssl)(require :aserve))
(net.aserve.client:do-http-request "https://franz.com")

This example does NOT modify the bundle. It does NO dynamic loading of OpenSSL. It does dynamically load aclissl3.dylib which is our library that statically binds to the OpenSSL libraries.

Please, try this on 14 and 15.

I’m confused. You wrote:

This means you don't even need to set DYLD_LIBRARY_PATH to run the example and get the SEGV.

but also:

it's simple enough to run this in macOS 14.7.1 and 15.1 … DYLD_LIBRARY_PATH …

Which is it?

FWIW, I tried doing this with DYLD_LIBRARY_PATH and that failed:

% sw_vers
ProductName:		macOS
ProductVersion:		15.1
BuildVersion:		24B83
%
% cd /Applications/AllegroCL64express.app/Contents/Resources 
% ./alisp
International Allegro CL Free Express Edition
…
Error: Loading sys:aclissl11.dylib failed with error:
       dlopen(/Applications/AllegroCL64express.app/Contents/Resources/aclissl11.dylib, 0x000A): Library not loaded: @rpath/libcrypto.1.1.dylib
…

And running the test with DYLD_LIBRARY_PATH would require me to install Homebrew on my test VM, which isn’t something I have time for today.

Do you have a statically linked version I can try?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"