dlopen not using pre-loaded dependencies (MacOS)

I wrote this little C program that uses the setenv/getenv on MacOS

#include <stdio.h>
#include <stdlib.h>
 
 
int main(int argc, const char* argv[])
{
  const char* env_name = "DYLD_LIBRARY_PATH";
  const char* value = "/opt/local/lib:/opt/freeware/lib:/opt/gnu/lib:/opt/lib";
 
  printf("DYLD_LIBRARY_PATH = %s\n",getenv(env_name));
  setenv(env_name,value,1);
  printf("DYLD_LIBRARY_PATH = %s\n",getenv(env_name));
 
  return 0;
}

The first line of output shows (null) since I don't have a default DYLD_LIBRARY_PATH environment variable defined. Second line of output shows the environment variable set to the given value. Note that the setting modifies the environment for this instance execution; once execution is over, the modification is lost.

This works fine as a normal user on my Mac system. MacOS Catalina 15.4.1

You might want to look at this for dynamic library usage on MacOS.

https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/DynamicLibraryUsageGuidelines.html

It covers dlopen as part of the discussion

Try using otool -L instead

You will want to dig into usage of @rpath in dynamic libraries.

Thanks for the response.

Unfortunately, last time we tried this approach it was evident that changing the DYLD_LIBRARY_PATH at runtime had no effect on the library loader itself. So even though you can see the env variable changed, the original path is what the loader uses.

Thanks for the response.

So here is the bottom line issue:

- the location of the plugins are not known ahead of time

- the location of the application(s) is not relative to the plugins

- multple flavours of the application could load these plugins, typically a Java executable.

- we have over 500 plugins developed to date

- our architecture manages versions, dependencies, code-signing, security, etc.

- multiple versions of a plugin may exist and execute concurrently (in separate execution instances) on the same machine

- only at runtime does an application know what plugins to load

- the plugins specify their dependencies, and hence (library) load order is known

- all native libraries are loaded with dlopen() using the full path

What I'm looking for in a solution:

- when a library is loaded, the already-loaded dependencies should resolve

- the dependencies were previously loaded by full-path dlopen() calls

- we know the location of the dependencies at runtime, so we could formulate a 'path' if such a mechanism existed (like modifying DYLD_LIBRARY_PATH) if that actually affected the library loader

- we can control "install_name" at build time if such a mechanism existed to make it usable (like SONAME works on Linux)

So the question is:

- Does MacOS have a way to resolve a library dependency (that has already been loaded) when the location of the dependency was not known before running the executable?

1) Linux, Windows, and macOS are all different operating systems. If you decide to try some cross-platform app, it is your responsibility to figure it out. Complaining that one operating system is different than the others is going to be unproductive.

2) The fact that one dynamic library has already been loaded by object A has no bearing on object B. Object B may require version 1.0 of the library whereas object A may have loaded version 2.0 of the same library, under the same name.

3) In Linux, the SONAME of the dylib incorporates the version. In Windows, DLLs are just a mess and shouldn't even be considered for comparison. Where is the version encoded in "libKayakNative.dylib"?

As far as I can tell, the solution you are looking for has already been given to you. All we can do is give you the answers. We can't make you click the mouse.

You seem to have misinterpreted my post. I'm not complaining, I'm here trying to figure out a solution.

I merely described Linux and Windows to illustrate how they behave, especially the SONAME on Linux as MacOS is similar, using dlopen and I was trying to find out if its difference in behaviour was expected, or if a similar mechanism could be utilized to achieve the goal (ie. "install_name"). I'm fully aware that all the OSes have issues in this area. I have the battle scars to prove it.

Our architecture takes care of plugin versioning at a higher level, we don't need to jam it into a library name where it doesn't belong.

The majority of our large customers are on Linux and Windows. I'm the one trying to keep the MacOS implementation of this architecture up-to-date and supported! I was coding for MacOS before it was MacOS (ie. NeXT), so it still has a place in my heart. Now, if you think the solution has been so glaringly pointed out to me, please elaborate.

Ssorry, but there is nothing I can say that already hasn't been said in this thread already.

Thanks for all the info Ken.

Correct, the plugins could be in various locations. This architecture is used in many products - some simply ship with a folder of plugins, while others would use a plugin repository (database and local filestore) that contains the plugins and can manage multiple versions of each.

A launch of an application may be simply a local application, or it may be a service launching an 'instance' and being told a 'plugin-set' to use from the repository. The repository exposes a web-service and the local service will query for the location of the plugins to use. In more advanced scenarios, the application will load (or be told to load) additional plugins later on, after the initial launch. All of this can happen on a single machine with a user based application, or up to a multi-server cloud scenario being co-ordinated over web-services. Hence the reason for all this architecture.

The plugins themselves contain binaries (native libraries) at various locations internally, they may have different binaries for different architectures (32 vs 64 bit) or CPU optimizations (AVX2, AVX512, etc). The plugin has the knowledge about it's own libraries and also which plugins it depends on. Thus a lot of the resolution for library loading is best done at runtime by the framework interacting with the plugin meta-information. Hence the reason full control over loading and library dependency resolution is so desired.

MacOS has always been a 'partially supported' platform - some smaller niche customers etc.. It has always had a hack-around solution involving a pre-launch script that inspects the plugins that will be loaded, sets up a DYLD_LIBRARY_PATH, and then launches. Far less than ideal and obviously breaks with some of the more powerful options that require run-time decisions.

Re RPATH - Can the RPATH mechanism be used in this scenario? It felt to me like it needs to be built in to the libraries/executable ahead of time. There's no way to alter the effective path at runtime is there?

Re - current working directory: That's funny, I saw that in the docs and the thought did cross my mind previously :-) But we also have dependencies on dependencies, and multiple dependences, so that wouldn't quite work as an evil hack.. :-(

Re - flat-namespace. Interesting idea, but unfortunately many plugins include 3rd party libraries that are out of our control. If the 3rd party libs were isolated to the plugin that included them, a relative path there may work. But sometimes a plugin just brings in the 3rd party libs for other plugins to use.

I'll try and wrap my head around the RPATH idea some more (ie. @rpath/libXXX.dylib) to see if there's a solution there.

Thanks again.

Hi Ken,

Yes, I'm coming to the same conclusion. A mechanism doesn't exist to support such dynamic library loading in a running app. The best reading I found in the end was the man-page for dyld, especially the last section.

It is an unfortunate consequence of the legacy state and mindset of OSes. Modern languages have evolved so much in comparison. Unfortunately, one can't live in the high-level language world exclusively, we need the low-level, the native code, for performance and real-time constraints.

It would have been so nice to see an API introduced that allows the app control over the loader path, instead of all the 'hacks' that have been introduced to embed @rpath, @loader_path, etc.. into the binaries themselves. Separation of concerns - why should a library dictate where it's depency exists at runtime! It shouldn't even be its business.

Bah - sorry for getting all philosophical :-)

Thank you so much for taking the time to investigate every corner, trying to find a solution.

Maybe someone from Apple will see this discussion and someday we'll have the add_dyld_path() API. Until then, ugly work-arounds will stay.

Craig

I’ve been looking at this off and on over the past few days. I want to start by confirming some factoids that have already been discussed here:

• Modifying

  DYLD_LIBRARY_PATH

  is a non-starter, alas. It is latched by

  dyld

  at launch time so, while you can change the environment variable, it doesn’t actually affect the behaviour of

  dyld

  .

• There’s definitely a disconnect between the way you want

  dyld

  to work and the way that

  dyld

  actually works. You are assuming that

  dyld

  first consults its table of loaded libraries and then, if the library is not there, goes off to search the file system. That’s not how it works. Rather, it searches the file system for the library and then, on finding it, consults its table of loaded libraries to see if it’s already loaded.

As to whether you can create a reasonable workaround, that very much depends on your specific requirements. The one avenue I think is worth exploring is links, both symbolic and hard.

dyld

1. Have all plug-in-to-library references and library-to-library reference be relative to

   @loader_path

   .

2. In a temporary directory, assemble links to the plug-in and all the libraries it references.

3. Point

   dlopen

   at the plug-in’s symlink in that temporary directory.

I prototyped this here and it seems to work. However, your product is way more complex than my prototype, so it’s hard to say if it’ll work in practice.

And yes, this is clearly very kludgy. Which brings me back to this:

Maybe someone from Apple will see this discussion and someday we'll have the

add_dyld_path

API.

I encourage you to file an enhancement request describing your requirements here. It’s clear that

dyld

Please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

[1] Hard links obviously, and symlinks explicitly.

Hi Quinn,

Just a quick reply to let you know I've seen your post. I've been way too busy on other tasks this past week.

Thanks for all the info. I will definately file an enhancement request when I have a moment!

Craig