dlopen not using pre-loaded dependencies (MacOS)

I wrote this little C program that uses the setenv/getenv on MacOS

#include <stdio.h>
#include <stdlib.h>


int main(int argc, const char* argv[])
{
  const char* env_name = "DYLD_LIBRARY_PATH";
  const char* value = "/opt/local/lib:/opt/freeware/lib:/opt/gnu/lib:/opt/lib";

  printf("DYLD_LIBRARY_PATH = %s\n",getenv(env_name));
  setenv(env_name,value,1);
  printf("DYLD_LIBRARY_PATH = %s\n",getenv(env_name));

  return 0;
}

The first line of output shows (null) since I don't have a default DYLD_LIBRARY_PATH environment variable defined. Second line of output shows the environment variable set to the given value. Note that the setting modifies the environment for this instance execution; once execution is over, the modification is lost.

This works fine as a normal user on my Mac system. MacOS Catalina 15.4.1

You might want to look at this for dynamic library usage on MacOS.

https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/DynamicLibraryUsageGuidelines.html

It covers dlopen as part of the discussion

Try using otool -L instead

You will want to dig into usage of @rpath in dynamic libraries.

So, the way the dynamic loader works is that it has a search algorithm that, given a library path (which might be relative), it figures out several paths to look for that library, in turn. The library will have to be found at one of those paths. It does not just match library names against already-loaded libraries. Of course, once it finds a valid path to a library, it checks if that specific library is already loaded and, if so, just increments its load count rather than loading it again.

The search algorithm is described in the dlopen man page. The reference from libCommonMedia.dylib to libKayakNative.dylib appears to be just a leaf file name with no directory components in the path. For that case:

"When path does not contain a slash character (i.e. it is just a leaf name), dlopen() searches the following until it finds a compatible Mach-O file: $LD_LIBRARY_PATH, $DYLD_LIBRARY_PATH, current working directory, $DYLD_FALLBACK_LIBRARY_PATH."

The right way to solve this is probably to make the inter-library references use paths relative to @rpath, @executable_path, or @loader_path. (And, the RPATH can, in turn, reference @executable_path or @loader_path.) So, for example, the main application can be configured with an RPATH that includes @executable_path/../Frameworks. Then, libCommonMedia.dylib's reference to libKayakNative.dylib could be @rpath/libKayakNative.dylib or similar.

Thanks for the response.

Unfortunately, last time we tried this approach it was evident that changing the DYLD_LIBRARY_PATH at runtime had no effect on the library loader itself. So even though you can see the env variable changed, the original path is what the loader uses.

Thanks for the response.

So here is the bottom line issue:

- the location of the plugins are not known ahead of time

- the location of the application(s) is not relative to the plugins

- multple flavours of the application could load these plugins, typically a Java executable.

- we have over 500 plugins developed to date

- our architecture manages versions, dependencies, code-signing, security, etc.

- multiple versions of a plugin may exist and execute concurrently (in separate execution instances) on the same machine

- only at runtime does an application know what plugins to load

- the plugins specify their dependencies, and hence (library) load order is known

- all native libraries are loaded with dlopen() using the full path

What I'm looking for in a solution:

- when a library is loaded, the already-loaded dependencies should resolve

- the dependencies were previously loaded by full-path dlopen() calls

- we know the location of the dependencies at runtime, so we could formulate a 'path' if such a mechanism existed (like modifying DYLD_LIBRARY_PATH) if that actually affected the library loader

- we can control "install_name" at build time if such a mechanism existed to make it usable (like SONAME works on Linux)

So the question is:

- Does MacOS have a way to resolve a library dependency (that has already been loaded) when the location of the dependency was not known before running the executable?

1) Linux, Windows, and macOS are all different operating systems. If you decide to try some cross-platform app, it is your responsibility to figure it out. Complaining that one operating system is different than the others is going to be unproductive.

2) The fact that one dynamic library has already been loaded by object A has no bearing on object B. Object B may require version 1.0 of the library whereas object A may have loaded version 2.0 of the same library, under the same name.

3) In Linux, the SONAME of the dylib incorporates the version. In Windows, DLLs are just a mess and shouldn't even be considered for comparison. Where is the version encoded in "libKayakNative.dylib"?

As far as I can tell, the solution you are looking for has already been given to you. All we can do is give you the answers. We can't make you click the mouse.

The main issue is that a loaded library is not an "already-loaded dependency" unless the path from which it was loaded matches the path that dyld's search algorithm comes up with when trying to satisfy a dependency.

Is there literally no relationship between the locations of plugins and the libraries on which they depend? How do you determine where things are, at run-time?

The RPATH mechanism is quite flexible. It's a stack of paths that multiple components can contribute to. The executable can contribute one or more, any libraries it loads can contribute others for their dependencies, etc. That said, the only dynamism is the actual run-time location and the @executable_path and @loader_path pseudo-paths. I would think most cases could be handled by making every library install name and reference of the form @rpath/libName.dylib.

If that can't be made to work, you could leave the install names as leaf names and maybe play games with the current working directory at load time. That's one part of the search algorithm that can be modified after initial process load (I think). You'd have to be careful about thread safety.

Finally, it may be possible to get something like what you want by forcing the use of the flat symbol namespace and dynamic symbol lookup. You would use the -flat_namespace linker option when linking the application, the plugins, and the libraries that they depend on. Also, when linking things you would not link them against the libraries they would normally depend on. Rather, you would use the -undefined dynamic_lookup option. That would cause dyld to search through all loaded libraries to satisfy undefined symbols. This would only work if there were no symbol collisions among all of the relevant libraries. Also, it means you wouldn't catch link problems at link time, only run time.

You seem to have misinterpreted my post. I'm not complaining, I'm here trying to figure out a solution.

I merely described Linux and Windows to illustrate how they behave, especially the SONAME on Linux as MacOS is similar, using dlopen and I was trying to find out if its difference in behaviour was expected, or if a similar mechanism could be utilized to achieve the goal (ie. "install_name"). I'm fully aware that all the OSes have issues in this area. I have the battle scars to prove it.

Our architecture takes care of plugin versioning at a higher level, we don't need to jam it into a library name where it doesn't belong.

The majority of our large customers are on Linux and Windows. I'm the one trying to keep the MacOS implementation of this architecture up-to-date and supported! I was coding for MacOS before it was MacOS (ie. NeXT), so it still has a place in my heart. Now, if you think the solution has been so glaringly pointed out to me, please elaborate.

Ssorry, but there is nothing I can say that already hasn't been said in this thread already.

Thanks for all the info Ken.

Correct, the plugins could be in various locations. This architecture is used in many products - some simply ship with a folder of plugins, while others would use a plugin repository (database and local filestore) that contains the plugins and can manage multiple versions of each.

A launch of an application may be simply a local application, or it may be a service launching an 'instance' and being told a 'plugin-set' to use from the repository. The repository exposes a web-service and the local service will query for the location of the plugins to use. In more advanced scenarios, the application will load (or be told to load) additional plugins later on, after the initial launch. All of this can happen on a single machine with a user based application, or up to a multi-server cloud scenario being co-ordinated over web-services. Hence the reason for all this architecture.

The plugins themselves contain binaries (native libraries) at various locations internally, they may have different binaries for different architectures (32 vs 64 bit) or CPU optimizations (AVX2, AVX512, etc). The plugin has the knowledge about it's own libraries and also which plugins it depends on. Thus a lot of the resolution for library loading is best done at runtime by the framework interacting with the plugin meta-information. Hence the reason full control over loading and library dependency resolution is so desired.

MacOS has always been a 'partially supported' platform - some smaller niche customers etc.. It has always had a hack-around solution involving a pre-launch script that inspects the plugins that will be loaded, sets up a DYLD_LIBRARY_PATH, and then launches. Far less than ideal and obviously breaks with some of the more powerful options that require run-time decisions.

Re RPATH - Can the RPATH mechanism be used in this scenario? It felt to me like it needs to be built in to the libraries/executable ahead of time. There's no way to alter the effective path at runtime is there?

Re - current working directory: That's funny, I saw that in the docs and the thought did cross my mind previously :-) But we also have dependencies on dependencies, and multiple dependences, so that wouldn't quite work as an evil hack.. :-(

Re - flat-namespace. Interesting idea, but unfortunately many plugins include 3rd party libraries that are out of our control. If the 3rd party libs were isolated to the plugin that included them, a relative path there may work. But sometimes a plugin just brings in the 3rd party libs for other plugins to use.

I'll try and wrap my head around the RPATH idea some more (ie. @rpath/libXXX.dylib) to see if there's a solution there.

Thanks again.

Yeah, I don't know if RPATH will work for you, but it's worth investigating. As I say, for each binary that's loaded, its RPATH(s) can be added to the list, and a given binary's RPATH can be relative to @loader_path or @executable_path. Now, for dependencies, as I understand it, only the RPATHs of the images leading to a specific dependency are used when searching for that dependency. (That is, if Program depends on libA and libB, libA which depends on libC, and libC depends on libD, then the RPATHs added by Program, libA, and libC are used to search for libD, but those from libB are not because it's not in the chain that leads to libD.) I'm not sure how RPATHs influence dlopen() calls, if they do.

One other thing occurs to me: if the wrapper script doesn't work for all the cases you want, you could have the app process set DYLD_FALLBACK_LIBRARY_PATH and then re-exec itself with the same arguments.

As you tell by the rather kludgy suggestions I'm making, you're really swimming against the tide with these requirements. 😐

Hi Ken,

Yes, I'm coming to the same conclusion. A mechanism doesn't exist to support such dynamic library loading in a running app. The best reading I found in the end was the man-page for dyld, especially the last section.

It is an unfortunate consequence of the legacy state and mindset of OSes. Modern languages have evolved so much in comparison. Unfortunately, one can't live in the high-level language world exclusively, we need the low-level, the native code, for performance and real-time constraints.

It would have been so nice to see an API introduced that allows the app control over the loader path, instead of all the 'hacks' that have been introduced to embed @rpath, @loader_path, etc.. into the binaries themselves. Separation of concerns - why should a library dictate where it's depency exists at runtime! It shouldn't even be its business.

Bah - sorry for getting all philosophical :-)

Thank you so much for taking the time to investigate every corner, trying to find a solution.

Maybe someone from Apple will see this discussion and someday we'll have the add_dyld_path() API. Until then, ugly work-arounds will stay.

Craig

I’ve been looking at this off and on over the past few days. I want to start by confirming some factoids that have already been discussed here:

• Modifying

  DYLD_LIBRARY_PATH

  is a non-starter, alas. It is latched by

  dyld

  at launch time so, while you can change the environment variable, it doesn’t actually affect the behaviour of

  dyld

  .

• There’s definitely a disconnect between the way you want

  dyld

  to work and the way that

  dyld

  actually works. You are assuming that

  dyld

  first consults its table of loaded libraries and then, if the library is not there, goes off to search the file system. That’s not how it works. Rather, it searches the file system for the library and then, on finding it, consults its table of loaded libraries to see if it’s already loaded.

As to whether you can create a reasonable workaround, that very much depends on your specific requirements. The one avenue I think is worth exploring is links, both symbolic and hard.

dyld

1. Have all plug-in-to-library references and library-to-library reference be relative to

   @loader_path

   .

2. In a temporary directory, assemble links to the plug-in and all the libraries it references.

3. Point

   dlopen

   at the plug-in’s symlink in that temporary directory.

I prototyped this here and it seems to work. However, your product is way more complex than my prototype, so it’s hard to say if it’ll work in practice.

And yes, this is clearly very kludgy. Which brings me back to this:

Maybe someone from Apple will see this discussion and someday we'll have the

add_dyld_path

API.

I encourage you to file an enhancement request describing your requirements here. It’s clear that

dyld

Please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

[1] Hard links obviously, and symlinks explicitly.

Hi Quinn,

Just a quick reply to let you know I've seen your post. I've been way too busy on other tasks this past week.

Thanks for all the info. I will definately file an enhancement request when I have a moment!

Craig