User Profile

Hi Quinn,Just a quick reply to let you know I've seen your post. I've been way too busy on other tasks this past week.Thanks for all the info. I will definately file an enhancement request when I have a moment!Craig

Hi Ken,Yes, I'm coming to the same conclusion. A mechanism doesn't exist to support such dynamic library loading in a running app. The best reading I found in the end was the man-page for dyld, especially the last section.It is an unfortunate consequence of the legacy state and mindset of OSes. Modern languages have evolved so much in comparison. Unfortunately, one can't live in the high-level language world exclusively, we need the low-level, the native code, for performance and real-time constraints.It would have been so nice to see an API introduced that allows the app control over the loader path, instead of all the 'hacks' that have been introduced to embed @rpath, @loader_path, etc.. into the binaries themselves. Separation of concerns - why should a library dictate where it's depency exists at runtime! It shouldn't even be its business.Bah - sorry for getting all philosophical :-)Thank you so much for taking the time to investigate every corner, trying to find a solution. Maybe someone from Apple will see this discussion and someday we'll have the add_dyld_path() API. Until then, ugly work-arounds will stay.Craig

Thanks for all the info Ken.Correct, the plugins could be in various locations. This architecture is used in many products - some simply ship with a folder of plugins, while others would use a plugin repository (database and local filestore) that contains the plugins and can manage multiple versions of each. A launch of an application may be simply a local application, or it may be a service launching an 'instance' and being told a 'plugin-set' to use from the repository. The repository exposes a web-service and the local service will query for the location of the plugins to use. In more advanced scenarios, the application will load (or be told to load) additional plugins later on, after the initial launch. All of this can happen on a single machine with a user based application, or up to a multi-server cloud scenario being co-ordinated over web-services. Hence the reason for all this architecture.The plugins themselves contain binaries (native libraries) at various locations internally, they may have different binaries for different architectures (32 vs 64 bit) or CPU optimizations (AVX2, AVX512, etc). The plugin has the knowledge about it's own libraries and also which plugins it depends on. Thus a lot of the resolution for library loading is best done at runtime by the framework interacting with the plugin meta-information. Hence the reason full control over loading and library dependency resolution is so desired.MacOS has always been a 'partially supported' platform - some smaller niche customers etc.. It has always had a hack-around solution involving a pre-launch script that inspects the plugins that will be loaded, sets up a DYLD_LIBRARY_PATH, and then launches. Far less than ideal and obviously breaks with some of the more powerful options that require run-time decisions.Re RPATH - Can the RPATH mechanism be used in this scenario? It felt to me like it needs to be built in to the libraries/executable ahead of time. There's no way to alter the effective path at runtime is there?Re - current working directory: That's funny, I saw that in the docs and the thought did cross my mind previously :-) But we also have dependencies on dependencies, and multiple dependences, so that wouldn't quite work as an evil hack.. :-(Re - flat-namespace. Interesting idea, but unfortunately many plugins include 3rd party libraries that are out of our control. If the 3rd party libs were isolated to the plugin that included them, a relative path there may work. But sometimes a plugin just brings in the 3rd party libs for other plugins to use.I'll try and wrap my head around the RPATH idea some more (ie. @rpath/libXXX.dylib) to see if there's a solution there. Thanks again.

You seem to have misinterpreted my post. I'm not complaining, I'm here trying to figure out a solution.I merely described Linux and Windows to illustrate how they behave, especially the SONAME on Linux as MacOS is similar, using dlopen and I was trying to find out if its difference in behaviour was expected, or if a similar mechanism could be utilized to achieve the goal (ie. "install_name"). I'm fully aware that all the OSes have issues in this area. I have the battle scars to prove it.Our architecture takes care of plugin versioning at a higher level, we don't need to jam it into a library name where it doesn't belong.The majority of our large customers are on Linux and Windows. I'm the one trying to keep the MacOS implementation of this architecture up-to-date and supported! I was coding for MacOS before it was MacOS (ie. NeXT), so it still has a place in my heart. Now, if you think the solution has been so glaringly pointed out to me, please elaborate.

Thanks for the response.So here is the bottom line issue:- the location of the plugins are not known ahead of time- the location of the application(s) is not relative to the plugins- multple flavours of the application could load these plugins, typically a Java executable.- we have over 500 plugins developed to date- our architecture manages versions, dependencies, code-signing, security, etc.- multiple versions of a plugin may exist and execute concurrently (in separate execution instances) on the same machine- only at runtime does an application know what plugins to load- the plugins specify their dependencies, and hence (library) load order is known- all native libraries are loaded with dlopen() using the full pathWhat I'm looking for in a solution:- when a library is loaded, the already-loaded dependencies should resolve- the dependencies were previously loaded by full-path dlopen() calls- we know the location of the dependencies at runtime, so we could formulate a 'path' if such a mechanism existed (like modifying DYLD_LIBRARY_PATH) if that actually affected the library loader- we can control "install_name" at build time if such a mechanism existed to make it usable (like SONAME works on Linux)So the question is:- Does MacOS have a way to resolve a library dependency (that has already been loaded) when the location of the dependency was not known before running the executable?

Thanks for the response.Unfortunately, last time we tried this approach it was evident that changing the DYLD_LIBRARY_PATH at runtime had no effect on the library loader itself. So even though you can see the env variable changed, the original path is what the loader uses.