dlopen different behavior in xcode

I'm experiencing a strange problem with dlopen in my app. I use it to load a plugin .dylib, which works fine generally (I have a few plugins already). I'm developing a new plugin feature.dylib that has an external shared library dependency libexternal.so and here's where I'm having trouble. After a lot of work with install_name_tool and otool, I have the following situation: if I run my app from terminal open MyApp.app or ./MyApp.app/Contents/MacOS/MyApp or from the finder by double clicking MyApp.app, the dlopen of feature.dylib is successful. However, if I run MyApp from within Xcode, dlopen feature.dylib fails with image not found for libexternal.so. I've checked this with debug and release builds, and with and without debugging (and if I run lldb from the terminal as lldb MyApp.app/Contents/MacOS/MyApp it works fine too) and the behavior is the same: running from inside Xcode, the dlopen fails, but outside Xcode, it works.

Some other details: otool -L libexternal.so shows its id is libexternal.so and otool -L feature.dylib shows its id is feature.dylib and it has the dependency @loader_path/libexternal.so. If I change it to @rpath/libexternal.so and add my plugins folder path to MyApp.app's Runpath Search Paths, then it works in Xcode (but I don't want to have to enumerate all the places a plugin might be found here).

This is on osx 10.15.7 with Xcode 12.4. I'm not using hardened runtime. feature.dylib and MyApp.app are signed with the same certificate, while libexternal.so is not signed. I thought this might be a problem until I saw it works outside Xcode.

Is it possible Xcode is somehow changing the behavior of dlopen?
Up vote post of diverdi Down vote post of diverdi
1.3k views
Posted by
diverdi
Reply
Add a Comment

Replies

Fresh eyes this morning and I was able to debug this a bit more: in my Xcode scheme options, I set Console to Use Terminal, which let me then see the full command line invocation Xcode uses to launch the app, which I was able to copy-paste to my own terminal and confirm the problem persists, so it's because of how Xcode is running my app. Process of elimination found the culprit. Xcode sets the env var (among others)

DYLD_LIBRARY_PATH=$SYMROOT:$OBJROOT:/usr/lib/system/introspection

My plugin feature.dylib gets built into $OBJROOT and then copied into my app's expected plugin folder $PLUGINROOT along with its library dependency libexternal.so. When I call dlopen to load the plugin, I use the absolute path: dlopen("$PLUGINROOT/feature.dylib", 0). I use DYLD_PRINT_LIBRARIES=1 to see what dlopen actually loads. When I run MyApp.app without setting DYLD_LIBRARY_PATH, I see that dlopen opens $PLUGINROOT/feature.dylib as I expected. However, when I run MyApp.app with the DYLD_LIBRARY_PATH used by Xcode, I can see that dlopen opens $OBJROOT/feature.dylib. So that explains why running from within Xcode causes the plugin load to fail, because libexternal.so does not exist in $OBJROOT.

This seems like a bug! Right? I'm providing an absolute path to dlopen and it's using a different path based on the env var!
Posted by
diverdi
Up vote reply of diverdi Down vote reply of diverdi
Add a Comment
Wow, man dlopen says this isn't a bug!

When path contains a slash but is not a framework path (i.e. a full path or a partial path to a dylib), dlopen() searches the following until it finds a compatible Mach-O file: $DYLD_LIBRARY_PATH (with leaf name from path ), then the supplied path (using current working directory for relative paths), then $DYLD_FALLBACK_LIBRARY_PATH (with leaf name from path ).

This seems crazy to me, but at least now I can figure out how to solve the problem.
Posted by
diverdi
Up vote reply of diverdi Down vote reply of diverdi
Add a Comment