Optionally load dynamic SwiftPM

I want to support iOS 12+, but also want to use a SwiftPM package that only supports iOS 13+. The SwiftPM library is dynamic, so I want to build it, copy it to Frameworks, but not link to it, and then load it at runtime on iOS 13+. The last step is not too complex:

if(@available(iOS 13, *)) {
        dlopen("MySwiftPMPackage.framework/MySwiftPMPackage", RTLD_LAZY);
        Class MySwiftPMPackageViewController = NSClassFromString(@"MySwiftPMPackageViewController");
        // create an instance of `MySwiftPMPackageViewController` and use it
    }

This works if I add the package to my target in "General > Frameworks, Libraries..." and "Build Phases > Target Dependencies" and then remove it from "Link binary with libraries" (despite the crash there in Xcode 14 the first time I try). It will then run on iOS 12 and iOS 13+. The problem is, every time I reload the project, the library is added back to "Link binary with libraries", presumably because the package refreshes. Is there any way to prevent it from being added back there?