First of all, I am going to summarize recent findings:
- The notarization service does not include the overall bundle in the ticketContents list so it does not recognize the overall bundle as a candidate for notarization. It lists just the dylib files and the main executable. As said by eskimo, the overall bundle should be listed as well.
- The dylib files themself do not seem to be the issue - I've changed deployment target to 14.4.4 so the swift's dylib files were no longer put into the bundle. The notarization service still did not included the overall bundle in the ticketContents list.
- Sending dmg instead of zip to the notarization service result in no change (except of including the dmg file in the ticketContents list). The overall bundle ticket is still missing.
As long as the eskimo is right that the overall bundle should be included in the ticketContents list, it seems that something is causing the notarization service to skip the overall bundle.
3. I've packed the bundle and transfered via web (so it becomes quarantined) to a fresh Catalina virtual machime
How did you pack it? Are you distributing it as a pkg file?
For the sake of these tests, I were transfering the same zip file as was sent to the notarization service.
When distributing to real users, the bundle is copied in a directory next to several other files, than tar-ed and gzip-ed. The server replaces a content of one "other" file found in the archive before serving it to the user while keeping other file entries intact in the stream. Little complicated but there is no known issue with this approach yet.
4. I've installed the third-party host application for which the our product is a plug-in onto the Catalina virtual machine
Did you run the host application first?
It does not matter. I've tried both variants.
5. I've placed the plug-in into a folder where the host application expects it
Where is that?
~/Library/Application Support/TheApp/Plugins
4. You are not running the host app first. Therefore, the host hasn't passed Gatekeeper yet. There could be some extra checks as part of the initial Gatekeeper check that are failing due to your differently-signed plug-in that is being loaded.
The failure appears on subsequent app starts as well.
3. Are you distributing this as a pkg file? If so, you could just run your fix as part of a post-install script. If not, you would write a custom installer (maybe even in AppleScript) that performs those tasks, in addition to installing the plugin.
1) The host app has its own plugin manager. When it downloads plugins, their signatures may be invalid or might not even exist. This may cause problems for said host app in the future. Apple has said that unsigned code will not run by default in a future version of macOS. But that's not your problem.
We are not distributing the plugin via the plugin manager (yet).
The user downloads the plugin from our website, uncompresses and double clicks on the bundle. This starts the host app (as the host app registers itself as a handler to that file extension) and the host app copies the bundle to its plugins folder located inside its Application Support subdirectory. On start-up, the host app tries to load all the bundles presents in that directory.
2) Again, revisit how those Apple dylibs are getting into your bundle. Are you still codesigning and notarizing them? What if you don't do that?
I were not codesigning and notarizing the dylib files in subsequent tests. I built the app to a new separate xcarchive for subsequent tests to have a clear starting point.
3) The host app provides some templates and instructions. Are you using those? Specifically, the instructions tell you to use the "--deep" flag on the codesign. That flag is frowned upon, but you might need it in this case, assuming 2) above doesn't fix it.
I've tried running codesign command initially as well. Now I let the Xcode to codesign (it did some codesigning as part of xcode archive).
I think eskimo's suggestion to set your deployment to 10.14.4 and omit those dylibs is probably the easiest solution. That will make your plug in available. In this market, there are likely some people running old versions so this might actually be a problem for you.
It did not help. I have not tried to run it yet but I followed eskimo's answer that the overall bundle should be listed in the notarization log. So I'm checking the log whether the overall bundle entry appeared.