It appears that MDQuery / NSMetadataQuery are perfectly capable of searching the Core Spotlight index. It seems that the Spotlight menu item / window uses this very API and gets preferential treatment by way of Apple-only entitlements.
It is perfectly understandable that this pre-existing API did not simply start returning Core Spotlight search results. Such results may not be tied to actual files on disk. Thus Core Spotlight results may catch a caller of MDQuery off guard when this caller expects to get a file path.
Yet it is sad to see that years after Core Spotlight stepped on the scene there still is no public API to search that data. Apple limits thus what power users can do and what developers can offer as extensions or alternatives to the default search tools.
In the case of Mail messages on Catalina, the situation is even worse. Core Spotlight entries for Mail messages do map to files in ~/Library/Mail. Callers of MDQuery / NSMetadataQuery could not only handle such results, but they have also come to expect these. Yet these remain unavailable to third-party applications.
This is at odds with Apple's current efforts in privacy and security. These efforts hinge on user consent. Yet, when it comes to Core Spotlight searches, Apple made the decision to block access to the data and left the user with no option to override. This closes the door on many automation and integration workflows.
What's even more surprising - if not sad - is that the move provides no obvious privacy or security benefit. The user still has the option to grant an application access to the Mail folders. Malicious applications can continue to abuse such access to harvest private data. The absence of a Spotlight index will at best slow these down.
It has been suggested that legitimate developers copy mail messages to a second location or create their own indexes. This is indeed what is likely to happen. Various developers, script authors, and power users will duplicate private data to locations not protected by Full Disk Access preferences. One a legitimate application has undertaken this task (and thus wasted disk space) a malicious application will need only seconds to uncover that new location using a Spotlight search and go about its nefarious business without the user even noticing.
We end up with broken applications, broken scripts, artificially limited third-party applications, wasted disk space, and personal data put at risk. Please reconsider.
