User Profile

I have built my own privacy controls UI for HoudahSpot 5.0.10. Check out HoudahSpot > Preferences > Privacy.It has been a lot of work. Involves a lot of guesswork. It is convoluted code that tries to cover edge cases on various OS versions. Still, it is far from perfect and can't ever be perfect. The available API simply does not provide enough information to create a smooth experience.

It appears that MDQuery / NSMetadataQuery are perfectly capable of searching the Core Spotlight index. It seems that the Spotlight menu item / window uses this very API and gets preferential treatment by way of Apple-only entitlements.It is perfectly understandable that this pre-existing API did not simply start returning Core Spotlight search results. Such results may not be tied to actual files on disk. Thus Core Spotlight results may catch a caller of MDQuery off guard when this caller expects to get a file path.Yet it is sad to see that years after Core Spotlight stepped on the scene there still is no public API to search that data. Apple limits thus what power users can do and what developers can offer as extensions or alternatives to the default search tools.In the case of Mail messages on Catalina, the situation is even worse. Core Spotlight entries for Mail messages do map to files in ~/Library/Mail. Callers of MDQuery / NSMetadataQuery could not only handle such results, but they have also come to expect these. Yet these remain unavailable to third-party applications.This is at odds with Apple's current efforts in privacy and security. These efforts hinge on user consent. Yet, when it comes to Core Spotlight searches, Apple made the decision to block access to the data and left the user with no option to override. This closes the door on many automation and integration workflows.What's even more surprising - if not sad - is that the move provides no obvious privacy or security benefit. The user still has the option to grant an application access to the Mail folders. Malicious applications can continue to abuse such access to harvest private data. The absence of a Spotlight index will at best slow these down. It has been suggested that legitimate developers copy mail messages to a second location or create their own indexes. This is indeed what is likely to happen. Various developers, script authors, and power users will duplicate private data to locations not protected by Full Disk Access preferences. One a legitimate application has undertaken this task (and thus wasted disk space) a malicious application will need only seconds to uncover that new location using a Spotlight search and go about its nefarious business without the user even noticing.We end up with broken applications, broken scripts, artificially limited third-party applications, wasted disk space, and personal data put at risk. Please reconsider.

I have submitted a feature request for public Core Spotlight API: FB7136032.I don't think the siloing of Core Spotlight is part of Apple's privacy effort or that it actually aligns with this. The privacy effort is focussed on user consent. Once consent is given the data should be readily available. This allows for application integration, automation, platform extension and avoids duplicated effort. To me it seems we are actually looking at an incomplete implementation. API to search Core Spotlight is missing. API to access mail messages is missing.For example: once access to photos is granted, photos can be accessed via the file system, via scripting, via PHPhotoLibrary, and via MLMediaLibrary frameworks. All sorts of things become (resp. remain) possible. All hinges on user consent.The current siloing and move to Core Spotlight has two problems:- Much information is no longer available as siloing proceeds faster than API evolution. E.g. there is no API to access not notes or email messages. This limits integration and automation opportunities. In some cases, third-party developers can resort to duplicating effort. E.g. by direct access to IMAP servers- Where public API to silos exist (PHPhotoLibrary, Contacts, …) the API lack the unifying nature of Spotlight / NSMetadataQuery. A public API to Core Spotlight should solve that.-- Full text of FB7136032In macOS Catalina, mail messages stored in ~/Library/Mail are no longer indexed by Spotlight. Since messages are still found by Spotlight searches, I assume indexing has been moved to CoreSpotlight.This continues a trend where Safari bookmarks, Safari history, Apple notes, etc. have been confined to private databases and made searchable in the Spotlight window by way of CoreSpotlight.Third-party applications have also adopted CoreSpotlight. Many “shoebox” applications can now make data available to the Spotlight window in ways that could not be done using the Spotlight index. Mostly because the data is not readily available as individual files.For the most part this change needs to be applauded as more data becomes available for searches.It however also creates problems:- more and more data is stored in proprietary monolithic databases- scripts and applications that rely on core applications as “system services” will break. E.g. it was possible to use Spotlight to watch for incoming mail messages and react- generally speaking applications are cut off from collaborationMuch of this could be fixed by opening up the API used by the Spotlight window to search the CoreSpotlight indexes. Currently the CSSearchQuery allows only for searching the current application’s data.Please extend CSSearchQuery or provide other public API to search all application data indexed by CoreSpotlight. I.e. an API to search Apple Mail messages, notes, …BTW, it may have been too early to move Apple Mail messages to CoreSpotlight. The technological need is not there (yet). Mail messages are still stored as individual .emlx files and could still be indexed. I assume CoreSpotlight addresses other needs of the Mail application. Breaking other applications that relied on Spotlight as “public API” to Mail messages could be avoided by having a transition period where Mail messages are indexed in both Spotlight and CoreSpotlight. Spotlight access to Mail messages could be deprecated and ultimately removed once public API through CoreSpotlight is released.

I have also filed a bug report: FB7135903.I consider Spotlight access to mail message public API. Removing this without deprecation notice or replacement is bound to break scripts and applications. It will create grief for both developers and users.--- Full text of the feedback reportIn macOS Catalina, mail messages stored in ~/Library/Mail are no longer indexed by Spotlight. Since messages are still found by Spotlight searches, I assume indexing has been moved to CoreSpotlight. This would appear to be an intentional change. I’d say an unfortunate one since it creates a regression in both public API and user expectations.In prior versions of macOS, Spotlight searches allowed programatic access to mail messages. Mail metadata was readily available and well documented as properties on MDItem. The removal of this OS feature breaks what in essence was public API.In macOS Mojave additional data protections were introduced. These include “Full Disk Access” where access to “application data like Mail” was explicitly mentioned. A year later Catalina removes all public access to Mail data.Basically third party developers and power users will be forced down a path of “private API”: navigate and parse the data storage in ~/Library/Mail by relying on undocumented data structures owned by the Mail application.This change also breaks user expectations. Power users have relied on access to mail messages and metadata. E.g. scripts using the mdfind command could be used to find messages and trigger actions on their arrival.Many more user rely on third party products like HoudahSpot to perform advanced searches that are not possible using the interface Mail.app provides. As a developer I am saddened to see a product of mine being restricted in functionality.As a user I am worried to see more and more application data confined to closed “silos”. Previous versions of macOS / OS X have removed Safari bookmarks and history, Apple notes, etc. from indexing and thus from access by third party applications. This reduces the extensibility, scriptability, flexibility and thus usefulness of these core applications and ultimately the platform as a whole.