How to enable Spotlight search for emails in macOS Catalina?

Spotlight search for emails neither works with MDQueryRef nor with mdfind in the Terminal.app even if the user has granted full disc access for the app in the security settings. The application logic of our app works for High Sierra bot not for Catalina any more. Is this a bug or a feature? Hopefully it's not a feature. Other content like documents, calendar events and contacts can be retrieved by MDQueryRef.


If I search with the default Spotlight interface (command-space), I can find emails. But even if I select "Show in Finder" in the result list, the Finder window is empty.


Even with sudo mdfind, I can't find any email. I also can't add an email to the index with mdimport to the index, neither with the logged in user nor with sudo.


Does someone has a solution? Hopefully we needn't index emails With Search Kit on our own in the future.

Replies

What’s happened here is that Mail has transition from using Spotlight to Core Spotlight to index the mail [1]. Given that, it makes sense that its results are no longer available via the Metadata framework. However, “makes sense” is not the same as “it’s correct”. If this change is causing you grief, my recommendation is that you file a bug report describing the problem.

Please post your bug number, just for the record.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

[1] This, btw, is the reason behind the indexing issue (r. 46611310) discussed in the macOS Catalina 10.15 Release Notes.

Thanks for your answer, that sound not good. We have filed afeedback with the number FB7016929. Do you have any solution for a workaround? Do we have to index the mail directory ourselves with SK Search Kit?

We have filed a feedback with the number FB7016929.

Thanks.

Do you have any solution for a workaround?

No, sorry. I’ve not looked into this officially, so I’m loathe to suggest anything concrete.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

We realized, that even Automator has a problem, if one use Spotlight in a workflow. Alternatively one can use Find Email Messages, but this takes minutes before Automator shows a result.

I will open a support ticket. We need to know, whether we have to invest in an own index solution or just need to wait for a bug fix. Do you have some information about the plan?

I have also filed a bug report: FB7135903.

I consider Spotlight access to mail message public API. Removing this without deprecation notice or replacement is bound to break scripts and applications. It will create grief for both developers and users.


--- Full text of the feedback report


In macOS Catalina, mail messages stored in ~/Library/Mail are no longer indexed by Spotlight. Since messages are still found by Spotlight searches, I assume indexing has been moved to CoreSpotlight. This would appear to be an intentional change. I’d say an unfortunate one since it creates a regression in both public API and user expectations.


In prior versions of macOS, Spotlight searches allowed programatic access to mail messages. Mail metadata was readily available and well documented as properties on MDItem. The removal of this OS feature breaks what in essence was public API.


In macOS Mojave additional data protections were introduced. These include “Full Disk Access” where access to “application data like Mail” was explicitly mentioned. A year later Catalina removes all public access to Mail data.


Basically third party developers and power users will be forced down a path of “private API”: navigate and parse the data storage in ~/Library/Mail by relying on undocumented data structures owned by the Mail application.



This change also breaks user expectations. Power users have relied on access to mail messages and metadata. E.g. scripts using the mdfind command could be used to find messages and trigger actions on their arrival.


Many more user rely on third party products like HoudahSpot to perform advanced searches that are not possible using the interface Mail.app provides. As a developer I am saddened to see a product of mine being restricted in functionality.


As a user I am worried to see more and more application data confined to closed “silos”. Previous versions of macOS / OS X have removed Safari bookmarks and history, Apple notes, etc. from indexing and thus from access by third party applications. This reduces the extensibility, scriptability, flexibility and thus usefulness of these core applications and ultimately the platform as a whole.

Thank you very much for supporting our cause.


We had opened a DTS support ticket, but it was immediately closed again. We asked in follow up, if we could at least get the information if the new behavior is a feature or a bug.


If it is a new feature, it has technical and economic consequences, because we have to invest in a probably expensive technical solution. Maybe the cimmunity manager @eskimo can help us.


Unfortunately, we can assume that bug reports will not be processed. I think renaming the bug tracking system to Feedback Assistant is a clear message.

We had opened a DTS support ticket, but it was immediately closed again.

I can’t discuss DTS business here on DevForums, but I can say that DTS does not support pre-release system software, and that includes the current 10.15 beta.

Maybe the community manager @eskimo can help us.

Ah um, I’m not a “community manager”, I’m just a guy whose boss tolerates him spending some fraction of his time answering questions here on the forums.

As to how this issue will pan out in the long term, I’m not able to say. There’s an obvious conflict between the original Spotlight architecture (every app has access to everything) and the Core Spotlight architecture (every app is siloed), and it’s also obvious that the latter is better aligned with Apple’s ongoing privacy efforts. You could imagine various ways that Apple could grant an app access to Core Spotlight results for specific other apps, but that support does not currently exist. The way to request that support is to file a bug about it (which I see that you’ve both done, thanks!).

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Ok, thanks very much for your answer.


Because there is already a mechanism to protect emails by default, there shouldn't be a conceptual problem.


  • If a user has granted full disc access, spotlight results for all data on the disc should be available. Unfortunately that's not the case for Catalina.
  • Apple could introduce another entitelement and security setting to access emails like calendar and addressbiik. I assume, that is unlikely.


I hope, that Apple will offer a solution for the final release or at least clarify its plan for Catalina,but I'M afraid that we have to preapare a workaround.

I have submitted a feature request for public Core Spotlight API: FB7136032.


I don't think the siloing of Core Spotlight is part of Apple's privacy effort or that it actually aligns with this. The privacy effort is focussed on user consent. Once consent is given the data should be readily available. This allows for application integration, automation, platform extension and avoids duplicated effort. To me it seems we are actually looking at an incomplete implementation. API to search Core Spotlight is missing. API to access mail messages is missing.


For example: once access to photos is granted, photos can be accessed via the file system, via scripting, via PHPhotoLibrary, and via MLMediaLibrary frameworks. All sorts of things become (resp. remain) possible. All hinges on user consent.


The current siloing and move to Core Spotlight has two problems:


- Much information is no longer available as siloing proceeds faster than API evolution. E.g. there is no API to access not notes or email messages. This limits integration and automation opportunities. In some cases, third-party developers can resort to duplicating effort. E.g. by direct access to IMAP servers


- Where public API to silos exist (PHPhotoLibrary, Contacts, …) the API lack the unifying nature of Spotlight / NSMetadataQuery. A public API to Core Spotlight should solve that.



-- Full text of FB7136032


In macOS Catalina, mail messages stored in ~/Library/Mail are no longer indexed by Spotlight. Since messages are still found by Spotlight searches, I assume indexing has been moved to CoreSpotlight.


This continues a trend where Safari bookmarks, Safari history, Apple notes, etc. have been confined to private databases and made searchable in the Spotlight window by way of CoreSpotlight.


Third-party applications have also adopted CoreSpotlight. Many “shoebox” applications can now make data available to the Spotlight window in ways that could not be done using the Spotlight index. Mostly because the data is not readily available as individual files.


For the most part this change needs to be applauded as more data becomes available for searches.


It however also creates problems:

- more and more data is stored in proprietary monolithic databases

- scripts and applications that rely on core applications as “system services” will break. E.g. it was possible to use Spotlight to watch for incoming mail messages and react

- generally speaking applications are cut off from collaboration


Much of this could be fixed by opening up the API used by the Spotlight window to search the CoreSpotlight indexes. Currently the CSSearchQuery allows only for searching the current application’s data.


Please extend CSSearchQuery or provide other public API to search all application data indexed by CoreSpotlight. I.e. an API to search Apple Mail messages, notes, …


BTW, it may have been too early to move Apple Mail messages to CoreSpotlight. The technological need is not there (yet). Mail messages are still stored as individual .emlx files and could still be indexed. I assume CoreSpotlight addresses other needs of the Mail application. Breaking other applications that relied on Spotlight as “public API” to Mail messages could be avoided by having a transition period where Mail messages are indexed in both Spotlight and CoreSpotlight. Spotlight access to Mail messages could be deprecated and ultimately removed once public API through CoreSpotlight is released.

We have at least the confirmation from Apple that there is currently no alternative command line tool for mdfind that can query emails.


The following workaround works, if the user has enabled full access to the hard drive. Unfortunately, it is not a good user experience to have to open the settings instead of confirming a dialog window. But it works.


If an app synchronizes the emails with its sandbox, all emails are back in Spotlight's index and available through its API for this app. Since the solution does not bypass security mechanisms and uses official interfaces like NSFileManager, I hope that this solution will be preserved in the final release.

+1 on this, filed with ID FB7264664


This effectively breaks our application, which relied on this for years to work.

Can you elaborate on this a little? How did you access emails from `CoreSpotlight` from inside of Mail and pull these into your own app?

It's rather simple. Your app can just use NSFileManager to synchronize the ~/Library/Mail directory to the NSHomeDirectory or any subdirectory of your application's Sandbox. Afterwards the elmx files and attachments are indexed by Spotlight. So you can use your exsiting implementation, that makes use of the Spotlight API to retrieve emails.


With NSFileManager you can check, whether the app has read access to the Library. If not you can show an NSAlert with some advice to grant full disk access for your application. I think this is a clean solution, that respects the security mechanisms of macOS. However it's not a good user experience to open the security settings for adding the app. A dialogue like for contacts or calender would be better.

I don't understand how that would work when the Mail app no longer indexes into physical .emxl files as it did before? I thought eskimo is trying to tell us that it now feeds this into the CoreSpotlight system and 'silos' its indexes away into a private database, which our sandboxed app won't be able to read any way.


.emxl files are fine - we can read those, and have been reading those in the past too, but Mail seems to have stopped saving information into those files. If we were to copy those files into our own sandbox, how do we grab new emails that the Mail app supposedly isn't saving into ~/Library/Mail?