16 Replies
      Latest reply on Oct 1, 2019 7:23 AM by houdah
      iQser_Developer Level 1 Level 1 (0 points)

        Spotlight search for emails neither works with MDQueryRef nor with mdfind in the Terminal.app even if the user has granted full disc access for the app in the security settings. The application logic of our app works for High Sierra bot not for Catalina any more. Is this a bug or a feature? Hopefully it's not a feature. Other content like documents, calendar events and contacts can be retrieved by MDQueryRef.

         

        If I search with the default Spotlight interface (command-space), I can find emails. But even if I select "Show in Finder" in the result list, the Finder window is empty.

         

        Even with sudo mdfind, I can't find any email. I also can't add an email to the index with mdimport to the index, neither with the logged in user nor with sudo.

         

        Does someone has a solution? Hopefully we needn't index emails With Search Kit on our own in the future.

        • Re: How to enable Spotlight search for emails in macOS Catalina?
          eskimo Apple Staff Apple Staff (12,075 points)

          What’s happened here is that Mail has transition from using Spotlight to Core Spotlight to index the mail [1].  Given that, it makes sense that its results are no longer available via the Metadata framework.  However, “makes sense” is not the same as “it’s correct”.  If this change is causing you grief, my recommendation is that you file a bug report describing the problem.

          Please post your bug number, just for the record.

          Share and Enjoy

          Quinn “The Eskimo!”
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"

          [1] This, btw, is the reason behind the indexing issue (r. 46611310) discussed in the macOS Catalina 10.15 Release Notes.

            • Re: How to enable Spotlight search for emails in macOS Catalina?
              iQser_Developer Level 1 Level 1 (0 points)

              Thanks for your answer, that sound not good. We have filed afeedback with the number FB7016929. Do you have any solution for a workaround? Do we have to index the mail directory ourselves with SK Search Kit?

              • Re: How to enable Spotlight search for emails in macOS Catalina?
                iQser_Developer Level 1 Level 1 (0 points)

                We realized, that even Automator has a problem, if one use Spotlight in a workflow. Alternatively one can use Find Email Messages, but this takes minutes before Automator shows a result.

                • Re: How to enable Spotlight search for emails in macOS Catalina?
                  houdah Level 1 Level 1 (0 points)

                  I have also filed a bug report: FB7135903.

                  I consider Spotlight access to mail message public API. Removing this without deprecation notice or replacement is bound to break scripts and applications. It will create grief for both developers and users.

                   

                  --- Full text of the feedback report

                   

                  In macOS Catalina, mail messages stored in ~/Library/Mail are no longer indexed by Spotlight. Since messages are still found by Spotlight searches, I assume indexing has been moved to CoreSpotlight. This would appear to be an intentional change. I’d say an unfortunate one since it creates a regression in both public API and user expectations.

                   

                  In prior versions of macOS, Spotlight searches allowed programatic access to mail messages. Mail metadata was readily available and well documented as properties on MDItem. The removal of this OS feature breaks what in essence was public API.

                   

                  In macOS Mojave additional data protections were introduced. These include “Full Disk Access” where access to “application data like Mail” was explicitly mentioned. A year later Catalina removes all public access to Mail data.

                   

                  Basically third party developers and power users will be forced down a path of “private API”: navigate and parse the data storage in ~/Library/Mail by relying on undocumented data structures owned by the Mail application.

                   

                   

                  This change also breaks user expectations. Power users have relied on access to mail messages and metadata. E.g. scripts using the mdfind command could be used to find messages and trigger actions on their arrival.

                   

                  Many more user rely on third party products like HoudahSpot to perform advanced searches that are not possible using the interface Mail.app provides. As a developer I am saddened to see a product of mine being restricted in functionality.

                   

                  As a user I am worried to see more and more application data confined to closed “silos”. Previous versions of macOS / OS X have removed Safari bookmarks and history, Apple notes, etc. from indexing and thus from access by third party applications. This reduces the extensibility, scriptability, flexibility and thus usefulness of these core applications and ultimately the platform as a whole.

                    • Re: How to enable Spotlight search for emails in macOS Catalina?
                      iQser_Developer Level 1 Level 1 (0 points)

                      Thank you very much for supporting our cause.

                       

                      We had opened a DTS support ticket, but it was immediately closed again. We asked in follow up, if we could at least get the information if the new behavior is a feature or a bug.

                       

                      If it is a new feature, it has technical and economic consequences, because we have to invest in a probably expensive technical solution. Maybe the cimmunity manager @eskimo can help us.

                       

                      Unfortunately, we can assume that bug reports will not be processed. I think renaming the bug tracking system to Feedback Assistant is a clear message.

                        • Re: How to enable Spotlight search for emails in macOS Catalina?
                          eskimo Apple Staff Apple Staff (12,075 points)

                          We had opened a DTS support ticket, but it was immediately closed again.

                          I can’t discuss DTS business here on DevForums, but I can say that DTS does not support pre-release system software, and that includes the current 10.15 beta.

                          Maybe the community manager @eskimo can help us.

                          Ah um, I’m not a “community manager”, I’m just a guy whose boss tolerates him spending some fraction of his time answering questions here on the forums.

                          As to how this issue will pan out in the long term, I’m not able to say.  There’s an obvious conflict between the original Spotlight architecture (every app has access to everything) and the Core Spotlight architecture (every app is siloed), and it’s also obvious that the latter is better aligned with Apple’s ongoing privacy efforts.  You could imagine various ways that Apple could grant an app access to Core Spotlight results for specific other apps, but that support does not currently exist.  The way to request that support is to file a bug about it (which I see that you’ve both done, thanks!).

                          Share and Enjoy

                          Quinn “The Eskimo!”
                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                          let myEmail = "eskimo" + "1" + "@apple.com"

                            • Re: How to enable Spotlight search for emails in macOS Catalina?
                              iQser_Developer Level 1 Level 1 (0 points)

                              Ok, thanks very much for your answer.

                               

                              Because there is already a mechanism to protect emails by default, there shouldn't be a conceptual problem.

                               

                              • If a user has granted full disc access, spotlight results for all data on the disc should be available. Unfortunately that's not the case for Catalina.
                              • Apple could introduce another entitelement and security setting to access emails like calendar and addressbiik. I assume, that is unlikely.

                               

                              I hope, that Apple will offer a solution for the final release or at least clarify its plan for Catalina,but I'M afraid that we have to preapare a workaround.

                              • Re: How to enable Spotlight search for emails in macOS Catalina?
                                houdah Level 1 Level 1 (0 points)

                                I have submitted a feature request for public Core Spotlight API: FB7136032.

                                 

                                I don't think the siloing of Core Spotlight is part of Apple's privacy effort or that it actually aligns with this. The privacy effort is focussed on user consent. Once consent is given the data should be readily available. This allows for application integration, automation, platform extension and avoids duplicated effort. To me it seems we are actually looking at an incomplete implementation. API to search Core Spotlight is missing. API to access mail messages is missing.

                                 

                                For example: once access to photos is granted, photos can be accessed via the file system, via scripting, via PHPhotoLibrary, and via MLMediaLibrary frameworks. All sorts of things become (resp. remain) possible. All hinges on user consent.

                                 

                                The current siloing and move to Core Spotlight has two problems:

                                 

                                - Much information is no longer available as siloing proceeds faster than API evolution. E.g. there is no API to access not notes or email messages. This limits integration and automation opportunities. In some cases, third-party developers can resort to duplicating effort. E.g. by direct access to IMAP servers

                                 

                                - Where public API to silos exist (PHPhotoLibrary, Contacts, …) the API lack the unifying nature of Spotlight / NSMetadataQuery. A public API to Core Spotlight should solve that.

                                 

                                 

                                -- Full text of FB7136032

                                 

                                In macOS Catalina, mail messages stored in ~/Library/Mail are no longer indexed by Spotlight. Since messages are still found by Spotlight searches, I assume indexing has been moved to CoreSpotlight.

                                 

                                This continues a trend where Safari bookmarks, Safari history, Apple notes, etc. have been confined to private databases and made searchable in the Spotlight window by way of CoreSpotlight.

                                 

                                Third-party applications have also adopted CoreSpotlight. Many “shoebox” applications can now make data available to the Spotlight window in ways that could not be done using the Spotlight index. Mostly because the data is not readily available as individual files.

                                 

                                For the most part this change needs to be applauded as more data becomes available for searches.

                                 

                                It however also creates problems:

                                - more and more data is stored in proprietary monolithic databases

                                - scripts and applications that rely on core applications as “system services” will break. E.g. it was possible to use Spotlight to watch for incoming mail messages and react

                                - generally speaking applications are cut off from collaboration

                                 

                                Much of this could be fixed by opening up the API used by the Spotlight window to search the CoreSpotlight indexes. Currently the CSSearchQuery allows only for searching the current application’s data.

                                 

                                Please extend CSSearchQuery or provide other public API to search all application data indexed by CoreSpotlight. I.e. an API to search Apple Mail messages, notes, …

                                 

                                BTW, it may have been too early to move Apple Mail messages to CoreSpotlight. The technological need is not there (yet). Mail messages are still stored as individual .emlx files and could still be indexed. I assume CoreSpotlight addresses other needs of the Mail application. Breaking other applications that relied on Spotlight as “public API” to Mail messages could be avoided by having a transition period where Mail messages are indexed in both Spotlight and CoreSpotlight. Spotlight access to Mail messages could be deprecated and ultimately removed once public API through CoreSpotlight is released.

                                • Re: How to enable Spotlight search for emails in macOS Catalina?
                                  iQser_Developer Level 1 Level 1 (0 points)

                                  We have at least the confirmation from Apple that there is currently no alternative command line tool for mdfind that can query emails.

                                   

                                  The following workaround works, if the user has enabled full access to the hard drive. Unfortunately, it is not a good user experience to have to open the settings instead of confirming a dialog window. But it works.

                                   

                                  If an app synchronizes the emails with its sandbox, all emails are back in Spotlight's index and available through its API for this app.  Since the solution does not bypass security mechanisms and uses official interfaces like NSFileManager, I hope that this solution will be preserved in the final release.

                                    • Re: How to enable Spotlight search for emails in macOS Catalina?
                                      Smithers Level 1 Level 1 (5 points)

                                      Can you elaborate on this a little? How did you access emails from `CoreSpotlight` from inside of Mail and pull these into your own app?

                                        • Re: How to enable Spotlight search for emails in macOS Catalina?
                                          iQser_Developer Level 1 Level 1 (0 points)

                                          It's rather simple. Your app can just use NSFileManager to synchronize the ~/Library/Mail directory to the NSHomeDirectory or any subdirectory of your application's Sandbox. Afterwards the elmx files and attachments are indexed by Spotlight. So you can use your exsiting implementation, that makes use of the Spotlight API to retrieve emails.

                                           

                                          With NSFileManager you can check, whether the app has read access to the Library. If not you can show an NSAlert with some advice to grant full disk access for your application. I think this is a clean solution, that respects the security mechanisms of macOS. However it's not a good user experience to open the security settings for adding the app. A dialogue like for contacts or calender would be better.

                                            • Re: How to enable Spotlight search for emails in macOS Catalina?
                                              Smithers Level 1 Level 1 (5 points)

                                              I don't understand how that would work when the Mail app no longer indexes into physical .emxl files as it did before? I thought eskimo is trying to tell us that it now feeds this into the CoreSpotlight system and 'silos' its indexes away into a private database, which our sandboxed app won't be able to read any way.

                                               

                                              .emxl files are fine - we can read those, and have been reading those in the past too, but Mail seems to have stopped saving information into those files. If we were to copy those files into our own sandbox, how do we grab new emails that the Mail app supposedly isn't saving into ~/Library/Mail?

                                  • Re: How to enable Spotlight search for emails in macOS Catalina?
                                    houdah Level 1 Level 1 (0 points)

                                    It appears that MDQuery / NSMetadataQuery are perfectly capable of searching the Core Spotlight index. It seems that the Spotlight menu item / window uses this very API and gets preferential treatment by way of Apple-only entitlements.

                                     

                                    It is perfectly understandable that this pre-existing API did not simply start returning Core Spotlight search results. Such results may not be tied to actual files on disk. Thus Core Spotlight results may catch a caller of MDQuery off guard when this caller expects to get a file path.

                                     

                                    Yet it is sad to see that years after Core Spotlight stepped on the scene there still is no public API to search that data. Apple limits thus what power users can do and what developers can offer as extensions or alternatives to the default search tools.

                                     

                                    In the case of Mail messages on Catalina, the situation is even worse. Core Spotlight entries for Mail messages do map to files in ~/Library/Mail. Callers of MDQuery / NSMetadataQuery could not only handle such results, but they have also come to expect these. Yet these remain unavailable to third-party applications.

                                     

                                    This is at odds with Apple's current efforts in privacy and security. These efforts hinge on user consent. Yet, when it comes to Core Spotlight searches, Apple made the decision to block access to the data and left the user with no option to override. This closes the door on many automation and integration workflows.

                                     

                                    What's even more surprising - if not sad - is that the move provides no obvious privacy or security benefit. The user still has the option to grant an application access to the Mail folders. Malicious applications can continue to abuse such access to harvest private data. The absence of a Spotlight index will at best slow these down.

                                     

                                    It has been suggested that legitimate developers copy mail messages to a second location or create their own indexes. This is indeed what is likely to happen. Various developers, script authors, and power users will duplicate private data to locations not protected by Full Disk Access preferences. One a legitimate application has undertaken this task (and thus wasted disk space) a malicious application will need only seconds to uncover that new location using a Spotlight search and go about its nefarious business without the user even noticing.

                                     

                                    We end up with broken applications, broken scripts, artificially limited third-party applications, wasted disk space, and personal data put at risk. Please reconsider.

                                  • Re: How to enable Spotlight search for emails in macOS Catalina?
                                    Smithers Level 1 Level 1 (5 points)

                                    +1 on this, filed with ID FB7264664

                                     

                                    This effectively breaks our application, which relied on this for years to work.