2 Replies
      Latest reply on Jun 10, 2019 7:39 AM by alanwbrown
      beerator Level 1 Level 1 (0 points)

        Hopefully this has a somewhat straightforward reason, but I'm having trouble figuring this out.

         

        I'm using trying to use SearchKit to derive the search term list from a particular document on the filesystem. I have code to do this, written by consulting the API and the various (now admitedly deprecated) programming guides Apple offers. However, the normal place this would be running is in a 'command line' application, running as root, spawned by the launchd as a LaunchDaemon. In this context it appears SearchKit produces no errors, but it also produces no search term information.

         

        If I extract the code from the daemon and create a new command line application via Xcode, I get a similar results - no errors, but also no results.

         

        If I extract the code and place it in a Cocoa Application templated by Xcode but without the sandbox the code appears to work correctly, generating no errors and all of the texted search terms derived from the text file.

         

        In all cases SearchKit reports the document as indexed, and when I've created a persistent index on the file system looking at the contents of this file indicates it contains data regarding the expected paths, but apparently no search term data. I've called SKLoadDefaultExtractorPlugIns to load the spotlight importers and SKIndexFlush before looking at the number of search terms to synchronise the backing store for the index after adding documents.

         

        In the past when this kind of problem has cropped up my experience was that the API required something like an NSRunLoop or similar such construct to post work to. SearchKit doesn't document such a dependency, nor does adding an NSRunLoop fix the problem. My other theory was that the system was preventing applications running the in System launchd domain from performing work with SearchKit, however the regular command line application not working would seem to discount that.

         

        Consulting the system log closely show the command line application loading spotlight importer plugins, with messages like

         

        skfoundationtest: (CoreFoundation) [com.apple.CFBundle:resources] Resource lookup at CFURL file:///System/Library/

          47434         Request       :  type: mdimporter path: Spotlight

          47435         Result        : SystemPrefs.mdimporter/ -- file:///System/Library/Spotlight/, Chat.mdimporter/ -- file:///System/Library/Spotlight/, iPhoto.mdimporter/ -- file:///System/Library/Spotlight/, PDF.mdimporter/ -- file:///System/Library/Spotlight/, RichText.mdimporter/ -- file:///System/Library/Spotlight/, Bookmarks.mdimporter/ -- file:///System/Library/Spotlight/, Office.mdimporter/ -- file:///System/Library/Spotlight/, PS.mdimporter/ -- file:///System/Library/Spotlight/, MIDI.mdimporter/ -- file:///System/Library/Spotlight/, Archives.mdimporter/ -- file:///System/Library/Spotlight/, Audio.mdimporter/ -- file:///System/Library/Spotlight/, iPhoto8.mdimporter/ -- file:///System/Library/Spotlight/, Automator.mdimporter/ -- file:///System/Library/Spotlight/, Application.mdimporter/ -- file:///System/Library/Spotlight/, Font.mdimporter/ -- file:///System/Library/Spotlight/, Mail.mdimporter/ -- file:///System/Library/Spotlight/, QuartzComposer.mdimporter/ -- file:///System/Library/Spotlight/, vCard.mdimporter/ -- file:///System/Library/Spotlight/, Image.mdimporter/ -- file:///System/Library/Spotlight/, iCal.mdimporter/ -- file:///System/Library/Spotlight/, CoreMedia.mdimporter/ -- file:///System/Library/Spotlight/,

         

        which seem to indicate SearchKit is operating.

         

        The SearchKit documentation makes some cryptic remarks about how it will perform searches "This function creates an asynchronous search object for querying the document contents in an index. It also initiates the search on a separate thread." which suggests to me that SearchKit will manage the threading itself however could this indicate the absence of some runloop like entity in my application?

         

        Anyone's help on this would be much appreciated, I'm going to continue investigating on my own but I'm running out of avenues.

         

        The code is morally equivalent to the following in all cases

         

        
        import Foundation
        import CoreServices
        import os
        
        class FileContentInspector {
            
            init?() {
                contentScanIndexBuffer = CFDataCreateMutable(kCFAllocatorDefault, 0)
                
                let stopWords: Set = []
                let indexOptions: [String: Any] = [
                    "kSKMinTermLength": 1,
                    "kSKMaximumTerms": 0,
                    "kSKStopWords": stopWords
                ]
                
                SKLoadDefaultExtractorPlugIns()
                guard let unmanagedIndex = SKIndexCreateWithMutableData(
                    contentScanIndexBuffer,
                    "fileContentScan" as CFString,
                    kSKIndexInvertedVector,
                    indexOptions as CFDictionary) else {
                        if #available(macOS 10.12, *) {
                            os_log("%u: failed to create index with URL", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(query).hashValue)
                        }
                        return nil
                }
                contentScanIndex = unmanagedIndex.takeRetainedValue()
            }
            
            func scanFileTerms(forURL url: URL, pattern: String, withCompletion: @escaping((Bool) -> Void)) -> Void {
                if #available(macOS 10.12, *) {
                    os_log("%u scanFileTerms %{public}@ %{public}@", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue, url.scheme ?? "unknown", url.absoluteString)
                }
                
                let doc = SKDocumentCreateWithURL(url as CFURL)?.takeRetainedValue()
                
                guard SKIndexAddDocument(contentScanIndex, doc, nil, true) else {
                    if #available(macOS 10.12, *) {
                        os_log("%u failed to add document to index", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue)
                    }
                    withCompletion(false)
                    return
                }
                
                if #available(macOS 10.12, *) {
                    os_log("%u terms index size %u bytes %{public}@ %{public}@", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue, CFDataGetLength(contentScanIndexBuffer), String(describing: SKDocumentCopyURL(doc)?.takeRetainedValue() as URL?), String(describing: SKDocumentGetSchemeName(doc)?.takeUnretainedValue() as String?))
                }
                
                let r = SKIndexFlush(contentScanIndex)
                if r == true {
                    if #available(macOS 10.12, *) {
                        os_log("%u flushed index", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue)
                    }
                } else {
                    if #available(macOS 10.12, *) {
                        os_log("%u failed to flush index", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue)
                    }
                    withCompletion(false)
                    return
                }
                
                let docID = SKIndexGetDocumentID(contentScanIndex, doc)
                
                // --
                let state = SKIndexGetDocumentState(contentScanIndex, doc)
                switch state {
                case kSKDocumentStateNotIndexed:
                    if #available(macOS 10.12, *) {
                        os_log("%u document %u is not indexed", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue, docID)
                    }
                case kSKDocumentStateIndexed:
                    if #available(macOS 10.12, *) {
                        os_log("%u document %u is indexed", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue, docID)
                    }
                case kSKDocumentStateAddPending:
                    if #available(macOS 10.12, *) {
                        os_log("%u document %u is add pending", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue, docID)
                    }
                case kSKDocumentStateDeletePending:
                    if #available(macOS 10.12, *) {
                        os_log("%u document %u is del pending", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue, docID)
                    }
                default:
                    if #available(macOS 10.12, *) {
                        os_log("%u document %u is in an unknown state", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue, docID)
                    }
                }
                
                if #available(macOS 10.12, *) {
                    os_log("%u terms numbered %u %u %u bytes", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue, SKIndexGetMaximumTermID(contentScanIndex), SKIndexGetDocumentTermCount(contentScanIndex, docID), CFDataGetLength(contentScanIndexBuffer))
                }
                
                if let termIDList = SKIndexCopyTermIDArrayForDocumentID(contentScanIndex, docID)?.takeRetainedValue() {
                    if #available(macOS 10.12, *) {
                        os_log("%u terms numbered %u %u %u %u %u", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue, docID, SKIndexGetDocumentCount(contentScanIndex), SKIndexGetMaximumTermID(contentScanIndex), SKIndexGetDocumentTermCount(contentScanIndex, docID), CFArrayGetCount(termIDList))
                    }
                    for i in 0..<cfarraygetcount(termidlist) {<br="">                let termIDPtr = (unsafeBitCast(CFArrayGetValueAtIndex(termIDList, i), to: CFNumber.self) as NSNumber)
                        let termID = termIDPtr.intValue
                        guard let term = SKIndexCopyTermStringForTermID(contentScanIndex, termID)?.takeRetainedValue() as String? else {
                            if #available(macOS 10.12, *) {
                                os_log("%u terms is term was not a string", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue)
                            }
                            withCompletion(false)
                            return
                        }
                        withCompletion(true)
                        return
                    }
                } else {
                    if #available(macOS 10.12, *) {
                        os_log("%u no search terms found", log: FileContentInspector.logger, type: .debug, ObjectIdentifier(self.query).hashValue)
                    }
                }
                withCompletion(false)
                return
            }
            
            var queryURL: URL? = nil
            var complationHandler: ((Bool) -> Void)? = nil
            let query: NSMetadataQuery = NSMetadataQuery()
            let contentScanIndexBuffer: CFMutableData
            let contentScanIndex: SKIndex
            @available(macOS 10.12, *) static let logger = OSLog(subsystem: "myapp", category: "filecontentinspector")
        }
        • Re: SearchKit not producing search terms
          beerator Level 1 Level 1 (0 points)

          I should also mention, in my test command line application running as a regular user (not from launchd where I know linking AppKit doesn't work properly) I also tried extending this with a manually created NSApplication and NSApplicationDelegate instance in case AppKit was doing some magic here. I did the work off func applicationDidFinishLaunching(_ notification: Notification) and while this executed correctly (so I assume I made all the necessary parts) it, again SearchKit produced no errors and no results.

            • Re: SearchKit not producing search terms
              alanwbrown Level 1 Level 1 (0 points)

              Just to close this out, the long and the short is that it appears SearchKit doesn't work when run as the root user (from what I can tell). I'm assuming this is for security reasons to revent importer plugins being loaded and run as root when they shouldn't be. Taking a look at the processes (and what user they're running as) which make up the Spotlight service sort of indicate this is the case.

               

              I'd been confused because my test programs where using FileManager.homeDirectoryForCurrentUser which I'd forgotten and was causing the test program to try indexing the wrong path when run as the wrong user. Put a somewhat embarrassing spanner in the works .