NSMetadataQuery threading issues

I was only able to get the threading to work by calling RunLoop.main.run().

Basically, yes, that's correct. NSMetadataQuery is an older API and, because of that, it's built around the run loop model because that's how basically ALL of our APIs worked.

Related to that point:

I cannot currently use Swift Concurrency (await/async/TaskGroup) because the executable must support macOS versions that don't support Swift Concurrency.

I'm not confident that any of those alternatives would actually make this work any better.

The only way I was able to allow the executable to return instead of being stuck forever on RunLoop.main.run() was to call exit(0) in the closure passed to Promise.done().

For what it's worth, calling "exit(0)" is actually that standard way most of the system stops the main thread. Strictly speaking, removing all of the input sources from a run loop will cause it to return but, in practice, an app of any significant complexity doesn't really "know" what all of it's input sources are, nor does it have any easy way to remove them. It's easier, faster, and more reliable to call "exit", so that's what the system does.

The exit(0) causes problems for testing.

Why? I'm not sure what issue you're actually having, but calling exit() like this is actually that standard way this kind of command line tool would terminate.

How can I allow the executable to exit without explicitly calling exit(0),

Well the direct alternative would be RunLoop.run(until:), which will block until the date is reached then return. A typical implementation of this kind of pattern looks something like this:

var done = false ... while(!done) { let nextStop = Date().addingTimeInterval(1) RunLoop.main.run(until: nextStop) }

When you've finished whatever work you want to do, your code sets done to "true" and the loop will then stop the next time run(until:) returns. Having said that, the reason this kind of code is less common than simply calling "exit" is that it isn't really any "better". It adds some latency the time interval to whatever comes "next" (after the loop returns) and it doesn't actually "do" anything better.

Putting that another way, once whatever needs to be done is "done", you nearly always end up with end up with a sequence something like this:

• Some more "work" needs to be done. I could wait for the runloop to leave the loop... or I could just "do" the work, either at the current call point or asynchronously on the runloop itself. This cycle repeats until all work is done.

• Having finished all work, I'm not ready to exit. I can either:

1. Wait for the RunLoop to return, then return from main(). This works fine but adds latency.

2. Call exit(). This works fine but does not add latency.

The system itself sees exactly the same thing, so #1 is basically the same as #2 only slower.

and how can I improve the threading overall?

I haven't looked at your code in great detail, but it's not clear to me that you actually have anything to improve. What problem are you actually having?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks for the info.

I read about CFRunLoopGetCurrent() online, so I updated the code around the call to appInfo(forAdamID:) to be the code at the end of this reply. The new version doesn't call exit(0) and doesn't interfere with my existing testing. Note that the rest of the code remains the same, so I still use the main DispatchQueue in appInfo(forAdamID:).

Is there anything wrong with doing this instead of using RunLoop.main.run() & exit(0)?

Is there anything inconsistent with using CFRunLoopGetCurrent() while using the main DispatchQueue in appInfo(forAdamID:)?

If it is inconsistent, how can I fix the inconsistencies?

Is there a current DispatchQueue? Or a way to get the DispatchQueue from the CFRunLoopGetCurrent()?

let runLoop = CFRunLoopGetCurrent()
_ = appInfo(forAdamID: adamID)
    .done { appInfo in
        CFRunLoopStop(runLoop)
        if let jsonData = try? JSONSerialization.data(withJSONObject: appInfo),
            let jsonString = String(data: jsonData, encoding: .utf8)
        {
            print(jsonString.replacingOccurrences(of: "\\/", with: "/"))
        }
    }
CFRunLoopRun()

Having had many conversation on this topic, run loop are really hard to explain and, even worse, it's hard to make sense of our APIs if you don't have a good concept of what's going on.

However, before I make my own run at this I should mention that you this is probably the best formal documentation on them and that Quinn has also written up his own wake.

My Take On RunLoops...

I think the best way to understand what's actually going on here is that there are actually two different somewhat different architectures at work here, only one of which we explicitly name. Those are:

1. "the run loop". This is a collection of inputs (mach ports, timers, etc) and corresponding callback (selectors, function pointers, etc.). This collection of data is attached to a particular thread and is, effectively, part of the data that is "a thread". Our documentation talks about how run loops are neither create nor destroyed and this is why. The "collection" that is "the run loop" can be "empty", but it can't really "not exist".

2. CFRunLoop, the API used to interact with and, in particular "run the run loop". I specifically say CFRunLoop because NSRunLoop/RunLoop is in fact a wrapper around CFRunLoop, not an "independent" object in its own right. RunLoop exists to provide a more convenient API than CFRunLoop (a job it's reasonably good at), not to replace it. The key point here is that there isn't any difference between CFRunLoopRun and RunLoop.run. Both of them ultimately call into an underlying CFRunLoop function with "runs the run loop", so which one you use doesn't actual change what's going on.

What "running the run loop" actually means is that the run call is receives from one of it's input source" and then calling whatever callback was configured for that input source. If none of the input sources have anything to deliver, then the thread blocks waiting for input.

Case in point, if you've spent any time looking at crash logs, you've almost certainly seen many thread doing this:

0   libsystem_kernel.dylib        	0x00000001e9028808 mach_msg2_trap + 8 (:-1)
1   libsystem_kernel.dylib        	0x00000001e902c008 mach_msg2_internal + 80 (mach_msg.c:201)
2   libsystem_kernel.dylib        	0x00000001e902bf20 mach_msg_overwrite + 436 (mach_msg.c:0)
3   libsystem_kernel.dylib        	0x00000001e902bd60 mach_msg + 24 (mach_msg.c:323)
4   CoreFoundation                	0x000000019ff48f5c __CFRunLoopServiceMachPort + 160 (CFRunLoop.c:2624)
5   CoreFoundation                	0x000000019ff48600 __CFRunLoopRun + 1208 (CFRunLoop.c:3007)
6   CoreFoundation                	0x000000019ff47cd8 CFRunLoopRunSpecific + 608 (CFRunLoop.c:3420)
7   Foundation                    	0x000000019ee68e4c -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 212 (NSRunLoop.m:373)
8   Foundation                    	0x000000019ee68c9c -[NSRunLoop(NSRunLoop) runUntilDate:] + 64 (NSRunLoop.m:420)

...which is how CFRunLoop blocks waiting for input. The key point here is that there isn't any conflict/issue between CFRunLoop and NSRunLoop. I almost always use to NSRunLoop, but that's simply because the API is nicer*.

*If you're wondering why CFRunLoop (and CoreFoundation more broadly) exists at all when NSRunLoop is "better", the main reason is "Because Apple has problems you probably don't have". More specifically, there are many system components which CANNOT use ObjectiveC (or call into Foundation) because they are dependencies of the ObjectiveC runtime (and/or Foundation). CoreFoundation lets them create ObjectiveC types (through toll free bridging) and common constructs (like the run loop) without actually using ObjectiveC.

With all that background, let me return to some of your specific questions (slightly paraphrased):

Is there anything wrong with using CFRunLoopStop instead of using RunLoop.main.run() & exit(0)?

There isn't, but there are some issues with your code. I would write your stop call as:

CFRunLoopStop(CFRunLoopGetCurrent())
or
CFRunLoopStop(RunLoop.currentRunLoop.getCFRunLoop)

The issue here is tied to this warning at the top of the RunLoop documentation:

Warning
The RunLoop class is generally not thread-safe, and you must call its methods only within the context of the current thread. Don’t call the methods of a RunLoop object running in a different thread, which might cause unexpected results.
Again, the vocabulary gets in the way of clarity. The issue here isn't that RunLoop isn't "thread safe". Indeed, many basic RunLoop methods like perform(_:) specifically exist so that one thread can use the run loop to execute code on another thread. Similarly, the entire reason RunLoop.main exists is so that other threads can send messages to the main thread.

The actual issue here is that the entire point of the run loop system is to create a mechanism where a set of events are delivered to a specific thread which then results in a corresponding callback on that thread. Because of that, calling ANY of the "run" methods from another thread would be inherently nonsensical.

As another example of this, you should never actually do this:

RunLoop.main.run()

but should instead to this:

RunLoop.currentRunLoop.run(

If you're on the main thread, those two calls are identical. If you're NOT on the main thread, then the first call crashed your app.

Finally, how GCD fits into this. The key thing to understand here is that:

DispatchQueue.main

...was specifically created to "connect" GCD into the main thread runloop architecture.

*Yes, I'm aware of dispatchMain, it's not relevant here.

Making that concrete, while their internal implementation details are different, these two calls:

DispatchQueue.main.async {
	query.start()
}

RunLoop.main.perform {
	query.start()
}

...do EXACTLY the same thing. Both of them schedule a block to run on the main runloop, which will the execute at the "next" opportunity, which will depend on whatever else is going on.

Or a way to get the DispatchQueue from the CFRunLoopGetCurrent()?

No, not for the general (any run loop) case. The idea of GCD is to break the connection between input processing and specific thread, and that connection is the entire point of run loops.

The main run loop is the exception. The main run loop is so critical to the system broader functionality, that removing Dispatch.main would mean developer would just end up adding in a bunch of extra calls to RunLoop.main.perform, so GCD added a special queue to avoid that.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks for the info.

If I start the run loop with CFRunLoopRun(), then CFRunLoopStop(…) stops the loop and lets my command-line executable exit.

If I start the run loop with RunLoop.main.run(), then CFRunLoopStop(…) doesn't stop the loop, so my command-line executable never exits.

All 3 of the following stopping commands seem to work with CFRunLoopRun(), but none of them works with RunLoop.main.run():

CFRunLoopStop(runLoop)
CFRunLoopStop(RunLoop.main.getCFRunLoop())
CFRunLoopStop(CFRunLoopGetCurrent())

Also, both of the following starting commands seem to work with CFRunLoopRun(), but neither of them works with RunLoop.main.run():

RunLoop.main.perform {…}
DispatchQueue.main.async {…}

Why does CFRunLoopRun(), but not RunLoop.main.run(), get stopped by CFRunLoopStop(…)?

It would make sense to me if RunLoop.main.run() wasn't stopped by any of them if CFRunLoopStop(RunLoop.main.getCFRunLoop()) didn't stop but CFRunLoopStop(CFRunLoopGetCurrent()) did, as maybe those are 2 different CFRunLoops, but I would assume that RunLoop.main & RunLoop.main.getCFRunLoop() use the same CFRunLoop underneath…

(When I say "works" above, I mean that CFRunLoopStop(…) actually lets my executable exit; if it doesn't work, the executable never returns)

them works with RunLoop.main.run():

Stop doing this. Do NOT call "run" on "main". It's an inherently risk practice that creates unnecessary confusion about how the RunLoop system actually works. With both CFRunLoop and RunLoop, you should only run the current run loop.

Why does CFRunLoopRun(), but not RunLoop.main.run(), get stopped by CFRunLoopStop(…)?

Because of a subtle (but documented) detail of run's implementation. The RunLoop.run() documentation says:

Puts the receiver into a permanent loop, during which time it processes data from all attached input sources.

Now, compare that to the documentation of RunLoop.run(mode:before:):

Runs the loop once, blocking for input in the specified mode until a given date.

The word "Runs the loop once" are the magic words. RunLoop.run(mode:before:) is the "core" API and, in fact, CFRunLoopStop works fine with it. RunLoop.run doesn't return because it's actual swift implementation would (literally) be:

while(RunLoop.current.run(mode: .default, before: Date.distantFuture)) {}

...which means CFRunLoopStop actually works as you'd expect, except for the fact that "run" immediately calls into run(mode:before:) again instead of returning.

In any case, here is a small sample that demonstrates exactly what's going on:

//
//  main.swift
//  RunLoopStopTest
//
//  Created by Kevin Elliott on 10/18/24.
//

import Foundation

print("Run RunLoop Run!")

let myTimer = Timer.scheduledTimer(withTimeInterval: 3.0, repeats: true) { _ in
    print(" Ping...")
    DispatchQueue.main.async {
        print("  Pong...")
        CFRunLoopStop(RunLoop.current.getCFRunLoop())
    }
}
print("CFRunLoopRun Called")
CFRunLoopRun()
print("CFRunLoopRun returned\n")
print("RunLoop Called")
RunLoop.current.run(mode: .default, before: Date.distantFuture)
print("RunLoop returned\n")
while(RunLoop.current.run(mode: .default, before: Date.distantFuture)) {
    print("   Ding!\n")
}

Which prints this when run:

Run RunLoop Run!
CFRunLoopRun Called
 Ping...
  Pong...
CFRunLoopRun returned

RunLoop Called
 Ping...
  Pong...
RunLoop returned

 Ping...
  Pong...
   Ding!

 Ping...
  Pong...
   Ding!

 Ping...
  Pong...
   Ding!
....

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks for all the info.

I've replaced all uses of anything with "main" to current. See Code section at end.

1

Basically, yes, that's correct. NSMetadataQuery is an older API and, because of that, it's built around the run loop model because that's how basically ALL of our APIs worked.

Is there any newer API that replaces NSMetadataQuery?

2

The key point here is that there isn't any difference between CFRunLoopRun and RunLoop.run

I think that confused me because there's actually a difference between the following two:

CFRunLoopRun()
RunLoop.run()

but not between the following two:

CFRunLoopRun()
RunLoop.run(mode:before:)

as per:

Because of a subtle (but documented) detail of run's implementation. The

Puts the receiver into a permanent loop, during which time it processes data from all attached input sources.

Now, compare that to the documentation of RunLoop.run(mode:before:):

Runs the loop once, blocking for input in the specified mode until a given date.

The word "Runs the loop once" are the magic words. RunLoop.run(mode:before:) is the "core" API and, in fact, CFRunLoopStop works fine with it. RunLoop.run doesn't return because it's actual swift implementation would (literally) be

Code

I've changed 4 lines from my original code. Each has a numbered // CHANGE comment at the end of the changed line.

Are there any better solutions than the changes that I've made?

import Foundation
import PromiseKit

guard CommandLine.arguments.count > 1 else {
    print("Missing adamID argument")
    exit(1)
}

guard let adamID = UInt64(CommandLine.arguments[1]) else {
    print("adamID argument must be a UInt64")
    exit(2)
}

_ = appInfo(forAdamID: adamID)
    .done { appInfo in
        if let jsonData = try? JSONSerialization.data(withJSONObject: appInfo),
            let jsonString = String(data: jsonData, encoding: .utf8)
        {
            print(jsonString.replacingOccurrences(of: "\\/", with: "/"))
        }
        CFRunLoopStop(CFRunLoopGetCurrent())  // CHANGE 1
    }

CFRunLoopRun()  // CHANGE 2

func appInfo(forAdamID adamID: UInt64) -> Promise<[String: Any]> {
    Promise { seal in
        let query = NSMetadataQuery()
        query.predicate = NSPredicate(format: "kMDItemAppStoreAdamID == %d", adamID)
        query.searchScopes = ["/Applications"]

        var observer: NSObjectProtocol?
        observer = NotificationCenter.default.addObserver(
            forName: NSNotification.Name.NSMetadataQueryDidFinishGathering,
            object: query,
            queue: .current  // CHANGE 3
        ) { _ in
            query.stop()
            defer {
                if let observer {
                    NotificationCenter.default.removeObserver(observer)
                }
            }

            var appInfo: [String: Any] = [:]
            for result in query.results {
                if let result = result as? NSMetadataItem {
                    var attributes = ["kMDItemPath"]
                    attributes.append(contentsOf: result.attributes)
                    for attribute in attributes {
                        let value = result.value(forAttribute: attribute)
                        switch value {
                        case let date as Date:
                            appInfo[attribute] = ISO8601DateFormatter().string(from: date)
                        default:
                            appInfo[attribute] = value
                        }
                    }
                }
            }

            seal.fulfill(appInfo)
        }

        RunLoop.current.perform {  // CHANGE 4
            query.start()
        }
    }
}

Is there any newer API that replaces NSMetadataQuery?

No, but keep in mind that "older" doesn't mean bad/broken/wrong. NSMetadataQuery actually works the way most of our APIs work, even newer APIs. More specifically, if an API:

• Provides some kind of delegate/callback mechanism.

• Doesn't require specifically configuring some kind of callback target (typically a dispatch_queue or an operation queue).

• Does allow a target to be specified, but works without that target being specified.

...then that API is in fact using exactly the same runloop pattern NSMetadataQuery uses. The only reason you're running into this at all is that you're creating a command line tool, which means you don't have a "main thread" the way a normal app would. Many developers use this kind of API for years without ever realizing how it actually works.

NOW, NSMetadataQuery actually DOES allow you to provide an alternate callback target through NSMetadataQuery.operationQueue. You certainly could use that approach, but I think it would only confuse things further*.

*The issue here is that your program will end when main returns, so if you provide an alternate callback target, what does your main thread do to avoid returning before the work is done. You could come up with all sorts of creative solutions for that, but they basically all end up blocking the main thread while some other thread "does the work". That's a pointless waste of a thread, just do the work on the main thread... which is what the run loop based approach actually does.

I think that confused me because there's actually a difference between the following two:

Yes. When I wrote that I was specifically focused on the idea that "CFRunLoop" and "NSRunLoop" are both manipulating the same underlying mechanism and hadn't looked closely at the specifics of the API.

One minor correction on this point:

but not between the following two:
CFRunLoopRun()
RunLoop.run(mode:before:)

This exact call:

RunLoop.current.run(mode: .default, before: Date.distantFuture)

is the equivalent to CFRunLoopRun.

The closest equivalent to the "full" RunLoop.run(mode:before:) would be "CFRunLoopRunInMode(::_:)"

However, note that even that isn't an exact equivalent, since that API includes the "returnAfterSourceHandled" option. These differences exist because NSRunLoop wasn't written to replace CFRunLoop, it was written to provide a clean API for using ObjC methods as callbacks. It's "run" methods were simply written to cover the basic/common cases, not to provide a full solution.

I've changed 4 lines from my original code. Each has a numbered // CHANGE comment at the end of the changed line.

First off, I want to be clear that basically "all" the variations of these 4 calls you could write would work fine in your particular case and would, in fact, end up doing EXACLTY the same thing. When the only thread is the main thread, "current runloop" and "main runloop" will always be the same thing, so the final behavior wil be exactly the same. My comment below are about understanding how things operate in a more complex environment when multiple threads are involved.

That code all looks reasonable but, strictly speaking, I would do this:

1. These stay the same and are correct:

CFRunLoopStop(CFRunLoopGetCurrent())  // CHANGE 1
CFRunLoopRun()  // CHANGE 2

1. Change these:

queue: .current  // CHANGE 3
to
queue: .main  // CHANGE 3

RunLoop.current.perform {  // CHANGE 4
to
RunLoop.main.perform {  // CHANGE 4

The difference here is the distinction between "doing work" (#1 & #2) and "scheduling/targeting work" (#3 & #4). In the first case, "doing work", you're avoiding "main" because what you want to have happen is for the current thread to run it's run loop, particularly since that is the ONLY way a run loop can be run.

In the second case, "scheduling/targeting work", what you're actually doing is asking for execution to occur on some specific run loop, so the key here is to ask for what you actually "want". In your case, you're running the main thread runloop, so you want code to run on the main run loop. There are cases where you'd want the code to execute on a different runloop, in which case you could either specify that other thread's runloop or (possibly) use "current".

Again, making sure this is clear, there isn't any functional difference between the different call configurations above. My goal here is to try and clarify the circumstances when you'd what to use "main".

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware