is XPC from app to CMIOExtension possible?

Is this to be expected?

I’m not really up to speed on CMIO stuff but this doesn’t surprise me. That’s a pattern I see in similar contexts.

According to the Daemons and Services Programming Guide an XPC service should have a CFBundlePackageType of XPC!, but a CMIOExtension is of type SYSX. It can't be both.

That doc is in the archive and hasn’t been updated to account for system extensions. Having said that there’s an ongoing terminological confusion when it comes to the term XPC services. Some folks use it to mean A small program embedded in an app, as defined in the xpcservice.plist man page. Other folks use it to mean An XPC listener with a published name. This name might be published by an XPC service, a launchd daemon or agent (via the MachServices property), a Service Management login item, a system extension, and so on.

I previously used XPC Service for the former and XPC service for the latter, but that’s way too subtle. I now use XPC service for the former and named XPC listener for the latter.

*phew*

As to your connection error, two things:

• Make sure your app isn’t sandboxed. There are ways to do this in a sandboxed app but during the bring up it’s best to just avoid the sandbox.

  If your main app is sandboxed, create a tiny test app just for this bring up.

• As to your service name, use the following to see what actually got published in the endpoints list:

  % sudo launchctl print system
  

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

There is no service published containing my company ID.

Bummer.

I’m not sure what’s going on here. This issue stands at the intersection of XPC and CMIO and my knowledge of the latter is rather limited. My advice is that you open a DTS tech support incident and I’ll work with DTS’s CMIO specialist to get to the bottom of this.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

We're having the same problem converting our virtual camera to an extension - nothing we tried works as a way to communicate from the host application to the extension. I would assume that a full sample would be coming along at some point, but it's frustrating to get the extension template "working" quickly, then have no way to go from there to something useful.

I'm not sure I'm doing it right.

That does sound very convoluted. I’m hoping to find a simpler way.

Someone [1] opened a TSI for this and I’ll post back here with the conclusion.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] For privacy reasons I don’t have an easy way for me to map DevForums names to real names.

I'd also like this forum to respect my line breaks in comments.

I recommend that you only use comments for… well… comments. Sadly, the current DevForums UI misleads folks into using them for replies. We plan to fix that.

As to how you transfer I/O Surface values over XPC, there are two ways to do that:

• Directly

• Using secure coding

The direct case is easier to explain so let’s start with that. Imagine this XPC protocol:

@objc
private protocol MainDirectSurfaceTransfer {
    func transfer(surface: xpc_object_t, reply: @escaping (_ error: NSError?) -> Void)
}

The trick is to configure your NSXPCInterface so that XPC knows what XPC object type to expect for the surface parameter. Here’s how to do that:

func surfaceTransferInterface() -> NSXPCInterface {
    let interface = NSXPCInterface(with: MainDirectSurfaceTransfer.self)
    
    // Create a dummy surface so that we can get its XPC object type.
    
    let dummy = MainDirect.makeSurface()!
    let dummyXPC = IOSurfaceCreateXPCObject(dummy)
    let type = xpc_get_type(dummyXPC)
    interface.setXPCType(type, for: #selector(MainDirectSurfaceTransfer.transfer(surface:reply:)), argumentIndex: 0, ofReply: false)

    return interface
}

Annoyingly, you have to create a dummy surface to get its XPC type because there’s no way to get it directly.

On the send side you call IOSurfaceCreateXPCObject to get the XPC surface from your object. And on the receive side you call IOSurfaceLookupFromXPCObject to do the reverse.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I'll type this in the answer box then.

If you can require 10.12 or later, it is much simpler than this. As I found out, an IOSurfaceRef cannot be directly transported over NSXPCConnection, but an IOSurface can. No need to do the CreateXPCObject or LookupFromXPCObject stuff unless you're on 10.7 to 10.11. The two types are toll-free bridged. The only thing I'm not sure about is which flavor of __bridge to use at each end.

I'm thinking I need write

IOSurfaceRef surface = CVPixelBufferGetIOSurface(pixelBuffer);
[remoteObjectProxy sendIOSurfaceToExtension:(__bridge_transfer IOSurface *)surface]; 

and at the receiving end

CVPixelBufferRef pixelBuffer;
CFPixelbufferCreateWithIOSurface(kCFAllocatorDefault, (__bridge_retained IOSurfaceRef)surface,...);

I guess I'll find out...

As far as I can tell, CMIO system extensions run under the system user _cmiodalassistants, which does expose the XPC services correctly. You can verify this by checking the exposed endpoints for that user:

sudo launchctl print user/262

The problem, it seems, is that the user session (which launches the main application) does not have access to the endpoints exposed by _cmiodalassistants, resulting in the error you are seeing.

CMIO system extensions run under the system user _cmiodalassistants

Yes.

which does expose the XPC services correctly

Well “correctly” is a matter of perspective (-: As I mentioned above, a developer opened a TSI about this issue. I’m actively discussing this with the CMIO team and I’ll post back here once I have a definitive answer.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

As we learned in the "Create camera extensions with CoreMedia IO" presentation at WWDC 2022, a Camera Extension can present a sink interface to the system, accessible to an app via the CMIOHardwareObject.h and CMIOHardwareStream.h APIs. The Camera Extension template only offers a source stream, but it is pretty easy to add a sink stream.

That's one way to get video into a Camera Extension, it works on 12.3 and it doesn't require a helper daemon to enable the app and extension to find one another.

In Ventura, the plan is to have the app's extension be owned by the app's owner, so XPC would be an option.

Easier again is to have the extension itself handle the video sourcing, while the accompanying app only provides control and status reporting, using the property interfaces of the extension. That's the intent behind the design.

it would be really really helpful to have official sample code that shows how the camera extension can receive frame data from the main app in which it was embedded.

Also, it’d be really useful if the camera extension could use some window server APIs such as CGWindowListCreateImage or the newer ScreenCaptureKit API: assuming the rights to access the screen were granted to the bundling app, could the camera extension inherit that right?

that’s the approch I’m using currently in my old DAL plugin for Screegle app to capture frames from an NSWindow that the main app has.

@smith_c is correct: we can add a sink stream to the same device and the app can feed CMSampleBuffers to this sink stream by connecting to it using the CoreMediaIO C API. In my code, the cameraStreamSink (modeled after cameraStreamSource) simply consumes buffers by waiting on self._streamSink.stream.consumeSampleBuffer(buf,...) and sends them immediately to the source stream by calling self._streamSource.stream.send(buf!,...)

I can provide source code if needed.

@ldeoue, I think an example of sending the frames from the app to the extension's sink stream would be very helpful.

Here’s a sample project showing how a camera extension can have a source and a sink streams, and how to feed the sink stream with sample buffers from the main app.

https://github.com/ldenoue/cameraextension

So, has anybody found any solution to the XPC connection problem? I seem to be facing the same issue.

One of the folks on this thread did open a DTS TSI for this and so I looked into it in depth. This post is a summary of my findings. Note that I’m coming at this from the perspective of someone who supports XPC and system extensions, not video (-:

First up, the CMIOExtensionMachServiceName property is a red herring. While the equivalent properties in other system extensions — like NSEndpointSecurityMachServiceName for an Endpoint Security extension — are intended to be used for XPC communications, that’s not the case here. Rather, CMIOExtensionMachServiceName is part of the machinery used by the system to load your extension. It’s not something you can use for XPC comms.

While it is possible to make outgoing XPC connections from your CMIO extension — see ssmith_c’s comment on this post — our general advice is that you avoid XPC entirely. Rather, divide your IPC into two categories:

• For low-bandwidth command and control stuff, use a custom property.

• If you need to pass a video stream into your CMIO extension, have the CMIO extension publish an output device.

Both of these are discussed in WWDC 2022 Session 10022 Create camera extensions with Core Media IO.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

If you need to pass a video stream into your CMIO extension, have the CMIO extension publish an output device.

This somewhat confuses me, since in my case CMIO extension is used to create a virtual webcam. And webcam is a video input device, not video output. What if unwanted application decides to pass some video data to my CMIO extension? Is there a way to prevent this? This doesn't seem to be covered in WWDC 2022 Session 10022.

Maybe there are other options for IPC with CMIO extension, besides XPC (not recommended) and publishing an output device?

@ssmith_c we are working on a CMIO camera extension and need to use a XPC service to connect to the extension as we want to only add the device when the parent/configuration app is also running. Further when the parent app is closed the device should be removed again. We tried using a custom property on the class implementing CMIOExtensionProviderSource but could not access that via the CMIO C API. So we need an other way of telling the extension to add the device at runtime thats why we landed on the XPC helper approach.

For this we have setup a XPC helper service as you described in the comment on this reply, but for some reasons we can not get the parent application or the extension to connect to the helper daemon.

Would it be possible for you to provide an example code?

Even if it is not the intended way of doing things I am pretty sure that this would help others as well.

@eskimo I'm faced with a similar situation where I'm tasked with replacing my XPC interface to a Camera DAL-Plugin with a Camera Extension. I've been using XPC to stream audio and video buffers from a gstreamer pipeline that is processing an rtsp network stream. The pipeline has both an audio branch and a video branch. Audio goes to an Audio HAL driver and video goes to the DAL-Plugin, both via XPC. This mechanism resides in a custom Framework that my app consumes. First and foremost, how do I now continue to stream the video buffers from my external gstreamer client to the Camera Extension? You say "If you need to pass a video stream into your CMIO extension, have the CMIO extension publish an output device." Please clarify this statement - how do I access the CMIO extension in the first place? Is it possible to supply the Extension provider source with my queue? Secondly, I wish to continue to use XPC to supply audio to my audio driver while not incurring any lipsync issues that I'm wondering may arise from having both an Audio HAL and a Camera Extension, with apparently very different architectural flow, running in parallel.

@cullenp you asked "how do I access the CMIO extension in the first place"

From an app, use the routines in CoreMediaIO/CMIOHardwareObject.h to iterate through the CMIO objects, starting at kCMIOObjectSystemObject until you find an object with kCMIODevicePropertyDeviceUID with a value equal to the value of the deviceID parameter used in your initializer of your CMIOExtensionDevice.

You also asked "Is it possible to supply the Extension provider source with my queue? "

I'm not sure what you're asking here. You find the sink stream's ID by querying the sink device you found using the code above. Using that sink stream ID, you can copy its queue using CMIOStreamCopyBufferQueue, and you use CMSimpleQueueEnqueue to put your samples onto the sink stream's queue.