is XPC from app to CMIOExtension possible?

Question

Created May ’22

Replies 19

Boosts 0

Views 7.9k

Participants 11

I built an app which hosts a CMIOExtension. The app works, and it can activate the extension. The extension loads in e.g. Photo Booth and shows the expected video (a white horizontal line which moves down the picture).

I have a couple of questions about this though. The sample Camera Extension is built with a CMIOExtension dictionary with just one entry, CMIOExtensionMachServiceName which is $(TeamIdentifierPrefix)$(PRODUCT_BUNDLE_IDENTIFIER)

This Mach service name won't work though. When attempting to activate the extension, sysextd says that the extensions has an invalid mach service name or is not signed, the value must be prefixed with one of the App Groups in the entitlement. So in order to get the sample extension to activate from my app, I have to change its CMIOExtensionMachServiceName to <my team ID>.com.mycompany.my-app-group.<myextensionname> Is this to be expected?

The template CMIOExtension generates its own video using a timer. My app is intended to capture video from a source, filter that video, then feed it to the CMIOExtension, somehow. The template creates an app group called "$(TeamIdentifierPrefix)com.example.app-group", which suggests that it might be possible to use XPC to send frames from the app to the extension.

However, I've been unable to do so. I've used NSXPCConnection * connection = [[NSXPCConnection alloc] initWithMachServiceName:, using the CMIOExtensionMachServiceName with no options and with the NSXPCConnectionPrivileged option. I've tried NSXPCConnection * connection = [[NSXPCConnection alloc] initWithServiceName: using the extension's bundle identifier. In all cases when I send the first message I get an error in the remote object proxy's handler:

Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service named <whatever name I try> was invalidated: failed at lookup with error 3 - No such process."

According to the "Daemons and Services Programming Guide" an XPC service should have a CFBundlePackageType of XPC!, but a CMIOExtension is of type SYSX. It can't be both.

Does the CMIOExtension loading apparatus cook up a synthetic name for the XPC service, and if so, what is it? If none, how is one expected to get pixel buffers into the camera extension?

Answered by DTS Engineer in 723807022

One of the folks on this thread did open a DTS TSI for this and so I looked into it in depth. This post is a summary of my findings. Note that I’m coming at this from the perspective of someone who supports XPC and system extensions, not video (-:

First up, the CMIOExtensionMachServiceName property is a red herring. While the equivalent properties in other system extensions — like NSEndpointSecurityMachServiceName for an Endpoint Security extension — are intended to be used for XPC communications, that’s not the case here. Rather, CMIOExtensionMachServiceName is part of the machinery used by the system to load your extension. It’s not something you can use for XPC comms.

While it is possible to make outgoing XPC connections from your CMIO extension — see ssmith_c’s comment on this post — our general advice is that you avoid XPC entirely. Rather, divide your IPC into two categories:

For low-bandwidth command and control stuff, use a custom property.
If you need to pass a video stream into your CMIO extension, have the CMIO extension publish an output device.

Both of these are discussed in WWDC 2022 Session 10022 Create camera extensions with Core Media IO.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Boost

Answer 1

kdg82 OP

Aug ’22

If you need to pass a video stream into your CMIO extension, have the CMIO extension publish an output device.

This somewhat confuses me, since in my case CMIO extension is used to create a virtual webcam. And webcam is a video input device, not video output. What if unwanted application decides to pass some video data to my CMIO extension? Is there a way to prevent this? This doesn't seem to be covered in WWDC 2022 Session 10022.

Maybe there are other options for IPC with CMIO extension, besides XPC (not recommended) and publishing an output device?

0

Answer 2

arkulpaK OP

Jan ’23

@ssmith_c we are working on a CMIO camera extension and need to use a XPC service to connect to the extension as we want to only add the device when the parent/configuration app is also running. Further when the parent app is closed the device should be removed again. We tried using a custom property on the class implementing CMIOExtensionProviderSource but could not access that via the CMIO C API. So we need an other way of telling the extension to add the device at runtime thats why we landed on the XPC helper approach.

For this we have setup a XPC helper service as you described in the comment on this reply, but for some reasons we can not get the parent application or the extension to connect to the helper daemon.

Would it be possible for you to provide an example code?

Even if it is not the intended way of doing things I am pretty sure that this would help others as well.

0

Answer 3

cullenp OP

Dec ’23

@eskimo I'm faced with a similar situation where I'm tasked with replacing my XPC interface to a Camera DAL-Plugin with a Camera Extension. I've been using XPC to stream audio and video buffers from a gstreamer pipeline that is processing an rtsp network stream. The pipeline has both an audio branch and a video branch. Audio goes to an Audio HAL driver and video goes to the DAL-Plugin, both via XPC. This mechanism resides in a custom Framework that my app consumes. First and foremost, how do I now continue to stream the video buffers from my external gstreamer client to the Camera Extension? You say "If you need to pass a video stream into your CMIO extension, have the CMIO extension publish an output device." Please clarify this statement - how do I access the CMIO extension in the first place? Is it possible to supply the Extension provider source with my queue? Secondly, I wish to continue to use XPC to supply audio to my audio driver while not incurring any lipsync issues that I'm wondering may arise from having both an Audio HAL and a Camera Extension, with apparently very different architectural flow, running in parallel.

0

Answer 4

ssmith_c OP

Dec ’23

@cullenp you asked "how do I access the CMIO extension in the first place"

From an app, use the routines in CoreMediaIO/CMIOHardwareObject.h to iterate through the CMIO objects, starting at kCMIOObjectSystemObject until you find an object with kCMIODevicePropertyDeviceUID with a value equal to the value of the deviceID parameter used in your initializer of your CMIOExtensionDevice.

You also asked "Is it possible to supply the Extension provider source with my queue? "

I'm not sure what you're asking here. You find the sink stream's ID by querying the sink device you found using the code above. Using that sink stream ID, you can copy its queue using CMIOStreamCopyBufferQueue, and you use CMSimpleQueueEnqueue to put your samples onto the sink stream's queue.

1