Camera Extension CMIOExtensionDevice update CMIOExtensionStreams in run time

Background:

Our CMIOExtensionDevice may change its supported formats in run time when video source changes. (i.e. set of resolutions change). A custom callback is triggered once this happens.

According to Apple doc CMIOExtensionStreamSource … “formats should not change during the life cycle of the associated stream” . So we can’t update formats of existing CMIOExtensionStreamSource once it has been created.

Initially, we thought stopping the stream -> removing the stream-> adding new stream -> starting the new stream would accomplish this.

CMIOExtensionStreamSource::stopStreamAndReturnError
CMIOExtensionDevice::removeStream
CMIOExtensionDevice::addStream
CMIOExtensionStreamSource::startStreamAndReturnError 

but we ran into strange CMIO behaviors. In the os_log we could see error messages like

CMIOExtensionProvider.m:3085:-[CMIOExtensionProvider stopStreamForClientID:streamID:reply:]_block_invoke Invalid streamID
CMIOExtensionProviderContext.m:887:-[CMIOExtensionProviderContext stopStream:message:]_block_invoke stopStream failed Error Domain=NSOSStatusErrorDomain Code=-50 "paramErr: error in user parameter list" UserInfo={NSLocalizedDescription=Invalid streamID}
CMIOExtensionSession.m:400:-[CMIOExtensionSessionStream stopStream:]_block_invoke stop stream index error Error Domain=NSOSStatusErrorDomain Code=-50 "paramErr: error in user parameter list"
CMIO_DAL_CMIOExtension_Stream.mm:1981:Stop_block_invoke 37 unexpected Error Domain=NSOSStatusErrorDomain Code=-50 "paramErr: error in user parameter list"
CMIO_DAL_CMIOExtension_Device.mm:1565:StopStream 36 could not find stream with ID 37

It seemed like CMIO or AVFoundation was not made aware of such add/remove change.

On a different approach, it would work if in the CMIOExtensionProvider level we remove the old device and a create new device + stream with updated list of formats. However, such action will cause the client app to move to next available AVCaptureDevice (i.e FaceTime Camera) to preview, leading to a strange user experience.

DTS engineer recommendations:

I see, I agree that this would create a strange user experience, that is, reporting support for a format but not actually having hardware connected that supports that format.

Unfortunately there is no other option, you can provide the array of all possible formats, and then maintain a dynamic sub-set of these formats that you actually support at a given time. Then, if a format index is set (by whatever client app is using the device) that isn’t supported at a particular time, you could throw an error from setStreamProperties. Or you can tear everything down and create a new device + stream with the currently supported formats.

Beyond those options, I recommend that you file an enhancement request using Feedback Assistant, to explain to the engineering team your need for a way to update the formats array during the life cycle of the associated stream.

We filed an enhancement request a while ago in Feedback Assistant. Thought I might share this in the forum.

You don't need to stop and start the stream at all. If the only thing that changed is the resolution, just go ahead and send the CMSampleBuffers with the new resolution to the CMIOExtensionStream. Client apps should handle this by reading resolution from the CMSampleBuffers they receive, not from the stream properties. Apple's apps (QuickTime, FaceTime) handle resolution changes just fine.

Camera Extension CMIOExtensionDevice update CMIOExtensionStreams in run time
 
 
Q