@drxibber and @amallik and @MarkBessey I posted a full example of how to declare a source and sink streams, and how to feed the sink sample buffers from the main app.
https://github.com/ldenoue/cameraextension
ldenoue
Users receive reputation points by contributing quality content to the forums.
Posts include post and replies published on the forums.
Post authors can mark replies as Accepted to indicate successful solutions.
Apple Developer Forums admins can mark replies as Apple Recommended to indicate an approved solution
Last seen
User for
Post
Replies
Boosts
Views
Activity
Here’s a sample project showing how a camera extension can have a source and a sink streams, and how to feed the sink stream with sample buffers from the main app.
https://github.com/ldenoue/cameraextension
I use a custom property for this:
let CMIOExtensionPropertyCustomPropertyData_just: CMIOExtensionProperty = CMIOExtensionProperty(rawValue: "4cc_just_glob_0000")
In you cameraDeviceSource class:
func myStreamingCounter() -> String {
return "sc=\(_streamingCounter)"
}
In your cameraStreamSource class:
public var just: String = "toto"
func streamProperties(forProperties properties: Set<CMIOExtensionProperty>) throws -> CMIOExtensionStreamProperties {
let streamProperties = CMIOExtensionStreamProperties(dictionary: [:])
if properties.contains(.streamActiveFormatIndex) {
streamProperties.activeFormatIndex = 0
}
if properties.contains(.streamFrameDuration) {
let frameDuration = CMTime(value: 1, timescale: Int32(kFrameRate))
streamProperties.frameDuration = frameDuration
}
if properties.contains(CMIOExtensionPropertyCustomPropertyData_just) {
streamProperties.setPropertyState(CMIOExtensionPropertyState(value: self.just as NSString), forProperty: CMIOExtensionPropertyCustomPropertyData_just)
}
return streamProperties
}
func setStreamProperties(_ streamProperties: CMIOExtensionStreamProperties) throws {
if let activeFormatIndex = streamProperties.activeFormatIndex {
self.activeFormatIndex = activeFormatIndex
}
if let state = streamProperties.propertiesDictionary[CMIOExtensionPropertyCustomPropertyData_just] {
if let newValue = state.value as? String {
self.just = newValue
if let deviceSource = device.source as? cameraDeviceSource {
self.just = deviceSource.myStreamingCounter()
}
}
}
Then in your main app, 2 functions:
func getJustProperty(streamId: CMIOStreamID) -> String? {
let selector = FourCharCode("just")
var address = CMIOObjectPropertyAddress(selector, .global, .main)
let exists = CMIOObjectHasProperty(streamId, &address)
if exists {
var dataSize: UInt32 = 0
var dataUsed: UInt32 = 0
CMIOObjectGetPropertyDataSize(streamId, &address, 0, nil, &dataSize)
var name: CFString = "" as NSString
CMIOObjectGetPropertyData(streamId, &address, 0, nil, dataSize, &dataUsed, &name);
return name as String
} else {
return nil
}
}
func setJustProperty(streamId: CMIOStreamID, newValue: String) {
let selector = FourCharCode("just")
var address = CMIOObjectPropertyAddress(selector, .global, .main)
let exists = CMIOObjectHasProperty(streamId, &address)
if exists {
var settable: DarwinBoolean = false
CMIOObjectIsPropertySettable(streamId,&address,&settable)
if settable == false {
return
}
var dataSize: UInt32 = 0
CMIOObjectGetPropertyDataSize(streamId, &address, 0, nil, &dataSize)
var newName: CFString = newValue as NSString
CMIOObjectSetPropertyData(streamId, &address, 0, nil, dataSize, &newName)
}
}
Finally, poll at regular intervals the value of this property: notice that you FIRST need to write to it and then read it:
@objc func veryslowTimer() {
if let screegleStream = screegleStream {
self.setJustProperty(streamId: screegleStream, newValue: "\(count)")
let just = self.getJustProperty(streamId: screegleStream)
if let just = just {
if just == "sc=1" {
needToStream = true
} else {
needToStream = false
}
}
}
}
And this FourCharCode extension:
extension FourCharCode: ExpressibleByStringLiteral {
public init(stringLiteral value: StringLiteralType) {
var code: FourCharCode = 0
// Value has to consist of 4 printable ASCII characters, e.g. '420v'.
// Note: This implementation does not enforce printable range (32-126)
if value.count == 4 && value.utf8.count == 4 {
for byte in value.utf8 {
code = code << 8 + FourCharCode(byte)
}
}
else {
print("FourCharCode: Can't initialize with '\(value)', only printable ASCII allowed. Setting to '????'.")
code = 0x3F3F3F3F // = '????'
}
self = code
}
public init(extendedGraphemeClusterLiteral value: String) {
self = FourCharCode(stringLiteral: value)
}
public init(unicodeScalarLiteral value: String) {
self = FourCharCode(stringLiteral: value)
}
public init(_ value: String) {
self = FourCharCode(stringLiteral: value)
}
public var string: String? {
let cString: [CChar] = [
CChar(self >> 24 & 0xFF),
CChar(self >> 16 & 0xFF),
CChar(self >> 8 & 0xFF),
CChar(self & 0xFF),
0
]
return String(cString: cString)
}
}
@smith_c is correct: we can add a sink stream to the same device and the app can feed CMSampleBuffers to this sink stream by connecting to it using the CoreMediaIO C API.
In my code, the cameraStreamSink (modeled after cameraStreamSource) simply consumes buffers by waiting on self._streamSink.stream.consumeSampleBuffer(buf,...) and sends them immediately to the source stream by calling self._streamSource.stream.send(buf!,...)
I can provide source code if needed.
Replying to my own question with code to declare custom string and data properties on the camera extension, and getting/setting them on the app side in Swift.
https://gist.github.com/ldenoue/84210280853f0490c79473b6edd25e9d
it would be really really helpful to have official sample code that shows how the camera extension can receive frame data from the main app in which it was embedded.
Also, it’d be really useful if the camera extension could use some window server APIs such as CGWindowListCreateImage or the newer ScreenCaptureKit API: assuming the rights to access the screen were granted to the bundling app, could the camera extension inherit that right?
that’s the approch I’m using currently in my old DAL plugin for Screegle app to capture frames from an NSWindow that the main app has.
In Signing & Capabilities, in the App Sandbox section => Network, tick the checkbox next to "Outgoing Connections (Client)"