Hey everyone, I'm working on an iOS app where I use AVPlayer to play videos, then process them through Metal to apply effects. The app has controls that let users tweak these effects in real-time, and I want the final processed video to be streamed via AirPlay. I use a custom rendering layer that uses a Metal texture to display the processed video on the screen an that works as intended. The problem is, when I try to AirPlay the video after feeding it the processed metal frames, it’s just streaming the original video from AVPlayer, not the version with all the Metal effects. The final processed output is a Metal texture that gets rendered in a MTKView. I even tried capturing that texture and sending it through a new AVPlayer setup, but AirPlay still grabs the original, unprocessed video instead of the final, fully-rendered output. It's also clear that the airplayed video has the full length of the original built in so it's not even that it's 'live streaming' the wrong feed. I need help figuring out how to make AirPlay stream the live, processed video with all the effects, not just the raw video. Any ideas? Happy to share my code if that helps but I'm not sure I have the right underlying approach yet. Thanks!

Hello, I'm developing a RealityKit based app. As part of this, I would like to have a material applied to 3d objects which is essentially contains a texture which is the live camera feed from the arsession. I have the code below which does apply a texture of the camera feed to the box but it essentially only shows the camera snapshot at the time the app loads and doesn't update continuously. I think the issue might be that there is some issue with how the delegate is setup and captureOutput is only called when the app loads instead of every frame. Open to any other approach or insight that gets the job done. Thank you for the help! class CameraTextureViewController: UIViewController { var arView: ARView! var captureSession: AVCaptureSession! var videoOutput: AVCaptureVideoDataOutput! var material: UnlitMaterial? var displayLink: CADisplayLink? var currentPixelBuffer: CVPixelBuffer? var device: MTLDevice! var commandQueue: MTLCommandQueue! var context: CIContext! var textureCache: CVMetalTextureCache! override func viewDidLoad() { super.viewDidLoad() setupARView() setupCaptureSession() setupMetal() setupDisplayLink() } func setupARView() { arView = ARView(frame: view.bounds) arView.autoresizingMask = [.flexibleWidth, .flexibleHeight] view.addSubview(arView) let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = [.horizontal, .vertical] arView.session.run(configuration) arView.session.delegate = self } func setupCaptureSession() { captureSession = AVCaptureSession() captureSession.beginConfiguration() guard let videoDevice = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back), let videoDeviceInput = try? AVCaptureDeviceInput(device: videoDevice), captureSession.canAddInput(videoDeviceInput) else { return } captureSession.addInput(videoDeviceInput) videoOutput = AVCaptureVideoDataOutput() videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "cameraQueue")) videoOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA] guard captureSession.canAddOutput(videoOutput) else { return } captureSession.addOutput(videoOutput) captureSession.commitConfiguration() DispatchQueue.global(qos: .userInitiated).async { [weak self] in self?.captureSession.startRunning() } } func setupMetal() { device = MTLCreateSystemDefaultDevice() commandQueue = device.makeCommandQueue() context = CIContext(mtlDevice: device) CVMetalTextureCacheCreate(nil, nil, device, nil, &textureCache) } func setupDisplayLink() { displayLink = CADisplayLink(target: self, selector: #selector(updateFrame)) displayLink?.preferredFrameRateRange = CAFrameRateRange(minimum: 60, maximum: 60, preferred: 60) displayLink?.add(to: .main, forMode: .default) } @objc func updateFrame() { guard let pixelBuffer = currentPixelBuffer else { return } updateMaterial(with: pixelBuffer) } func updateMaterial(with pixelBuffer: CVPixelBuffer) { let ciImage = CIImage(cvPixelBuffer: pixelBuffer) var tempPixelBuffer: CVPixelBuffer? let attrs = [ kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue, kCVPixelBufferMetalCompatibilityKey: kCFBooleanTrue ] as CFDictionary CVPixelBufferCreate(kCFAllocatorDefault, CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer), kCVPixelFormatType_32BGRA, attrs, &tempPixelBuffer) guard let tempPixelBuffer = tempPixelBuffer else { return } context.render(ciImage, to: tempPixelBuffer) var textureRef: CVMetalTexture? let width = CVPixelBufferGetWidth(tempPixelBuffer) let height = CVPixelBufferGetHeight(tempPixelBuffer) CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, tempPixelBuffer, nil, .bgra8Unorm, width, height, 0, &textureRef) guard let metalTexture = CVMetalTextureGetTexture(textureRef!) else { return } let ciImageFromTexture = CIImage(mtlTexture: metalTexture, options: nil)! guard let cgImage = context.createCGImage(ciImageFromTexture, from: ciImageFromTexture.extent) else { return } guard let textureResource = try? TextureResource.generate(from: cgImage, options: .init(semantic: .color)) else { return } if material == nil { material = UnlitMaterial() } material?.baseColor = .texture(textureResource) guard let modelEntity = arView.scene.anchors.first?.children.first as? ModelEntity else { let mesh = MeshResource.generateBox(size: 0.2) let modelEntity = ModelEntity(mesh: mesh, materials: [material!]) let anchor = AnchorEntity(world: [0, 0, -0.5]) anchor.addChild(modelEntity) arView.scene.anchors.append(anchor) return } modelEntity.model?.materials = [material!] } } extension CameraTextureViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } currentPixelBuffer = pixelBuffer } } extension CameraTextureViewController: ARSessionDelegate { func session(_ session: ARSession, didUpdate frame: ARFrame) { // Handle AR frame updates if necessary } }

Hi I'm working on a project that uses RealityKit including the placement of 3d objects. However, I want to be able to run the background camera through Metal post-processing before being rendered but haven't been able to find a working approach. I'm open to it rendering directly into the ARview or a separate MTKview or swiftui layer. I've tried using the default xcode project of an Augmented Reality App with Metal Content. However it seems to use a 1.33 aspect camera by default instead of the iphone 15s standard ratio which works by default when I use the regular realitykit pathway and doesnt seem to have the proper ratio available as an option Open to any approach that gets the job done here. Thank you, Any direction would be

Hello As part of my app, I am using Metal shaders on CustomMaterials created and managed using RealityKit. Using the ECS approach, I have a Shader system that iterates through all my materials every frame and passes a SIMD4 of variables (that I can manage on the swift side) that can be interpreted and used every frame on the Metal side to influence elements of the shader. This does work as intended but is limited to just 4 variables when I need more for my use case. I've experimented with trying multiple simd4 or other approaches for passing these to metal and be useable but I haven't had very much luck. I was hoping for some recommendations on the best scalable approach. Swift: class ShaderSystem: System { static let query = EntityQuery(where: .has(ModelComponent.self)) private var startTime: Date required init(scene: Scene) { startTime = Date() } func update(context: SceneUpdateContext) { let audioLevel = AudioSessionManager.shared.audioLevel let elapsedTime = Float(Date().timeIntervalSince(startTime)) guard let sceneType = SceneManager.shared.currentScenes.keys.first else { return } let sceneTime = SceneComposer.shared.getSceneTime(for: sceneType) let multiplier = ControlManager.shared.getControlValue(parameterName: "elapsedTimeMultiplier") ?? 1.0 for entity in context.scene.performQuery(Self.query) { guard var modelComponent = entity.components[ModelComponent.self] as? ModelComponent else { continue } modelComponent.materials = modelComponent.materials.map { material in guard var customMaterial = material as? CustomMaterial else { return material } // Passing audioLevel, elapsedTime, sceneTime, and multiplier customMaterial.custom.value = SIMD4<Float>(audioLevel, elapsedTime, sceneTime, multiplier) return customMaterial } entity.components[ModelComponent.self] = modelComponent } } } metal: struct CustomMaterialUniforms { float4 custom; }; [[visible]] void fractalShader(realitykit::surface_parameters params) { auto uniforms = params.uniforms(); float4 customValues = uniforms.custom_parameter(); float audioLevel = customValues.x; .... Thank you for the assistance