RealityKit VideoMaterial not respecting compositing instructions

Hello,

Im trying to programmatically alter a a video frame before applying it to geometry using VideoMaterial. What I'm finding is that the output appears as though no videoCompositor was applied to the playerItem. Is this expected behavior?

Please file a bug report for this issue using Feedback Assistant.

Is there a work around besides using an ExportSession to bounce the movie to disk?

Yes, there are options that do not involve writing back to disk, though admittedly they are complicated. For example, you could use a DrawableQueue instead of VideoMaterial, and update the drawable with your altered video frames during playback.

Yes, there are options that do not involve writing back to disk, though admittedly they are complicated. For example, you could use a DrawableQueue instead of VideoMaterial, and update the drawable with your altered video frames during playback.

Ok, I can look into that... is there anyway to preserve 3d stereoscopic content when doing this?

Ok, I can look into that... is there anyway to preserve 3d stereoscopic content when doing this?

There is, you could utilize a ShaderGraphMaterial with a Camera Index Switch node to render left and right stereo content.

I also recommend that you file a separate enhancement request for sample code that demonstrates this approach using Feedback Assistant.

Thanks, I've submitted FB13615798 and FB13615765

Ok, so a couple nights of trial and error I have both ends of this implemented. I can get a video drawing to a material via DrawableQueue, and I can display different imageFiles to each eye in a shaderGraphMaterial. I can't figure out how to glue the two bits together though... I changed the image files in the shader into color inputs, and then at load time I try to:

if var material = cube.model?.materials.first as? ShaderGraphMaterial{
    do{
        try material.setParameter(name: "left", value:.textureResource(drawableTextureManager.textureResource) ) // <-this throws a ShaderGraphMaterial.Error.incorrectTypeForParameterName
        //try material.setParameter(name: "left", value:.color(CGColor(red: 1.0, green: 0.5, blue: 0.0, alpha: 1.0)) ) // <-this works
    } catch {
        print("oops")
    }
    cube.model?.materials = [material]                        
}

So I guess my questions are:

• are color inputs the correct way to get textures into a ShaderGraphMaterial?
• who do I setParameter with a texture/TextureResource/DrawableQueue

Thanks!

Hey [@mike woodworth](https://developer.apple.com/forums/profile/mike woodworth),

First of all, kudos to you for implementing both ends, this is definitely a challenging task!

are color inputs the correct way to get textures into a ShaderGraphMaterial?

You actually will want to be using an "image file" input, not a color input.

who do I setParameter with a texture/TextureResource/DrawableQueue

Your setParameter code looks fine actually, it should stop throwing once you change your input to an "image file" input in your shader graph.

Thanks, I got everything working great!

Hi Mike, could you describe how you set up the GeometrySwitchCameraIndex node in the ShaderGraph editor? I'm having trouble rendering different images for each eye. Much appreciated!

Hello Mike. How do you update the drawableTextureManager.textureResource or DrawableQueue using video frames? Would you mind share your snippet? Thanks