How to display a RealityKit Perspective Camera View in a visionOS SwiftUI 2D window?

I am developing an immersive visionOS app based on RealityKit and SwiftUI. This app has ModelEntities that have a PerspectiveCamera entity as child. I want to display the camera view in a 2D window in visionOS.

I am creating the camera, and add it to the entity with

let cameraEntity = PerspectiveCamera()
cameraEntity.camera.far = 10000
cameraEntity.camera.fieldOfViewInDegrees = 60
cameraEntity.camera.near = 0.01
entity.addChild(cameraEntity)

My app is not AR. The immersive view is programmatically generated. In iOS, I could use an ARView with non AR camera mode. However, ARView is not available in visionOS.

How can I show the camera view in a SwiftUI 2D window in the immersive space?

Answered by DTS Engineer in 801164022

Hey @Reinhard_Maenner,

"""

  • I cannot imagine that one can define a camera without a way to get access to its output. But if this is currently really the case, please confirm it, and I will write a bug report.

"""

That is in fact the case, please file an enhancement request using Feedback Assistant to request the functionality you are looking for.

"""

  • What is an existing Metal workflow (I don't have experience in Metal programming).

"""

RealityRenderer renders its scene to a MTLTexture. You can use that MTLTexture in an "existing Metal workflow", whether that be to use as a source for a DrawableQueue or a LowLevelTexture in RealityKit, or as the presentation source for a CAMetalLayer.

"""

  • How do I set up the RealityRenderer

"""

Here is a minimal code example I had on hand:

@Observable
@MainActor
final class OffscreenRenderModel {
        
    private let renderer: RealityRenderer
    
    private let colorTexture: MTLTexture
        
    init(scene: Entity) throws {
        renderer = try RealityRenderer()
        
        renderer.entities.append(scene)
        
        let camera = PerspectiveCamera()
        renderer.activeCamera = camera
        renderer.entities.append(camera)
        
        let textureDesc = MTLTextureDescriptor()
        textureDesc.pixelFormat = .rgba8Unorm
        textureDesc.width = 512
        textureDesc.height = 512
        textureDesc.usage = [.renderTarget, .shaderRead]
        
        let device = MTLCreateSystemDefaultDevice()!
        colorTexture = device.makeTexture(descriptor: textureDesc)!
    }
    
    func render() throws {
        
        let cameraOutputDesc = RealityRenderer.CameraOutput.Descriptor.singleProjection(colorTexture: colorTexture)
        
        let cameraOutput = try RealityRenderer.CameraOutput(cameraOutputDesc)

        try renderer.updateAndRender(deltaTime: 0.1, cameraOutput: cameraOutput, onComplete: { renderer in
            
            guard let colorTexture = cameraOutput.colorTextures.first else { fatalError() }
            
            // The colorTexture holds the rendered scene.
        })
    }
}

""" how do I use its output to display it in a 2D SwiftUI window in a immersive space? """ You can use the colorTexture in conjunction with a CAMetalLayer.

Lastly, I will say that, given you stated you are new to Metal, I strongly recommend that you start by reading over some of the documentation before you dive into this implementation, specifically the Essentials, GPU Devices, Resources, and Presentation sections.

Hello @reinermaenner,

I want to display the camera view in a 2D window in visionOS.

I think what you are describing might best align with the capabilities of PortalComponent. Take a look at that, and if it doesn't do quite what you want, please come back to describe what you are after in further detail.

Best regards,

Greg

Hi Greg, thanks for your quick replay, and the interesting hint to PortalComponent that I was not aware of. Unfortunately, this does not solve my problem. Sorry that I did not explain it better. Let me try again:

I am developing a visionOS app that is kind of a game. It uses an immersive space in which a board is shown with entities on it. These entities are primitive beings that move on the board according to simple rules. The user sees this boards with the beings in the immersive space.

This board is shown using a RealityView with attachments. An attachment is shown after the user selects one of the beings by a SpatialTapGesture. The attachment shows then infos for the selected being in a 2D window.

What I want to show in this 2D window is how the board with the beings looks from the perspective of the selected being. Therefore, this being has to have a camera attached to it.

In RealityKit, I can create an Entity of type PerspectiveCamera, and attach it to the being as child. PerspectiveCamera is available in VisionOS 1.0+.

However, I did not find a way to access the video stream or the image of this camera, and to display it in the 2D window of the attachment.

In iOS this could be realized by using an ARView with non AR camera mode. However, ARView is not available in visionOS.

So, how could I access the video stream or the image of the PerspectiveCamera in visionOS?

Hey @reinermaenner,

Ok I see, thanks for providing that additional detail!

PerspectiveCamera requires an ARView in nonAR mode to operate, but as you've noticed ARView is not available on visionOS.

Instead of that approach, you could try making use of RealityRenderer. This would enable you to render your scene from a particular viewpoint to a texture, but note that that render will not include the passthrough camera, or content in other windows.

Best regards,

Greg

Hi Greg, thanks for this suggestion, but I have 2 questions:

  1. Since visionOS has a PerspectiveCamera (documented in visionOS 1.0+), how can one access the output of this camera? I cannot imagine that one can define a camera without a way to get access to its output. But if this is currently really the case, please confirm it, and I will write a bug report. My guess is that visionOS should make ARView not completely unavailable; it should allow it only with camera mode non AR.
  2. I looked up the docu for RealityRenderer. It says it is "A renderer that displays a RealityKit scene in an existing Metal workflow." What is an existing Metal workflow (I don't have experience in Metal programming). How do I set up the RealityRenderer, and how do I use its output to display it in a 2D SwiftUI window in a immersive space? The current docu is nearly non-existent. Is there an Apple suggestion how to use it?
Accepted Answer

Hey @Reinhard_Maenner,

"""

  • I cannot imagine that one can define a camera without a way to get access to its output. But if this is currently really the case, please confirm it, and I will write a bug report.

"""

That is in fact the case, please file an enhancement request using Feedback Assistant to request the functionality you are looking for.

"""

  • What is an existing Metal workflow (I don't have experience in Metal programming).

"""

RealityRenderer renders its scene to a MTLTexture. You can use that MTLTexture in an "existing Metal workflow", whether that be to use as a source for a DrawableQueue or a LowLevelTexture in RealityKit, or as the presentation source for a CAMetalLayer.

"""

  • How do I set up the RealityRenderer

"""

Here is a minimal code example I had on hand:

@Observable
@MainActor
final class OffscreenRenderModel {
        
    private let renderer: RealityRenderer
    
    private let colorTexture: MTLTexture
        
    init(scene: Entity) throws {
        renderer = try RealityRenderer()
        
        renderer.entities.append(scene)
        
        let camera = PerspectiveCamera()
        renderer.activeCamera = camera
        renderer.entities.append(camera)
        
        let textureDesc = MTLTextureDescriptor()
        textureDesc.pixelFormat = .rgba8Unorm
        textureDesc.width = 512
        textureDesc.height = 512
        textureDesc.usage = [.renderTarget, .shaderRead]
        
        let device = MTLCreateSystemDefaultDevice()!
        colorTexture = device.makeTexture(descriptor: textureDesc)!
    }
    
    func render() throws {
        
        let cameraOutputDesc = RealityRenderer.CameraOutput.Descriptor.singleProjection(colorTexture: colorTexture)
        
        let cameraOutput = try RealityRenderer.CameraOutput(cameraOutputDesc)

        try renderer.updateAndRender(deltaTime: 0.1, cameraOutput: cameraOutput, onComplete: { renderer in
            
            guard let colorTexture = cameraOutput.colorTextures.first else { fatalError() }
            
            // The colorTexture holds the rendered scene.
        })
    }
}

""" how do I use its output to display it in a 2D SwiftUI window in a immersive space? """ You can use the colorTexture in conjunction with a CAMetalLayer.

Lastly, I will say that, given you stated you are new to Metal, I strongly recommend that you start by reading over some of the documentation before you dive into this implementation, specifically the Essentials, GPU Devices, Resources, and Presentation sections.

How to display a RealityKit Perspective Camera View in a visionOS SwiftUI 2D window?
 
 
Q