RealityView in macOS, Skybox, and lighting issue

I am testing RealityView on a Mac, and I am having troubles controlling the lighting.

I initially add a red cube, and everything is fine. (see figure 1)

I then activate a skybox with a star field, the star field appears, and then the red cube is only lit by the star field.

Then I deactivate the skybox expecting the original lighting to return, but the cube continues to be lit by the skybox. The background is no longer showing the skybox, but the cube is never lit like it originally was.

Is there a way to return the lighting of the model to the original lighting I had before adding the skybox?

I seem to recall ARView's environment property had both a lighting.resource and a background, but I don't see both of those properties in RealityViewCameraContent's environment.

Sample code for 15.1 Beta (24B5024e), Xcode 16.0 beta (16A5171c)

struct MyRealityView: View {
    
    @Binding var isSwitchOn: Bool
    @State private var blueNebulaSkyboxResource: EnvironmentResource?
    
    var body: some View {
            
            RealityView { content in
                
                // Create a red cube 10cm on a side
                let mesh = MeshResource.generateBox(size: 0.1)
                let simpleMaterial = SimpleMaterial(color: .red, isMetallic: false)
                let model = ModelComponent(
                    mesh: mesh,
                    materials: [simpleMaterial]
                )
                let redBoxEntity = Entity()
                redBoxEntity.components.set(model)
                content.add(redBoxEntity)
                
                // Load skybox
                let blueNeb2Name = "BlueNeb2"
                blueNebulaSkyboxResource = try? await EnvironmentResource(named: blueNeb2Name)
            }
            update: { content in
                if (blueNebulaSkyboxResource != nil) && (isSwitchOn == true) {
                    content.environment = .skybox(blueNebulaSkyboxResource!)
                }
                else {
                    content.environment = .default
                }
            }
            .realityViewCameraControls(CameraControls.orbit)
    }
}

Figure 1 (default lighting before adding the skybox):

Figure 2 (after activating skybox with star field; cube is lit by / reflects skybox):

Figure 3 (removing skybox by setting content.environment to .default, cube still reflects skybox; it is hard to see):

Answered by Todd2 in 800487022

I have found a different approach that works for me. I have abandoned setting the content.environment property and use a sky dome model for the background and use ImageBasedLightComponent and ImageBasedLightReceiverComponent to choose the lighting for the cube.

For more information on this approach, see WWDC session Optimize your 3D assets for spatial computing, and jump to sections

  • 15:07 - Sky dome setup
  • 16:03 - Image-based lighting

I did everything programmatically (instead of using Reality Composer Pro), but it pretty much works the same.

Sample code (caveat, I have no idea if this is the preferred approach, but it works for me):

import SwiftUI
import RealityKit
import os.log

struct MyRealityView: View {
    
    @Binding var useNebulaForLighting: Bool
    @Binding var showNebula: Bool
    
    @State private var nebulaIbl: ImageBasedLightComponent?
    @State private var indoorIbl: ImageBasedLightComponent?
    
    @State private var iblEntity: Entity?
    @State private var litCube: Entity?
    @State private var skydome: Entity?
    
    var body: some View {
            
            RealityView { content in
                
                // Create a red cube 1m on a side
                let mesh = MeshResource.generateBox(size: 1.0)
                let simpleMaterial = SimpleMaterial(color: .red, isMetallic: false)
                let model = ModelComponent(
                    mesh: mesh,
                    materials: [simpleMaterial]
                )
                let redBoxEntity = Entity()
                redBoxEntity.components.set(model)
                content.add(redBoxEntity)
                litCube = redBoxEntity
                
                // Get hi-res texture to show as background
                let immersion_name = "BlueNebula"
                guard let resource = try? await TextureResource(named: immersion_name) else {
                    fatalError("Unable to load texture.")
                }
                var material = UnlitMaterial()
                material.color = .init(texture: .init(resource))
                
                // Create sky dome sphere
                let sphereMesh = MeshResource.generateSphere(radius: 1000)
                let sphereModelComponent = ModelComponent(mesh: sphereMesh, materials: [material])
                
                // Create an entity and set its model component
                let sphereEntity = Entity()
                sphereEntity.components.set(sphereModelComponent)

                // Trick/hack to make the texture image point inward to the viewer.
                sphereEntity.scale *= .init(x: -1, y: 1, z: 1)
                
                // Add sky dome to the scene
                skydome = sphereEntity
                skydome?.isEnabled = showNebula
                content.add(skydome!)
                
                // Create Image Based Lighting entity for scene
                iblEntity = Entity()
                content.add(iblEntity!)
                
                // Load low-res nebula resource for image based lighting
                if let environmentResource = try? await EnvironmentResource(named: "BlueNeb2") {
                    let iblSource = ImageBasedLightComponent.Source.single(environmentResource)
                    let iblComponent = ImageBasedLightComponent(source: iblSource)
                    nebulaIbl = iblComponent
                }
                
                // Load low-res indoor light resource for image based lighting
                if let environmentResource = try? await EnvironmentResource(named: "IndoorLights") {
                    let iblSource = ImageBasedLightComponent.Source.single(environmentResource)
                    let iblComponent = ImageBasedLightComponent(source: iblSource)
                    indoorIbl = iblComponent
                }
                
                // Set initial settings
                applyModelSettings()
            }
            update: { content in
                applyModelSettings()
            }
            .realityViewCameraControls(CameraControls.orbit)
    }
    
    func applyModelSettings() {
        
        // Set image based lighting
        if (useNebulaForLighting == true)
            && (litCube != nil)
            && (nebulaIbl != nil) {
            iblEntity!.components.set(nebulaIbl!)
            let iblrc = ImageBasedLightReceiverComponent(imageBasedLight: iblEntity!)
            litCube?.components.set(iblrc)
        }
        else if (useNebulaForLighting == false)
                    && (litCube != nil)
                    && (indoorIbl != nil) {
            iblEntity!.components.set(indoorIbl!)
            let iblrc = ImageBasedLightReceiverComponent(imageBasedLight: iblEntity!)
            litCube?.components.set(iblrc)
        }
        
        // set skydome's status
        skydome?.isEnabled = showNebula
    }
}
Accepted Answer

I have found a different approach that works for me. I have abandoned setting the content.environment property and use a sky dome model for the background and use ImageBasedLightComponent and ImageBasedLightReceiverComponent to choose the lighting for the cube.

For more information on this approach, see WWDC session Optimize your 3D assets for spatial computing, and jump to sections

  • 15:07 - Sky dome setup
  • 16:03 - Image-based lighting

I did everything programmatically (instead of using Reality Composer Pro), but it pretty much works the same.

Sample code (caveat, I have no idea if this is the preferred approach, but it works for me):

import SwiftUI
import RealityKit
import os.log

struct MyRealityView: View {
    
    @Binding var useNebulaForLighting: Bool
    @Binding var showNebula: Bool
    
    @State private var nebulaIbl: ImageBasedLightComponent?
    @State private var indoorIbl: ImageBasedLightComponent?
    
    @State private var iblEntity: Entity?
    @State private var litCube: Entity?
    @State private var skydome: Entity?
    
    var body: some View {
            
            RealityView { content in
                
                // Create a red cube 1m on a side
                let mesh = MeshResource.generateBox(size: 1.0)
                let simpleMaterial = SimpleMaterial(color: .red, isMetallic: false)
                let model = ModelComponent(
                    mesh: mesh,
                    materials: [simpleMaterial]
                )
                let redBoxEntity = Entity()
                redBoxEntity.components.set(model)
                content.add(redBoxEntity)
                litCube = redBoxEntity
                
                // Get hi-res texture to show as background
                let immersion_name = "BlueNebula"
                guard let resource = try? await TextureResource(named: immersion_name) else {
                    fatalError("Unable to load texture.")
                }
                var material = UnlitMaterial()
                material.color = .init(texture: .init(resource))
                
                // Create sky dome sphere
                let sphereMesh = MeshResource.generateSphere(radius: 1000)
                let sphereModelComponent = ModelComponent(mesh: sphereMesh, materials: [material])
                
                // Create an entity and set its model component
                let sphereEntity = Entity()
                sphereEntity.components.set(sphereModelComponent)

                // Trick/hack to make the texture image point inward to the viewer.
                sphereEntity.scale *= .init(x: -1, y: 1, z: 1)
                
                // Add sky dome to the scene
                skydome = sphereEntity
                skydome?.isEnabled = showNebula
                content.add(skydome!)
                
                // Create Image Based Lighting entity for scene
                iblEntity = Entity()
                content.add(iblEntity!)
                
                // Load low-res nebula resource for image based lighting
                if let environmentResource = try? await EnvironmentResource(named: "BlueNeb2") {
                    let iblSource = ImageBasedLightComponent.Source.single(environmentResource)
                    let iblComponent = ImageBasedLightComponent(source: iblSource)
                    nebulaIbl = iblComponent
                }
                
                // Load low-res indoor light resource for image based lighting
                if let environmentResource = try? await EnvironmentResource(named: "IndoorLights") {
                    let iblSource = ImageBasedLightComponent.Source.single(environmentResource)
                    let iblComponent = ImageBasedLightComponent(source: iblSource)
                    indoorIbl = iblComponent
                }
                
                // Set initial settings
                applyModelSettings()
            }
            update: { content in
                applyModelSettings()
            }
            .realityViewCameraControls(CameraControls.orbit)
    }
    
    func applyModelSettings() {
        
        // Set image based lighting
        if (useNebulaForLighting == true)
            && (litCube != nil)
            && (nebulaIbl != nil) {
            iblEntity!.components.set(nebulaIbl!)
            let iblrc = ImageBasedLightReceiverComponent(imageBasedLight: iblEntity!)
            litCube?.components.set(iblrc)
        }
        else if (useNebulaForLighting == false)
                    && (litCube != nil)
                    && (indoorIbl != nil) {
            iblEntity!.components.set(indoorIbl!)
            let iblrc = ImageBasedLightReceiverComponent(imageBasedLight: iblEntity!)
            litCube?.components.set(iblrc)
        }
        
        // set skydome's status
        skydome?.isEnabled = showNebula
    }
}
RealityView in macOS, Skybox, and lighting issue
 
 
Q