I am using RealityView for an iOS program. Is it possible to turn off the camera passthrough, so only my virtual content is showing? I am looking to create VR experience. I have a work around where I turn off occlusion and then create a sphere around me (e.g., with a black texture), but in the pre-RealityView days, I think I used something like this: arView.environment.background = .color(.black) Is there something similar in RealityView for iOS? Here are some snippets of my current work around inside RealityView. First create the sphere to surround the user: // Create sphere let blackMaterial = UnlitMaterial(color: .black) let sphereMesh = MeshResource.generateSphere(radius: 100) let sphereModelComponent = ModelComponent(mesh: sphereMesh, materials: [blackMaterial]) let sphereEntity = Entity() sphereEntity.components.set(sphereModelComponent) sphereEntity.scale *= .init(x: -1, y: 1, z: 1) content.add(sphereEntity) Then turn off occlusion: // Turn off occlusion let configuration = SpatialTrackingSession.Configuration( tracking: [], sceneUnderstanding: [], camera: .back) let session = SpatialTrackingSession() await session.run(configuration)

If I import a USDZ model with blendMode set to alpha, occlusion does not work on iPhone with iOS 18. How should transparent materials and occlusion be properly used in the new RealityKit? Additionally, new artifacts have appeared when working with transparent objects overlapping each other. The transparency results do not blend but rather parts of the model just not rendering.

In RealityKit, I know that an HDR image is pre-calculated, and through the settings of the ImageBasedLight Component, a specified specular object can reflect the content of the HDR image. If a mirror object is originally very large, such as a large-area continuous glass door, after specifying an IBL image for these glass doors, the image reflected by the mirror will be obviously deformed when it moves in space. Because IBL is a picture of the surrounding environment at a point, while the glass door is a surface. Is there a truly real-time specular reflection calculation setup in RealityKit that can reflect the model on the opposite side of the glass door?

How do I enable the system hand controls within an immersive space? I have an ImmersiveScene and would like to enable the new 2.0 system controls like the home button and volume slider. ImmersiveSpace(id: appModel.immersiveSpaceID) { ImmersiveView() } .immersionStyle(selection: .constant(.full), in: .full) .upperLimbVisibility(.visible) While I can see my hands and arms in this view, I cannot get the "New Hand Gestures" to show up when on visionOS 2.0. When I leave the immersive space, they appear.

I'll leave this here for anyone who's interested but it is possible to slightly use Windows VR on ARM Mac, right now it's just some demos but I am still working on solutions: https://www.youtube.com/watch?v=qbucnU0dpDo&t=431s&ab_channel=NightSightProductions

TLDR: Timeline does not play animation when Repeat Forever is checked. Hi! I have created a timeline for my model that does a built-in emphasize animation. Then I added a behavior to my model and has set OnAddedToScene with action to run that timeline. It works perfect well on my device. But I want the timeline to be looped. I realized that there's no loop option in the timeline, but I noticed that I can loop it if I insert it into another timeline(The loop checkbox shows up). So I did that and had my model's behavior to run that timeline. But then the model doesn't play the animation as intended. Note: I am not making a VisionPro app, but an iOS app leveraging ARKit and RealityKit Environment: iPhone 13 Pro Max with iOS18.0 Code: struct ARViewContainer: UIViewRepresentable { func makeUIView(context: Context) -> ARView { let arView = ARView(frame: .zero) arView.session.run() Task { do { let anchor = AnchorEntity(plane: .horizontal) let emojiScene = try await Entity(named: "SunglassesScene", in: bubbleAR anchor.addChild(emojiScene) arView.scene.addAnchor(anchor) } catch { print("Failed to load models: \(error)") } } return arView } } Thank you!

I’m encountering a 1-meter size limit on the visual presentation of objects presented in an immersive environment in vision os, both in the simulator and in the device For example, if I load a USDZ object that’s 1.0x0.5x0.05 meters, all of the 1.0x0.5 meter side is visible. If I scale it by a factor of 2.0, only a 1.0x1.0 viewport onto the object is shown, even though the object size reads out as scaled when queried by usdz.visualBounds(relativeTo: nil).extents and if the USDZ is animated the animation, the animation reflects the motion of the entire object I haven’t been able to determine why this is the case, nor any way to adjust/mitigate it. Is this a wired constraint of the system or is there a workaround. Target environment is visionos 1.2

In visionOS 2 beta, I have a character loaded from a Reality Composer Pro scene standing on the floor, but he isn't casting a shadow on the floor. I added a GroundingShadowComponent in RealityView, and he does cast shadows on himself (e.g., his hands cast shadows on his shoes), but I don't see any shadow on the floor. Do I need to enable something to have my character cast a show on the real-world floor?

Summary: I’m working on a VisionOS project where I need to dynamically load a .bundle file containing RealityKit content from the app’s Application Support directory. The .bundle is saved to disk after being downloaded or retrieved as an On-Demand Resource (ODR). Sample project with the issue: Github repo. Play the target test-odr to use with the local bundle and have the crash. Overall problem: Setup: Add a .bundle named RealityKitContent_RealityKitContent.bundle to the app’s resources. This bundle contains a Reality file with two USDA,: “Immersive” and “Scene”. Save to Disk: save the bundle to the Application Support directory, ensuring that the file is correctly copied and saved. Load the Bundle: load the bundle from the saved URL using Bundle(url: bundleURL) to initialize the Bundle object. Load Entity from Bundle: load a specific entity (“Scene”) from the bundle. When trying to load the entity using let storedEntity = try await Entity(named: "Scene", in: bundle), the app crashes with an EXC_BREAKPOINT error. ContentsOf Method Issue: If I use the Entity.load(contentsOf:realityFileURL, withName: entityName) method, it always loads the first root entity found (in this case, “Immersive”) rather than “Scene”, even when specifying the entity name. This is why I want to use the Bundle to load entities by name more precisely. Issue: The crash consistently occurs on the Entity(named: "Scene", in: bundle) line. I have verified that the bundle exists and is accessible at the specified path and that it contains the expected .reality file with multiple entities (“Immersive” and “Scene”). The error code I get is EXC_BREAKPOINT (code=1, subcode=0x1d135d4d0). What I’ve Tried: • Ensured the bundle is properly saved and accessible. • Checked that the bundle is initialized correctly from the URL. • Tested loading the entity using the contentsOf method, which works fine but always loads the “Immersive” entity, ignoring the specified name. Hence, I want to use the Bundle-based approach to load multiple USDA entities selectively. Question: Has anyone faced a similar issue or knows why loading entities using Entity(named:in:) from a disk-based bundle causes this crash? Any advice on how to debug or resolve this, especially for managing multiple root entities in a .reality file, would be greatly appreciated.

We've been using our app for the past year, and a user came back today that after three minutes, their phone starts getting hot and the screen dims. He is using 17.6.1 with an iPhone 14 max. No one else is seeing an issue, but with the posts online about 17.6.1 battery drain, I wonder if our AR app is somehow more sensitive to the issue.

Is there anyway to reset the scan memory Vision Pro stores on-device, so that upon every new scanning in my application, it starts from scratch rather than getting instantly recognized. In Apple Vision Pro Privacy overview (https://www.apple.com/privacy/docs/Apple_Vision_Pro_Privacy_Overview.pdf), it is stated: "visionOS builds a three-dimensional model to map your surroundings on-device. Apple Vision Pro uses a combination of camera and LiDAR data to map the area around you and save that model on-device. The model enables visionOS to alert you about real-life obstacles, as well as appropriately reflect the lighting and shadows of your physical space. visionOS uses audio ray tracing to analyze your room’s acoustic properties on-device to adapt and match sound to your space. The underlying scene mesh is stored on-device and encrypted with your passcode if one is set" How to access and erase the, and I quote, “underlying scene mesh stored on-device”?

Hi everyone, I'm developing an ARKit app using RealityKit and encountering an issue where a video displayed on a 3D plane shows up as a pink screen instead of the actual video content. Here's a simplified version of my setup: func createVideoScreen(video: AVPlayerItem, canvasWidth: Float, canvasHeight: Float, aspectRatio: Float, fitsWidth: Bool = true) -> ModelEntity { let width = (fitsWidth) ? canvasWidth : canvasHeight * aspectRatio let height = (fitsWidth) ? canvasWidth * (1/aspectRatio) : canvasHeight let screenPlane = MeshResource.generatePlane(width: width, depth: height) let videoMaterial: Material = createVideoMaterial(videoItem: video) let videoScreenModel = ModelEntity(mesh: screenPlane, materials: [videoMaterial]) return videoScreenModel } func createVideoMaterial(videoItem: AVPlayerItem) -> VideoMaterial { let player = AVPlayer(playerItem: videoItem) let videoMaterial = VideoMaterial(avPlayer: player) player.play() return videoMaterial } Despite following the standard process, the video plane renders pink. Has anyone encountered this before, or does anyone know what might be causing it? Thanks in advance!

This effect was mentioned in https://developer.apple.com/wwdc24/10153 (the effect is demonstrated at 28:00), in which the demonstration is you can add coordinates by looking somewhere on the ground and clicking., but I don't understand his explanation very well. I hope you can give me a basic solution. I am very grateful for this!

So, I've been messing around with SteamVR on Apple Silicon and it runs as expected under Rosetta translation, I've even got a game to run. But for some reason SteamVR cannot detect a headset, even when using one that SteamVR has drivers for such as the 2017 Vive headset. Would there be any explanation as to why this is because SteamVR works as expected so that leads me to believe it's something with MacOS.

I have a scene setup that uses places images on planes to mimic an RPG-style character interaction. There's a large scene background image and a smaller character image in the foreground. Both are added as content to a RealityView. There's one attachment that is a dialogue window for interaction with the character, and it is attached to the character image. When the scene changes, I need the images and the dialogue window to refresh. My current approach has been to remove everything from the scene and add the new content in the update closure. @EnvironmentObject var narrativeModel: NarrativeModel @EnvironmentObject var dialogueModel: DialogueViewModel @State private var sceneChange = false private let dialogueViewID = "dialogue" var body: some View { RealityView { content, attachments in //at start, generate background image only and no characters if narrativeModel.currentSceneIndex == -1 { content.add(generateBackground(image: narrativeModel.backgroundImage!)) } } update : { content, attachments in print("update called") if narrativeModel.currentSceneIndex != -1 { print("sceneChange: \(sceneChange)") if sceneChange { //remove old entitites if narrativeModel.currentSceneIndex != 0 { content.remove(attachments.entity(for: dialogueViewID)!) } content.entities.removeAll() //generate the background image for the scene content.add(generateBackground(image: narrativeModel.scenes[narrativeModel.currentSceneIndex].backgroundImage)) //generate the characters for the scene let character = generateCharacter(image: narrativeModel.scenes[narrativeModel.currentSceneIndex].characterImage) content.add(character) print(content) if let character_attachment = attachments.entity(for: "dialogue"){ print("attachment clause executes") character_attachment.position = [0.45, 0, 0] character.addChild(character_attachment) } } } } attachments: { Attachment(id: dialogueViewID){ DialogueView() .environmentObject(dialogueModel) .frame(width: 400, height: 600) .glassBackgroundEffect() } } //load scene images .onChange(of:narrativeModel.currentSceneIndex){ print("SceneView onChange called") DispatchQueue.main.async{ self.sceneChange = true } print("SceneView onChange toggle - sceneChange = \(sceneChange)") } } If I don't use the dialogue window, this all works just fine. If I do, when I click the next button (in another view), which increments the current scene index, I enter some kind of loop where the sceneChange value gets toggled to true but never gets toggled back to false (even though it's changed in the update closure). The reason I have the sceneChange value is because I need to update the content and attachments whenever the scene index changes, and I need a state variable to trigger the update function to do this. My questions are: Why might I be entering this loop? Why would it only happen if I send a message in the dialogue view attachment, which is a whole separate view? Is there a better way to be doing this?

Hi everyone, I'm looking for a way to convert an FBX file to USDZ directly within my iOS app. I'm aware of Reality Converter and the Python USDZ converter tool, but I haven't been able to find any documentation on how to do this directly within the app (assuming the user can upload their own file). Any guidance on how to achieve this would be greatly appreciated. I've heard about Model I/O and SceneKit, but I haven't found much information on using them for this purpose either. Thanks!

In RealityView I have two entities that contain tracking components and collision components, which are used to follow the hand and detect collisions. In the Behaviors component of one of the entities, there is an instruction to execute action through onCollision. However, when I test, they cannot execute action after collisions. Why is this?

Can we get the raw sensor data from the apple vision pro?

Steps to Reproduce: Create a SwiftUI view that initializes an ARKit session and a camera frame provider. Attempt to run the ARKit session and retrieve camera frames. Extract the intrinsics and extrinsics matrices from the camera frame’s sample data. Attempt to project a 3D point from the world space onto the 2D screen using the retrieved camera parameters. Encounter issues due to lack of detailed documentation on the correct usage and structure of the intrinsics and extrinsics matrices. struct CodeLevelSupportView: View { @State private var vm = CodeLevelSupportViewModel() var body: some View { RealityView { realityViewContent in } .onAppear { vm.receiveCamera() } } } @MainActor @Observable class CodeLevelSupportViewModel { let cameraSession = CameraFrameProvider() let arSession = ARKitSession() init() { Task { await arSession.requestAuthorization(for: [.cameraAccess]) } } func receiveCamera() { Task { do { try await arSession.run([cameraSession]) guard let sequence = cameraSession.cameraFrameUpdates(for: .supportedVideoFormats(for: .main, cameraPositions: [.left])[0]) else { print("failed to get cameraAccess authorization") return } for try await frame in sequence { guard let sample = frame.sample(for: .left) else { print("failed to get camera sample") return } let leftEyeScreenImage:CVPixelBuffer = sample.pixelBuffer let leftEyeViewportWidth:Int = CVPixelBufferGetWidth(leftEyeScreenImage) let leftEyeViewportHeight:Int = CVPixelBufferGetHeight(leftEyeScreenImage) let intrinsics = sample.parameters.intrinsics let extrinsics = sample.parameters.extrinsics let oneMeterInFront:SIMD3<Float> = .init(x: 0, y: 0, z: -1) projectWorldLocationToLeftEyeScreen(worldLocation: oneMeterInFront, intrinsics: intrinsics, extrinsics: extrinsics, viewportSize: (leftEyeViewportWidth,leftEyeViewportHeight)) } } catch { } } } //After the function implementation is completed, it should return a CGPoint?, representing the point of this worldLocation in the LeftEyeViewport. If this worldLocation is not visible in the LeftEyeViewport (out of bounds), return nil. func projectWorldLocationToLeftEyeScreen(worldLocation:SIMD3<Float>,intrinsics:simd_float3x3,extrinsics:simd_float4x4,viewportSize:(width:Int,height:Int)) { //The API documentation does not provide the structure of intrinsics and extrinsics, making it hard to done this function. } }

AFAIK there's no way to programmatically detect when an ImmersiveSpaceContent is dismissed by a user (i.e. by pressing the home button). By comparison, ImmersiveView has .onAppear() and .onDisappear(): ImmersiveSpace(id: appModel.immersiveSpaceID) { ImmersiveView() .environment(appModel) .onAppear { appModel.immersiveSpaceState = .open } .onDisappear { appModel.immersiveSpaceState = .closed } } In comparison: // No similar callbacks for here: struct MyImmersiveSpace: ImmersiveSpaceContent { var body: CompositorLayer { /* ... */ } }