I have a VideoMaterial inside a RealityView and want to attach this to a DockingRegion inside an immersive environment. It appears that adding the VideoMaterial entity as a child of the docking region somewhat works, but there are no lighting effects (specular, diffuse) from the playing video. So essentially, how can you add a VideoMaterial to a DockingRegion and achieve the same reflections/behavior as using AVPlayerViewController. The latter is not an option as I need custom controls.

Is it possible to manage the behavior of timeline totally from code? I am exploring the Compose interactive 3D content in Reality Composer Pro sample project after seeing the related video, but the example shows only the use of Behaviors from RCP to activate timelines actions. I was wondering if it is possible to, somehow, retrieve some kind of timeline controller that allows me access to its informations just like the AnimationPlaybackController does with single animations. What I would like to achieve is being able to play/pause/retrieve timestamp from them in order to allow synchronization between different users on SharePlay

Regarding the Apple Vision Pro, is there a possibility to get a kind of rental device or leasing device from Apple? Background: I'm a fresh VisionOS dev. I have a low budget. I'm living in Japan. That means Vision Pro is roughly 150% the USD price. Regards

I created an Object & Hand Tracking app based on the sample code released here by Apple. https://developer.apple.com/documentation/visionos/exploring_object_tracking_with_arkit The app worked great and everything was fine, but I realized I was coding on Xcode 16 beta 3, so I installed the latest Xcode 16 from the App Store and tested by app there, and it completely crashed. No idea why. Here is the console dyld[1457]: Symbol not found: _$ss13withTaskGroup2of9returning9isolation4bodyq_xm_q_mScA_pSgYiq_ScGyxGzYaXEtYas8SendableRzr0_lF Referenced from: <3AF14FE4-0A5F-381C-9FC5-E2520728FC65> /private/var/containers/Bundle/Application/F74E88F2-874F-4AF4-9D9A-0EFB51C9B1BD/Hand Tracking.app/Hand Tracking.debug.dylib Expected in: <2F158065-9DC8-33D2-A4BF-CF0C8A32131B> /usr/lib/swift/libswift_Concurrency.dylib It was working perfectly fine on Xcode 16 beta 3, which makes me think it's an Xcode 16 issue, but no idea how to fix this. I also installed Xcode 16.2 beta (the newest beta) but same error. Please help if anyone knows what is wrong!

As the title states, this severely limits the flexibility of multi-window applications in creating a good user experience. Even effects like the ones shown below cannot be achieved.

Im asking myself we are the limits of RealityView. For example is it possible to place an entity on postion (x=800m,y=0,z=-900m) What happens if i walk from my (0,0,0) to this point, will i see the entity then ? Does someone know where are the limits ?

As mentioned in https://forums.developer.apple.com/forums/thread/756736?answerId=810096022#810096022 Is there any update about the full support to WebXR AR Module, which should enable immersive-ar mode? Are the features such as DOM overlays and WebGPU bindings on the roadmap? Is it possible to capture stereoscopic video either internally or externally or via airplay for debugging purposes? Thanks

I have an app with a visionOS target, and I want to add an iOS target. Both are based on RealityKit. I want to use a SpatialTapGesture to get the tap coordinate local to the entity tapped. In visionOS this is easy: SpatialTapGesture(coordinateSpace: .local) .targetedToAnyEntity() .onEnded { tap in let entity = tap.entity let localPoint3D = tap.convert(tap.location3D, from: .local, to: entity) // … } However, according to the docs, the convert function seems to exist only in visionOS, not in iOS. So how can I do this conversion in iOS? PS: This was already posted on StackOverflow without success. There, I tried to find a workaround, but I failed.

I am trying to create reflections and lighting in a fully virtual scene, and the way the documentation is written, the VirtualEnvironmentProbeComponent would be key. I cannot seem to get it to work, however. Has anyone got this figured out?

Hi, I’m working on a portfolio project for Vision Pro these days. I have two projects and each of projects are made with Unity and made with Xcode(using ARKit and RealityKit tracking feature). Is it able to combine these two projects in an app? For example, using the buttons made with SwiftUI in a Reality Composer Pro, jumping to a scene in Unity, and back from a scene in Unity to a scene in Reality Composer Pro in an app.

I'm seeking detailed information about the rotation matrix of the iPhone's front-facing (selfie) camera when using ARKit. Specifically, I need to understand: The exact rotation matrix applied to the front-facing camera's output in ARKit. Whether this matrix is consistent across all iPhone models or if there are variations. If there are any transformations applied to align the camera's coordinate system with the device's orientation, particularly in portrait mode. How this rotation matrix relates to the transform property of `ARFrame.camera

I'm exploring face tracking and experimenting with ARKit's ARSCNFaceGeometry face mesh. I'm running a minimal demo application on the latest iPad Pro M4 11-inch, and I've provided the code below. I've heard that Apple still offers some of the best face tracking technology on consumer devices, largely because they are one of the few that combine depth and image data. Both a colleague and I tested the demo, and while it works as well or better than some other solutions we tried, we weren’t particularly impressed compared to Google’s MediaPipe or Nvidia’s Maxine, both of which rely solely on image data without depth. In our case, the ARKit face mesh doesn’t always align perfectly with the chin, and as the face rotates, in some areas vertices shift by up to a centimeter from their original position. This led us to question whether our demo app was using the TrueDepth sensor at all. To test this, we used a piece of cardboard with a small hole punched in it and taped it over the sensor array, leaving only the camera exposed. On the iOS lock screen, this prevents FaceID from working, but we still get a clear image from the camera. With the TrueDepth sensor blocked, the face mesh tracking in our app still functioned, but honestly, we couldn’t detect a significant difference in tracking performance with or without the TrueDepth sensor obscured. Could we be setting up the face tracking configuration incorrectly? Or has face tracking in newer versions of iOS become less dependent on the TrueDepth sensor? The controller: import SwiftUI import ARKit struct FaceTrackingView1: UIViewControllerRepresentable { func makeUIViewController(context: Context) -> FaceTrackingViewController1 { return FaceTrackingViewController1() } func updateUIViewController(_ uiViewController: FaceTrackingViewController1, context: Context) { } } class FaceTrackingViewController1: UIViewController, ARSCNViewDelegate, ARSessionDelegate { var sceneView: ARSCNView! override func viewDidLoad() { super.viewDidLoad() sceneView = ARSCNView(frame: view.bounds) sceneView.delegate = self sceneView.automaticallyUpdatesLighting = true view.addSubview(sceneView) let config = ARFaceTrackingConfiguration() sceneView.session.run(config) } override func viewWillDisappear(_ animated: Bool) { super.viewWillDisappear(animated) sceneView.session.pause() } func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? { guard anchor is ARFaceAnchor else { return nil } let faceGeometry = ARSCNFaceGeometry(device: sceneView.device!)! let faceNode = SCNNode(geometry: faceGeometry) faceNode.geometry?.firstMaterial?.fillMode = .lines // Makes it a wireframe mesh return faceNode } func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) { guard let faceAnchor = anchor as? ARFaceAnchor, let faceGeometry = node.geometry as? ARSCNFaceGeometry else { return } faceGeometry.update(from: faceAnchor.geometry) } } The view: import SwiftUI struct ContentView: View { @State private var isFaceTrackingActive = false var body: some View { VStack { Text("Face mesh tracking demo") .font(.title) .padding() Button(action: { isFaceTrackingActive.toggle() }) { Text("Start Face Tracking") .font(.title2) .padding() .background(Color.blue) .foregroundColor(.white) .cornerRadius(10) } .fullScreenCover(isPresented: $isFaceTrackingActive) { FaceTrackingView1() } } .padding() } } #Preview { ContentView() }

Hi, I am trying to stream spatial video in realtime from my iPhone 16. I am able to record spatial video as a file output using: let videoDeviceOutput = AVCaptureMovieFileOutput() However, when I try to grab the raw sample buffer, it doesn't include any spatial information: let captureOutput = AVCaptureVideoDataOutput() //when init camera session.addOutput(captureOutput) captureOutput.setSampleBufferDelegate(self, queue: sessionQueue) //finally func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { //use sample buffer (but no spatial data available here) } Is this how it's supposed to work or maybe I am missing something? this video: https://developer.apple.com/videos/play/wwdc2023/10071 gives us a clue towards setting up spatial streaming and I've got the backend all ready for 3D HLS streaming. Now I am only stuck at how to send the video stream to my server.

Hi everyone, I'm looking for some guidance on how to create an .arobject file. Is there a way to generate it from a 3D model (like a .objcap or .usdz or .fbx/.obj file)? Or can it only be generated by scanning a real-world object using the ARKitScanner project? Any advice or resources on this would be greatly appreciated! I've only found this project for scanning real-world objects. Thanks in advance!

Hi everyone, I’m working on an app for VisionOS that needs to recognize individual rooms in a hallway based on the person the room belongs to (using the name displayed on each office door). Is there any sample code or resource that can guide me in implementing this feature? Thanks in advance for your help!

We are developing a mixed reality app for Vision Pro using Reality Composer Pro, but we're consistently encountering a Protobuf-related crash whenever we enter the immersive space. Our Reality Composer Pro package is quite complex, with numerous objects. Could the complexity of the package be contributing to the issue, or could something else be at play? No errors are being flagged in the code or Reality Composer itself. Here’s the error log: [libprotobuf FATAL /Library/Caches/com.apple.xbs/Sources/REKit/ThirdParty/protobuf/src/google/protobuf/io/zero_copy_stream_impl_lite.cc:276] CHECK failed: (count) &gt;= (0): libc++abi: terminating due to uncaught exception of type google::protobuf::FatalException: CHECK failed: (count) &gt;= (0): Any insight on what might be causing this would be appreciated.

I am new here, my name is Axel - Hi! 👋 I am both a visionOS beginner and an expert in things 3D, now figuring my way into Reality Composer Pro. Most things are intuitive and easy to understand, yet there is one thing I cannot seem to figure out, and I feel really stupid because it must be there, and that's Keyframing: I would simply make a new Timeline and animate published parameters in a Shader Graph over time. I know how to do this via Xcode and Custom Components, but that can't be it because that will break even on simple to medium animation in terms of previewing and fine-tuning, complete overkill for a simple value animation. Since key framing is the most basic functionality of 3D apps next to move, I am sure it is in RCP somewhere. Anyone got a pointer for me?

Hello Developers, I am currently in the initial planning stages of my bachelor thesis in computer science, where I will be developing an application in collaboration with a manufacturer of large-scale machinery. One of the core features I aim to implement is the ability for multiple Apple Vision Pro users to view the same object in augmented reality simultaneously, each from their respective positions relative to the object. I am still exploring how best to achieve this feature. My initial approach involves designating one device as the host of a "room" within the application, allowing other users to join. If I can accurately determine the relative positions of all users to the host device, it should be possible to display the AR content correctly in terms of angle, size, and location for each user. Despite my research, I haven't found much information on similar projects, and I would appreciate any insights or suggestions. Specifically, I am curious about common approaches for synchronizing AR experiences across multiple devices. Given that the Apple Vision Pro does not have a GPS sensor, I am also looking for alternative methods to precisely determine the positions of multiple devices relative to each other. Any advice or shared experiences would be greatly appreciated! Best regards, Revin

Hello Apple Team, Is it possible to change the zoom factor, exposure, white balance and other settings, of an iOS ARKit session? I know how to do it using an AVCaptureSession, however, I can't figure out how to access the AVCaptureDeviceInput of the current AR session. Thanks PS: I'm using ARkit and RealityKit on iOS 17

I've tried importing a 3D KTX file into Reality Composer Pro ShaderGraph, but got nothing output. Realitykit doesn't support Image3D yet? First, I use pyktx to create a 3D KTX texture file and it seems correct in Finder preview. import numpy as np from pyktx import KtxTexture2, KtxTextureCreateInfo, KtxTextureCreateStorage, VkFormat size = 32 image = np.zeros((size, size, size, 3), dtype=np.uint8) for i in range(size): image[i, :, :, 0] = np.interp(i, [0, size], [50, 255]) image[:, i, :, 1] = np.interp(i, [0, size], [100, 255]) image[:, :, i, 2] = np.interp(i, [0, size], [150, 255]) info = KtxTextureCreateInfo( gl_internal_format=None, vk_format=VkFormat.VK_FORMAT_R8G8B8_SRGB, base_width=image.shape[2], base_height=image.shape[1], base_depth=image.shape[0], num_dimensions=3, num_levels=1, num_layers=1, num_faces=1, generate_mipmaps=False, is_array=False, ) texture = KtxTexture2.create(info, KtxTextureCreateStorage.ALLOC) for _ in range(1): texture.set_image_from_memory(0, 0, _, image[_].tobytes()) texture.write_to_named_file(f'{size}.ktx') Then I import it into ShaderGraph like this but seems nothing could be read. In addition, I tested 2D KTX file and it worked. I also tested different vk_format or KTX1/KTX2 but did not work. I also tested Image3DPixel/Image2DArray/Image3DRead and none of them work as long as it's 3D.