Hey, In the "Explore object tracking for visionOS" session we explore how a Globe can be tracked, and objects can be anchored to various positions. My question is if the physical Globe is rotated, will the anchored objects also respond to this in real-time? I would like to overlap a virtual map on top of a physical globe, so when the user rotates the physical globe, the virtual map also seamlessly responds. Is this possible using Object Tracking? Thanks

Are you planning on publishing a complete sample code project related to the Explore object tracking for visionOS session (wwdc2024/10101)? The animation at 12:50 where the globe opens up was especially impressive. Seeing how that was done while tracking to the globe would be very interesting. (I realize that we would have to create our own globe object in order for the code to work.)

When I try to train an object tracking model in Create ML, the progress bar stays stuck at 0% for about a minute. However, the training cancels soon afterwards without an additional error message. I've tried multiple different USDZ files and tried on two different Apple silicon Mac devices. Is anyone else experiencing this issue? Thank you!

Background: The app that I am working on lets the user place things in their surroundings and recovers those placements the next time their enter the immersive scene. From the documentation and discussions I have had, World Tracked Anchors are local to the device. My questions are: What happens to these anchors when the user updates their device to the next generation? What happens to these anchors if the user gets an Apple Care replacement? Are they backed up and restored via iCloud? If not, I filed a feedback about it a few months back :D FB13613066

What I want to do: I want to turn only the walls of a room into RealityKit Entities that I can collide with, or turn into occlusion surfaces. This requires adding and maintaining RealityKit entities that with mesh information from the RoomAnchor. It also requires creating a "collision shape" from the mesh information. What I've explored: A RoomAnchor can provide me MeshAnchor.Geometry's that match only the "wall" portions of a Room. I can use this mesh information to create RealityKit entities and add them to my immersive view. But those Mesh's don't come with UUIDs, so I'm not sure how I could know which entities meshes need to to be updated as the RoomAnchor is updated. As such I just keep adding duplicate wall entities. A RoomAnchor also provides me with the UUIDs of its plane anchors, but no way to connect those to the provided meshes that I've discovered so far. Here is how I add the green walls from the RoomAnchor wall meshes. Note: I don't like that I need to wrap this in a task to satisfy the async nature of making a shape from a mesh. could be stuck with it, though. Warning: this code will keep adding walls, even if there are duplicates and will likely cause performance issues :D. func updateRoom(_ anchor: RoomAnchor) async throws { print("ROOM ID: \(anchor.id)") anchor.geometries(of: .wall).forEach { mesh in Task { let newEntity = Entity() newEntity.components.set(InputTargetComponent()) realityViewContent?.addEntity(newEntity) newEntity.components.set(PlacementUtilities.PlacementSurfaceComponent()) collisionEntities[anchor.id]?.components.set(OpacityComponent(opacity: 0.2)) collisionEntities[anchor.id]?.transform = Transform(matrix: anchor.originFromAnchorTransform) // Generate a mesh for the plane do { let contents = MeshResource.Contents(planeGeometry: mesh) let meshResource = try MeshResource.generate(from: contents) // Make this plane occlude virtual objects behind it. // entity.components.set(ModelComponent(mesh: meshResource, materials: [OcclusionMaterial()])) collisionEntities[anchor.id]?.components.set(ModelComponent(mesh: meshResource, materials: [SimpleMaterial.init(color: .green, roughness: 1.0, isMetallic: false)])) } catch { print("Failed to create a mesh resource for a plane anchor: \(error).") return } // Generate a collision shape for the plane (for object placement and physics). var shape: ShapeResource? = nil do { let vertices = anchor.geometry.vertices.asSIMD3(ofType: Float.self) shape = try await ShapeResource.generateStaticMesh(positions: vertices, faceIndices: anchor.geometry.faces.asUInt16Array()) } catch { print("Failed to create a static mesh for a plane anchor: \(error).") return } if let shape { let collisionGroup = PlaneAnchor.verticalCollisionGroup collisionEntities[anchor.id]?.components.set(CollisionComponent(shapes: [shape], isStatic: true, filter: CollisionFilter(group: collisionGroup, mask: .all))) // The plane needs to be a static physics body so that objects come to rest on the plane. let physicsMaterial = PhysicsMaterialResource.generate() let physics = PhysicsBodyComponent(shapes: [shape], mass: 0.0, material: physicsMaterial, mode: .static) collisionEntities[anchor.id]?.components.set(physics) } collisionEntities[anchor.id]?.components.set(InputTargetComponent()) } } }

I'm implementing an AR app with Image Tracking capabilities. I noticed that it takes very long for the entities I want to overlay on a detected image to show up in the video feed. When debugging using debugOptions.insert(.showAnchorOrigins), I realized that the image is actually detected very quickly, the anchor origins show up almost immediately. And I can also see that my code reacts with adding new anchors for my ModelEntities there. However, it takes ages for these ModelEntities to actually show up. Only if I move the camera a lot, they will appear after a while. What might be the reason for this behaviour? I also noticed that for the first image target, a huge amount of anchors are being created. They start from the image and go all up towards the user. This does not happen with subsequent (other) image targets.

I really love the way how you can add SwiftUI views as attachments to a RealityView on visionOS. As I am now porting my app to iOS as well, I was wondering if something like this is possible in ARView as well? I've only seen custom libraries trying to mimic UI elements so far.

Hi, I'm trying to test object recognition using ARKit. I scanned a couple of objects using the Apple demo app, copied the .arobject files to my laptop. I added them to my new project in Assets like shown in the image. However, as I follow the tutorial to load these objects to use as reference objects, I run into an error. let configuration = ARWorldTrackingConfiguration() guard let referenceObjects = ARReferenceObject.referenceObjects(inGroupNamed: "Test", bundle: Bundle.main) else { let ro = ARReferenceObject.referenceObjects(inGroupNamed: "Test", bundle: nil); let ro1 = ARReferenceObject.referenceObjects(inGroupNamed: "Gallery", bundle: .main); fatalError("Resource not found") } Here, we fail the guard statement and the ro and ro1 both are set to nil. I created a new project with just this one statement and that fails too. I'm using SwiftUI instead of UIKit if that makes a difference and am calling this in the makeUIView() function . Any pointers to what I might be doing wrong here are appreciated.

[tldr version: all the point cloud capture apps rushed out an update when the iPhone 15 Pro was released because they were capturing far fewer points on that device. The same is observed with the new M4 iPad Pro. What was the fix for compatibility with these new devices?] I am running an ARKit replay file through "Displaying a Point Cloud Using Scene Depth" from WWDC20 and recording the ARConfidenceLevel values of the incoming ARDepthData. I am doing this side-by-side on an iPhone 12 Pro and an iPhone 15 Pro. The ARKit replay file was originally recorded on the 12 Pro. We get a certain percentage of points where the ARConfidenceLevel is not ".high" when running on the iPhone 12 Pro. It's varies a lot by frame but averages about 5% and is the same on all devices prior to the iPhone 15 Pro. The same test using the same iPhone 12 Pro replay file on an iPhone 15 Pro gives about twice as many points where the ARConfidenceLevel is not ".high" (about 10% on average on this particular replay file). This corresponds with real-world usage of our app on the iPhone 15 Pro where far fewer points are captured on that device compared with all previous models. (Our app filters out points where the ARConfidenceLevel is not ".high".) Apple's interpretation of the same LiDAR data is clearly different on the iPhone 15 Pro and M4 iPad Pro when compared to earlier devices. Can you please advise how to maintain equivalent behaviour on the new devices? Steps to reproduce: Run "Displaying a Point Cloud Using Scene Depth" from WWDC20 session 10611: Explore ARKit 4 following the instructions at https://developer.apple.com/documentation/arkit/arkit_in_ios/environmental_analysis/displaying_a_point_cloud_using_scene_depth Use Xcode's setting to replay data to ARKit while running on an iPhone 12 Pro (using any replay file recorded on that device). An iPhone 13 or 14 Pro will work just as well. Record what % of points have an ARConfidenceLevel that is not .high. Now do it again running the same replay file on an iPhone 15 Pro. Note that the % of points have an ARConfidenceLevel that is not .high is much higher.

Hi, I'm working on a virtual furniture placement app. I have used Object Placement example, and wonder if is possible to backup a anchor to cloud or share anchor to another device which will let users to view the same model in the same place? Thanks

I have found that my Vision Pro device can get into a state where my app is no longer receiving fresh SceneReconstructionProvider updates. It reports that the SceneReconstructionProvider goes into the DataProviderState.running state, and .anchorUpdates will report a set of stale mesh anchors when first fired up, but does not produce any further updates. Once the device gets into this state, I can force quit the app, and even uninstall and re-install it, and I get the same few mesh updates, but no fresh updates until I restart the device. Sample async function below. I can confirm that print("WE FELL OFF THE END OF sceneReconstruction.anchorUpdates") never gets executed, so it stays inside the sceneReconstruction.anchorUpdates loop. let session = ARKitSession() var handTracking = HandTrackingProvider() let sceneReconstruction = SceneReconstructionProvider() let planeDetection = PlaneDetectionProvider(alignments: [.horizontal, .vertical]) let worldTracking = WorldTrackingProvider() ... func start() async { do { await requestAuth() if dataProvidersAreSupported && isReadyToRun && !isRunning { // print("ARKitSession starting.") try await session.run([sceneReconstruction, handTracking, planeDetection, worldTracking]) startCount += 1 // TODO: Fail gracefully if we have to attempt start too many (# TBD) times } else { print("dataProvidersAreSupported: \(dataProvidersAreSupported). isReadyToRun: \(isRunning)") print("handTracking.state: \(handTracking.state), sceneReconstruction.state: \(sceneReconstruction.state) worldTracking.state: \(worldTracking.state), planeDetection.state; \(planeDetection.state)") } }catch { print("ARKitSession error:", error) } } ... func processReconstructionUpdates() async { while (true) { for await update in sceneReconstruction.anchorUpdates { let meshAnchor = update.anchor guard let shape = try? await ShapeResource.generateStaticMesh(from: meshAnchor) else { continue } switch update.event { case .added: let entity = try! await generateModelEntity(geometry: meshAnchor.geometry) entity.transform = Transform(matrix: meshAnchor.originFromAnchorTransform) entity.collision = CollisionComponent(shapes: [shape], isStatic: true) entity.components.set(InputTargetComponent()) entity.name = "mesh" entity.physicsBody = PhysicsBodyComponent(mode: .static) let sortComponent = ModelSortGroupComponent(group: modelSortGroup, order: 1) entity.components.set(sortComponent) entity.components.set(OpacityComponent(opacity: 0.5)) meshEntities[meshAnchor.id] = entity meshesParent.addChild(entity, preservingWorldTransform: true) case .updated: guard let entity = meshEntities[meshAnchor.id], let updatedEntity = try? await generateModelEntity(geometry: meshAnchor.geometry) else { continue } entity.transform = Transform(matrix: meshAnchor.originFromAnchorTransform) entity.collision?.shapes = [shape] if let newMesh = updatedEntity.model?.mesh { entity.model?.mesh = newMesh } case .removed: meshEntities[meshAnchor.id]?.removeFromParent() meshEntities.removeValue(forKey: meshAnchor.id) } print("We now have '\(meshEntities.count)' mesh entities") } print("WE FELL OFF THE END OF sceneReconstruction.anchorUpdates") try? await Task.sleep(nanoseconds: 1_000_000) }

What is the reason the hand-tracking joints have these axes? I'm trying to create a virtual hands model and that's a mess.

We want to make a multi-person networked application of vision pro. The first step is to need multiple vision pro with the same spatial information, so as to make the world coordinates consistent. Can we completely copy the spatial information created by one of the vision pro scans to other devices?

How is it possible to add a schema for ar to a usd file using the python tools (or any other way). Following the instructions in: https://developer.apple.com/documentation/arkit/arkit_in_ios/usdz_schemas_for_ar/actions_and_triggers/preliminary_behavior The steps are to have the following declaration: class Preliminary_Behavior "Preliminary_Behavior" ( inherits = </Typed> ) and a usd file #usda 1.0 def Preliminary_Behavior "TapAndFlip" { rel triggers = [ <Tap> ] rel actions = [ <Entry> ] def Preliminary_Trigger "Tap" ( inherits = </TapGestureTrigger> ) { rel affectedObjects = [ </Cube> ] } def Preliminary_Action "Entry" ( inherits = </GroupAction> ) { uniform token type = "parallel" rel actions = [ <Flip> ] } def Preliminary_Action "Flip" ( inherits = </EmphasizeAction> ) { rel affectedObjects = [ </Cube> ] uniform token motionType = "flip" } } def Cube "Cube" { } How do these parts fit together? I saved the usda file, but it didn't have any interactions. Obviously, I have to add that declaration, but how do I do this? is this all in an AR Xcode project? Or can I do this with python tools (I would prefer something very lightweight).

I built two parts of my app a bit disjointed: my physics component, which controls all SceneReconstruction, HandTracking, and WorldTracking. my spatial GroupActivities component that allows you to see personas of those that join the activity. My problem: When trying to use any DataProvider in a spatial experience, I get the ARKit Session Event: dataProviderStateChanged, which disables all of my providers. My question: Has anyone successfully been able to find a workaround for this? I think it would be amazing to have one user be able to be the "host" for the activity and the scene reconstruction provider still continue to run for them.

Hi all, I am generating some USDZ files that will be used in quicklook and be accessible with the Vision Pro. I was wondering are there any examples with USDZ files with actions? like the ability to change a state of assesses by tapping on them? I know this works with .reality files, but I would like to use Python to create some automatically generated USDZ files that allow some interaction. I'm currently stuck! So an example of the capabilities would be great - or pointing to some code that has done this in python. Thanks!

I am developing an immersive application featured with hands interacting my virtual objects. When my hand passes through the object, the rendered color of my hand is like blending hand color with object's color together, both semi transparent. I wonder if it is possible to make my hand be always "opaque", or say the alpha value of rendered hand (coz it's VST) is always 1, but the object's alpha value could be varied in terms of whether it is interacting with hand. (I was thinking this kind of feature might be supported by a specific component (just like HoverEffectComponent), but I didn't find that out)

Hi, We are having problems with IOs Quick Look not working. Specifically, the AR button being greyed out after having opened the Scene / AR model previously. This is all running off our Web-App. What we have figured out is clearing the device's cache solves the issue and the greyed out button turns blue and clickable again. We are receiving this issue very inconsistently though - on iPad as well as iPhone and on both newer and older IOs versions. Very happy for any responses and advice to solve this issue as its behaviour makes the quick look function - even if it's great (when it works) unviable for Production (because it doesn't work consistently). Best Regards

Currently in an app I am working on, we are adding collision shapes/components to objects by using the ShapeResource.generateConvex method to generate the shape from the mesh of our ModelEntity. Unfortunately, this does not result in a totally accurate collision shape. The following example is how the collision component looks currently. Is there anyway to generate a collision shape that fits the exact bounds of the ModelEntity?

Im trying to use a RealityView with attachments and this error is being thowen. Am i using the RealityView wrong? I've seen other people use a RealityView with Attachments in visionOS... Please let this be a bug... RealityView { content, attachments in contentEntity = ModelEntity(mesh: .generatePlane(width: 0.3, height: 0.5)) content.add(contentEntity!) } attachments: { Text("Hello!") }.task { await loadImage() await runSession() await processImageTrackingUpdates() }