Hi everyone I am working on a small project that requires World Anchors so that I can persist my content through whenever the user chooses to leave/close the app. However I can't manage to make my Arkit session to run even though I think all the privacy permissions have been set and allowed correctly. Here is a sample code in an empty scene: // // WorldTrackingView.swift // SH_AVP_Demo // // Created by 李希 on 9/19/24. // import SwiftUI import RealityKit import RealityKitContent //import VisionKit import ARKit import Foundation import UIKit import simd struct WorldTrackingView_test: View { @State var myCube = Entity() @Environment(.scenePhase) var myScenePhase var body: some View { RealityView { content in //Load Scene if let Scene = try? await Entity.load(named: "WorldTrackingScene", in: realityKitContentBundle){ //Add scene to the view content.add(Scene) //Look for the cube entity if let cubeEntity = Scene.findEntity(named: "Cube"){ myCube = cubeEntity // Create collission for the cube myCube.generateCollisionShapes(recursive: true) // Allow inputs to interact myCube.components.set(InputTargetComponent(allowedInputTypes: .indirect)) // set some ground shadows myCube.components.set(GroundingShadowComponent(castsShadow: true)) } } } // Add drag gesture that targets any entity in the scene .gesture(DragGesture().targetedToAnyEntity() //Do something when the cube position changes .onChanged{ value in value.entity.position = value.convert(value.location3D, from: .local, to: value.entity.parent!) myCube = value.entity // Test and see if the Arkit runs with different data providers var session = ARKitSession() var worldData = WorldTrackingProvider() let planeData = PlaneDetectionProvider() let sceneData = SceneReconstructionProvider() do { Task{ try await session.run([worldData]) for await update in worldData.anchorUpdates { switch update.event { case .added, .updated: // Update the app's understanding of this world anchor. print("Anchor position updated.") case .removed: // Remove content related to this anchor. print("Anchor position now unknown.") } } } }catch{ print("session not running \(error.localizedDescription)") return } } //At the end of the gesture save anchor .onEnded{ value in } ) } } #Preview(immersionStyle: .mixed) { WorldTrackingView() } All is does is to generate a cube in an immersive view. The cube has collision and input components added to so that I can interact with it using a drag gesture. I decided to start an arkit session with a WorldTrackingProvider() but I keep getting the following error: ARPredictorRemoteService <0x117e0c620>: Service configured with error: Error Domain=com.apple.arkit.error Code=501 "(null)" Remote Service was invalidated: <ARPredictorRemoteService: 0x117e0c620>, will stop all data_providers. ARRemoteService: remote object proxy failed with error: Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service with pid 81 named com.apple.arkit.service.session was invalidated from this process." UserInfo={NSDebugDescription=The connection to service with pid 81 named com.apple.arkit.service.session was invalidated from this process.} ARRemoteService: weak self released before invalidation ARRemoteService: remote object proxy failed with error: Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service with pid 81 named com.apple.arkit.service.prediction was invalidated from this process." UserInfo={NSDebugDescription=The connection to service with pid 81 named com.apple.arkit.service.prediction was invalidated from this process.} ARRemoteService: weak self released before invalidation If I switch it with a PlaneDetectionProvider() or a SceneReconstructionProvider() I get print statements in my terminal, but none if i replace it with a WorldTrackingProvider(). Any idea what could be causing this? Same code was working before a recent for xcode I believe.

Hello, I want to know who is in charge for the integration of a new ARKit update for the Unreal Engine. We want to use the AR function from the AVP with Ubreal, but the ARKit version is too old.

https://developer.apple.com/documentation/compositorservices/drawing_fully_immersive_content_using_metal I'm following this doc to use metal in vision os. I noticed that the tangent is being deprecated which is being used in the sample https://developer.apple.com/documentation/compositorservices/layerrenderer/drawable/view will the sample code be updated?

When I wanted to call the Reality Composer Pro scene containing Object Tracking, I tried the following code: RealityView { content in if let model = try? await Entity(named: "Scene", in: realityKitContentBundle) { content.add(model) } } Obviously, this is wrong. We need to add some configurations that can enable Object Tracking to Reality View. What do we need to add? Note:I have seen https://developer.apple.com/videos/play/wwdc2024/10101/, but I don't know much about it.

Hi everyone, I want to add new joint in addition to joints that provided by ARKit. for example extract the position of wrist and elbow, then add new joint between them in the middle of arm. I can't find a good documentation that can explain ARKit very well. If there is another information that I can use, please share it with me. thanks.

In lots of houses there are different levels but are still on the same floor. What i mean is that there are things like stairs on the entrance that only have a few steps and would count basically as the same story. RoomPlan already does a nice job recognizing them during the scanning but after the StructureBuilder or the optimization step it is not really satisfying. Has anyone managed to handle those cases? Or do you have to scan a specific way to capture such small differences within a level?

I want to know are depth map and RGB image are perfectly aligned(do both have the same principle point)? If yes then how the depth- map is created. The depth map on iphone12 has 256x192 resolution as opposed to an RGB image (1920x1440). I am interested in exact pixel-wise depth. Is it possible to get the raw depth map of 1920x1440 resolution ? How is the depth-map is created at 256 x 192 resolution? Behind the scenes does the pipeline captures it at 1920 x1440 resolution and then resize it to 256x192? I have so many questions as there are no intrinsic, extrinsic, and calibration data given regarding the lidar. I would greatly appreciate it if someone can explain the steps from a computer-vision perspective. Many Thanks

Hello, I'm using the ARSessionDelegate function: func session(_ session: ARSession, didUpdate frame: ARFrame) to extract an HD Image let hdframe = try? await session.captureHighResolutionFrame().capturedImage which I am later on use to detect text on the image using VN. I'm using the HD picture, because the text bits I'm looking for can be very tiny. let requestHandler = VNImageRequestHandler(cgImage: image)//, orientation: .up, options: [:]) let textRequest = VNRecognizeTextRequest() let vnRequests = [textRequest] try requestHandler.perform(vnRequests) My issue is that, each time a captured HD image is extracted from the AR Scene, a shutter sound is played. I'm aware that shutter sounds are important for privacy, but I'm doing this in a very high frequency, which means that my app is currently unusable, when not muted. My two questions are: Is there any way to disable the sound in this case? Is there a better way to constantly scan the AR video stream for text than this approach?

I would like to implement the following but I am not sure if this is a supported use case based on the current documentation: Run one ARKitSession with a WorldTrackingProvider in Swift for mixed immersion Metal rendering (to get the device anchor for the layer renderer drawable & view matrix) Run another ARKitSession with a WorldTrackingProvider and a CameraFrameProvider in a different library (that is part of the same app) using the ARKit C API and using the transforms from the anchors in that session to render objects in the Swift application part. In general, is this a supported use case or is it necessary to have one shared ARKitSession? Assuming this is supported, will the (device) anchors from both WorldTrackingProviders reference the same world coordinate system? Are there any performance downsides to having multiple ARKitSessions? Thanks

Hello there, I'm currently working on a Hand Tracking System. I've already placed some spheres on some joint points on the left and right hand. Now I want to access the translation/position value of these entities in the update(context: Scene) function. Now my question is, is it possible to access them via .handAnchors(), or which types of .handSkeleton.joint(name) are referencing the same entity? (E.g. is AnchorEntity(.hand(.right, location: .indexFingerTip)) the same as handSkeleton.joint(.indexFingerTip). The goal would be to access the translation of the joints where a sphere has been placed per hand and to be able to update the data every frame through the update(context) function. I would very much appreciate any help! See code example down below: ImmersiveView.swift import SwiftUI import RealityKit import ARKit struct ImmersiveView: View { public var body: some View { RealityView { content in /* HEAD */ let headEntity = AnchorEntity(.head) content.add(headEntity) /* LEFT HAND */ let leftHandWristEntity = AnchorEntity(.hand(.left, location: .wrist)) let leftHandIndexFingerEntity = AnchorEntity(.hand(.left, location: .indexFingerTip)) let leftHandWristSphere = ModelEntity(mesh: .generateSphere(radius: 0.02), materials: [SimpleMaterial(color: .red, isMetallic: false)]) let leftHandIndexFingerSphere = ModelEntity(mesh: .generateSphere(radius: 0.01), materials: [SimpleMaterial(color: .orange, isMetallic: false)]) leftHandWristEntity.addChild(leftHandWristSphere) content.add(leftHandWristEntity) leftHandIndexFingerEntity.addChild(leftHandIndexFingerSphere) content.add(leftHandIndexFingerEntity) } } } TrackingSystem.swift import SwiftUI import simd import ARKit import RealityKit public class TrackingSystem: System { static let query = EntityQuery(where: .has(AnchoringComponent.self)) private let arKitSession = ARKitSession() private let worldTrackingProvider = WorldTrackingProvider() private let handTrackingProvider = HandTrackingProvider() public required init(scene: RealityKit.Scene) { setUpSession() } private func setUpSession() { Task { do { try await arKitSession.run([worldTrackingProvider, handTrackingProvider]) } catch { print("Error: \(error)") } } } public func update(context: SceneUpdateContext) { guard worldTrackingProvider.state == .running && handTrackingProvider.state == .running else { return } let _ = context.entities(matching: Self.query, updatingSystemWhen: .rendering) if let avp = worldTrackingProvider.queryDeviceAnchor(atTimestamp: currentTime) { let hands = handTrackingProvider.handAnchors(at: currentTime) ... } } }

Devices running iOS 18 using RealityKit do not seem to receive lighting supplied via ARKit Environment Texturing (https://developer.apple.com/documentation/arkit/arworldtrackingconfiguration/2977509-environmenttexturing). Instead just a default IBL is used by RealityKit. This happens with RealityView as well as ARView. It also happens when I explicitly opt-in to environment texturing: let worldTrackingConfig = ARWorldTrackingConfiguration() worldTrackingConfig.environmentTexturing = .automatic arView.session.run(worldTrackingConfig) Even the Xcode AR Template has this issue. I'm attaching a screenshot of the sample app running on iOS 18 where it's broken and from iOS 17 where it works as expected. I hope this can get resolved quickly since I see it as a major regression. Feedback ID: FB15091335 UPDATE: It works on my older iPhone XS (iOS 18 22A5282m) Broken on iPad Pro (11-inch) (3rd generation) (iPadOS 18.0 (22A5350a)) Maybe it's related to LiDAR? Thank you! iOS 17 (works): iOS 18 (broken):

Hello We are exploring the iOS 17 RoomPlan updates that allow for a custom ARSession to be passed into the RoomCaptureSession via the new initializer. let roomCaptureSession = RoomCaptureSession(arSession: myARSession) Currently we use our ARSession to extract sceneDepth from the ARFrames via the delegate callback. This works prior to activation of the RoomCaptureSession via session.run(configuration). However, when we do call run on the RoomCaptureSession, sceneDepth is no longer present on the incoming ARFrames. Are these mutually exclusive? Should we expect ARFrame depth data to be present when a RoomCaptureSession is running with the shared ARSession?

Is there anyway to reset the scan memory Vision Pro stores on-device, so that upon every new scanning in my application, it starts from scratch rather than getting instantly recognized. In Apple Vision Pro Privacy overview (https://www.apple.com/privacy/docs/Apple_Vision_Pro_Privacy_Overview.pdf), it is stated: "visionOS builds a three-dimensional model to map your surroundings on-device. Apple Vision Pro uses a combination of camera and LiDAR data to map the area around you and save that model on-device. The model enables visionOS to alert you about real-life obstacles, as well as appropriately reflect the lighting and shadows of your physical space. visionOS uses audio ray tracing to analyze your room’s acoustic properties on-device to adapt and match sound to your space. The underlying scene mesh is stored on-device and encrypted with your passcode if one is set" How to access and erase the, and I quote, “underlying scene mesh stored on-device”?

We are experiencing a crash when attempting to capture a high resolution frame. The crash only happens on the A12 and A12X devices (iPhone11,8 or iPad8,7). The crash did occur on older versions of iOS and continues to happen with more recent versions, iOS 17.6 and iPadOS 17.5.1. Any ideas on how to get a high-resolution image & frame from an AR Session using these devices? To reproduce the crash: Use an A12 or A12X device. Other devices have not produced the same result. Setup an AR session that can capture high resolution static images: Create a class that conforms to UIViewController and ARSessionDelegate and connect it to an Interface with an ARView. @IBOutlet var arView: ARView! During view setup configure the AR session: let configuration = ARBodyTrackingConfiguration() if let hiResFormat = ARBodyTrackingConfiguration.recommendedVideoFormatForHighResolutionFrameCapturing { configuration.videoFormat = hiResFormat } self.arView.session.run(configuration) Trigger code to capture the high resolution image (we call this from a button @IBAction: Task { let hiResFrame = try? await self.arView.session.captureHighResolutionFrame() print("crash above. this print never occurs") } The crash occurs with both the sync & async versions of the captureHighResolutionFrame call. Code above crashes in Apple AltruisticBodyPoseKit : 0 libsystem_kernel.dylib 0xc2ec __pthread_kill + 8 1 libsystem_pthread.dylib 0x7c0c pthread_kill + 268 2 libsystem_c.dylib 0x75ba0 abort + 180 3 libsystem_c.dylib 0x74eac err + 282 4 AltruisticBodyPoseKit 0x4b2b0 cva::MatrixData<int, 0ul, 0ul, false>::allocate(unsigned long) (.cold.1) + 42 5 AltruisticBodyPoseKit 0x20dac std::__1::vector<std::__1::pair<cva::Matrix<double, 3u, 1u, false>, cva::Matrix<double, 2u, 1u, false> >, std::__1::allocator<std::__1::pair<cva::Matrix<double, 3u, 1u, false>, cva::Matrix<double, 2u, 1u, false> > > >::vector(unsigned long) + 1186 6 AltruisticBodyPoseKit 0x2058c btr::(anonymous namespace)::EstimatePoseFromCorrespondences(btr::CameraPoseInfo&, btr::Correspondences2d3d const&, bool) + 564 7 AltruisticBodyPoseKit 0x2018c btr::BodyRegistration::RegisterBody(float vector[2] const*, unsigned long, float vector[3] const*, unsigned long, simd_float4x4 const*, unsigned long, simd_float3x3 const*, simd_float4x4 const*) + 1228 8 AltruisticBodyPoseKit 0x4354c -[ABPKCameraRegistration estimateCameraPoseFromMatchingwithImageIntrinsics:imageResolution:joints2d:jointsLifted3D:jointsLifted3DCount:] + 1160 9 ARKitCore 0x131d30 -[AR3DSkeletonRegistrationTechnique _estimateCameraPoseFromMatchingImageData:to3DData:worldTrackingPose:pCameraFromBody:depthData:pScaleOut:] + 396 10 ARKitCore 0x131818 -[AR3DSkeletonRegistrationTechnique requestResultDataAtTimestamp:context:] + 388 11 ARKitCore 0x91fd8 -[ARParentTechnique technique:didOutputResultData:timestamp:context:onTechniques:] + 1400 12 ARKitCore 0x91a28 -[ARParentTechnique technique:didOutputResultData:timestamp:context:] + 112 13 ARKitCore 0x8150c -[ARExposureLightEstimationTechnique requestResultDataAtTimestamp:context:] + 352 14 ARKitCore 0x91fd8 -[ARParentTechnique technique:didOutputResultData:timestamp:context:onTechniques:] + 1400 15 ARKitCore 0x91a28 -[ARParentTechnique technique:didOutputResultData:timestamp:context:] + 112 16 ARKitCore 0xcd9c4 -[ARWorldAlignmentTechnique requestResultDataAtTimestamp:context:] + 1044 17 ARKitCore 0x91fd8 -[ARParentTechnique technique:didOutputResultData:timestamp:context:onTechniques:] + 1400 18 ARKitCore 0x91a28 -[ARParentTechnique technique:didOutputResultData:timestamp:context:] + 112 19 ARKitCore 0x92584 -[ARParentTechnique _submitResultsForTimestamp:context:] + 396 20 ARKitCore 0x90124 __71-[ARParentTechnique requestResultDataAtTimestamp:context:onTechniques:]_block_invoke_3 + 72

Hello, Has anyone had success with implementing object tracking in Unity or adding native tracking capability to the VisionOS project built from Unity? I am working on an application for Vision Pro mainly in Unity using Polyspatial. The application requires me to track objects and make decisions based on tracked object's location. I was able to create an object tracking application on Native Swift, but could not successfully combine this with my Unity project yet. Each separate project (Main Unity app using Polyspatial and the native app on Swift) can successfully build and be deployed onto VisionPro. I know that Polyspatial and ARFoundation does not have support for ARKit's object tracking feature for VIsion Pro as of today, and they only support image tracking inside Unity. For that reason I have been exploring different ways of creating a bridge for two way interaction of the native tracking functionality and the other functionality in Unity. Below are the methods I tried and failed so far: Package the tracking functionality as a Swift Plugin and access this in Unity, and then build for Vision Pro: I can create packages and access them for simple exposed variables and methods, but not for outputs and methods from ARKit, which throw dependency errors while trying to make the swift package. Build project from Unity to VIsion Pro and expose a boolean to start/stop tracking that can be read by the native code, and then carry the tracking classes into the built project. In this approach I keep getting an error that says _TrackingStateChanged cannot be found, which is the class that exposes the bool toggled by the Unity button press: using System.Runtime.InteropServices; public class UnityBridge { [DllImport("__Internal")] private static extern void TrackingStateChanged(bool isTracking); public static void NotifyTrackingState() { // Call the Swift method TrackingStateChanged(TrackingStartManager.IsTrackingActive()); } } This seems to be translated to C++ code in the ill2cpp output from Unity, and even though I made sure that all necessary packages were added to the target, I keep receiving this error. from the UnityFramework plugin: Undefined symbol: _TrackingStateChanged I have considered extending the current Image Tracking approach in ARFoundation to include object tracking, but that seems to be too complicated for my use case and time frame for now. The final resort will be to forego Unity implementation and do everything in native code. However, I really want to be able to use Unity's conveniences and I have very limited experience with Swift development.

arScnView = ARSCNView(frame: CGRect.zero, options: nil) arScnView.delegate = self arScnView.automaticallyUpdatesLighting = true arScnView.allowsCameraControl = true addSubview(arScnView) arSession = arScnView.session arSession.delegate = self config = ARWorldTrackingConfiguration() config.sceneReconstruction = .meshWithClassification config.environmentTexturing = .automatic func session(_ session: ARSession, didAdd anchors: [ARAnchor]) { anchors.forEach({ anchor in if let meshAnchor = anchor as? ARMeshAnchor { let node = meshAnchor.toSCNNode() self.arScnView.scene.rootNode.addChildNode(node) } if let environmentProbeAnchor = anchor as? AREnvironmentProbeAnchor { // Can I retrieve the texture map corresponding to ARMeshAnchor from Environment Probe Anchor? // Or how can I retrieve the texture map corresponding to ARMeshAnchor? } }) } How can I scan a 3D scene and save it as USDZ? I want to achieve the following scenario?

The light of RealityView can only be effective on virtual objects. I hope it can be projected into the real world. What API can be implemented?

We are attempting to update the texture on a node. The code below works correctly when we use a color, but it encounters issues when we attempt to use an image. The image is available in the bundle, and it image correctly in other parts of our application. This texture is being applied to both the floor and the wall. Please assist us with this issue." for obj in Floor_grp[0].childNodes { let node = obj.flattenedClone() node.transform = obj.transform let imageMaterial = SCNMaterial() node.geometry?.materials = [imageMaterial] node.geometry?.firstMaterial?.diffuse.contents = UIColor.brown obj.removeFromParentNode() Floor_grp[0].addChildNode(node) }

When running a modified version of the RoomPlan Demo I get frequent Session Interrupted conditions. In looking at the traces I find a status of SensorDidPause in the interruption Side of the error but am mystified as to how to determine which sensor it was that paused and how to diagnose it. It appears there is a bitmap of available and active sensor devices in the sensor info passed with the session data on the error. In looking at the error status I can see that one or two of the motion sensors have had a problem. How do I do further diagnostic checks on what the cause of the error is? I am also curious why the error occurred as soon as the AR Session for my test started via the “session.run” call. The documentation in this area seems difficult to find. Attached are traces from running the test and stack dumps for the calls. Please send me guidance on how to proceed. The device in question is an iPad iPhone(3) that is attached to the Mac mini named “Hawkeye”. There is no known direct involvement for the Hawkeye system

I'd like to capture the room with materials obtained through the camera while scanning with RoomPlan. Is there any way to capture room surface material and render the object while capturing the room geometry using RoomPlan?