Thanks for the extended reply @Vision Pro Engineer. I actually approached this in the same way but you added some crucial extra details that I missed. I will add them to my own code and verify. I’ll get back with results and maybe if I get some time make this into a swift package for everyone to use.

Thanks for looking into this @Vision Pro Engineer While waiting I actually tried making the hand bodies dynamic and used a System to update their positions that hand tracking producted. This works fine for a while and seems like an easy workaround because the door behaves perfect. However, after a while and to my surprise, the spheres that are dynamic and represent the fingers end up at different positions than the hand tracking data (some kind of exploded offset which is relatively stable) and never return back despite my code setting their positions to tracking data the same way as at the beginning where it worked. This effect mainly starts to occur when dynamic entities overlap but strangely keeps on going despite me moving the hands away and collision free. I have therefore tried to create a DTS (Case-ID: 8614262) and pointed it to this thread because I have a reproducible case and a video documenting it but that DTS has not yet been handled at all. For me, the DTS / this thread stand for a “recommended / documented way for hand-object interactions” which in my opinion is definitely a common case but is not well communicated/documented. I also briefly tried to work with forces/impulses instead but I could never get rid of heavy oscillations despite my best efforts to dampen things. I can’t be that you need to be an expert in this topic (if I would, I would write my own physics engine).

Ok, I will file a feedback request. I do have a problem with the code though. My own code is very similar (and works in general) but as soon as I have another kinematic entity (like a fingertip or a sphere I move via code) the door CAN be moved and not only rotated. I tried with a PhysicsRevoluteJoint and a custom joint with fixed position. It will move away for a moment when pushed and then gets back to its position when it can and I have not yet found out to actually fix it in place so it only rotates. Is there a way to prevent this?

Here is a test project I made: https://www.transfernow.net/dl/20240804cujYO3mm Link is valid until 11 Aug

An obvious answer I thought of is to quantise the scanned mesh. An additional step of detecting outliers within a plane (as detected by a plane provider) and correcting them. This would result in much stabler meshes that won't move other physics objects when updating the mesh.

I figured it out with the help of an Apple engineer. He basically confirmed that my initial (much simpler) idea was right and so I made a small demo immersive view that shows how to construct physical joints from code: import SwiftUI import RealityKit import RealityKitContent extension Entity { func makePhysical( _ mesh: MeshResource, mode: PhysicsBodyMode, _ linDamp: Float, _ angDamp: Float, _ color: UIColor, _ position: simd_float3, relative: Entity? = nil) -> ModelEntity { let color = SimpleMaterial(color: color, isMetallic: true) let entity = ModelEntity(mesh: mesh, materials: [color]) entity.setPosition(position, relativeTo: relative) self.addChild(entity) let shape = ShapeResource.generateConvex(from: mesh) let material = PhysicsMaterialResource.generate(staticFriction: 100000, dynamicFriction: 100000, restitution: 1) var physicsBody = PhysicsBodyComponent(shapes: [shape], mass: 0.1, material: material, mode: mode) physicsBody.linearDamping = linDamp physicsBody.angularDamping = angDamp entity.components.set(physicsBody) entity.components.set(CollisionComponent(shapes: [shape], isStatic: false)) return entity } } extension PhysicsCustomJoint { static func addFixedJoint(_ entity1: Entity, _ entity2: Entity, _ pos1: simd_float3, _ pos2: simd_float3) { let joint = PhysicsFixedJoint( pin0: entity1.pins.set(named: UUID().uuidString, position: pos1), pin1: entity2.pins.set(named: UUID().uuidString, position: pos2) ) try! joint.addToSimulation() } static func addBallJoint( _ entity1: Entity, _ entity2: Entity, _ pos1: simd_float3, _ pos2: simd_float3) { var joint = PhysicsCustomJoint( pin0: entity1.pins.set(named: UUID().uuidString, position: pos1), pin1: entity2.pins.set(named: UUID().uuidString, position: pos2) ) joint.angularMotionAroundX = .unlimited joint.angularMotionAroundY = .unlimited joint.angularMotionAroundZ = .unlimited joint.linearMotionAlongX = .fixed joint.linearMotionAlongY = .fixed joint.linearMotionAlongZ = .fixed try! joint.addToSimulation() } } struct ImmersiveView: View { var body: some View { RealityView { content in let root = Entity() content.add(root) var simulation = PhysicsSimulationComponent() // simulation.gravity = simd_float3(0, -9.8, 0) simulation.solverIterations = .init(positionIterations: 2, velocityIterations: 2) root.components.set(simulation) let ball1 = root.makePhysical(.generateSphere(radius: 0.05), mode: .kinematic, 0, 0, .red, [0, 1.75, -0.75]) let rod1 = root.makePhysical(.generateCylinder(height: 0.4, radius: 0.025), mode: .dynamic, 0.2, 0, .green, [0, -0.2, 0], relative: ball1) let ball2 = root.makePhysical(.generateSphere(radius: 0.05), mode: .dynamic, 0.2, 0, .blue, [0, 0.2, 0], relative: rod1) let rod2 = root.makePhysical(.generateCylinder(height: 0.4, radius: 0.025), mode: .dynamic, 0.2, 0, .magenta, [0, -0.2, 0], relative: ball2) let ball3 = root.makePhysical(.generateSphere(radius: 0.05), mode: .dynamic, 0.2, 0, .yellow, [0, 0.2, 0], relative: rod2) let rod3 = root.makePhysical(.generateCylinder(height: 0.4, radius: 0.025), mode: .dynamic, 0.2, 0, .cyan, [0, -0.2, 0], relative: ball3) let ball4 = root.makePhysical(.generateSphere(radius: 0.05), mode: .dynamic, 0.2, 0, .black, [0, 0.2, 0], relative: rod3) let block = root.makePhysical(.generateCylinder(height: 2, radius: 0.025), mode: .kinematic, 0, 0, .white, .zero, relative: nil) PhysicsCustomJoint.addBallJoint(ball1, rod1, .zero, [0, 0.2, 0]) PhysicsCustomJoint.addFixedJoint(rod1, ball2, [0, -0.2, 0], .zero) PhysicsCustomJoint.addBallJoint(ball2, rod2, .zero, [0, 0.2, 0]) PhysicsCustomJoint.addFixedJoint(rod2, ball3, [0, -0.2, 0], .zero) PhysicsCustomJoint.addBallJoint(ball3, rod3, .zero, [0, 0.2, 0]) PhysicsCustomJoint.addFixedJoint(rod3, ball4, [0, -0.2, 0], .zero) Timer.scheduledTimer(withTimeInterval: 0.01, repeats: true) { _ in Task { let f = Date().timeIntervalSince1970 * 4 let y = await ball1.position.y var x = Float(sin(f) / 5.0) var z = Float(cos(f) / 5.0) await ball1.setPosition([x, y, z - 0.75], relativeTo: nil) x = Float(sin(-f * 0.97341) / 3.0) z = Float(cos(-f * 0.97341) / 3.0) await block.setPosition([x, y - 0.3, z - 0.75], relativeTo: nil) } } } } }

Thanks, I will report it even though it actually says that ornaments are not supported in immersive views as a red log row in Xcode. Since I reported this I already made my own system that attaches child views to the main view to simulate ornaments which gives me some freedom to place them a bit more strategic too. Thanks for answering so quickly