I had the same problem. Very strange that there is no obvious and easy way to handle this sort of interaction. I wasted a lot of time here because it seems like the 2D location field of the drag gesture is still affected by the camera orientation in 3D space. I ended up using the location3D field, removing view rotation (e.g. transform to local space), calculating the horizontal offset, and reapplying the view rotation: .gesture(DragGesture(minimumDistance: 0.0).targetedToAnyEntity().onChanged {e in if refs.entityDragStartPos == nil { refs.entityDragStartPos = e.entity.position } // Get the 3D drag delta let delta = simd_float3(e.gestureValue.location3D.vector) - simd_float3(e.gestureValue.startLocation3D.vector) // Get the look rotation let direction = e.inputDevicePose3D!.rotation.quaternion.act(simd_double3(0, 0, 1)) let lookRot = simd_quatf(from: SIMD3<Float>(0, 0, 1), to: simd_normalize(simd_float3(Float(direction.x), 0, Float(direction.z)))) // Get the 3D drag delta in local space (e.g. independent of view direction) // by rotating using inverse look rotation let drag = lookRot.inverse.act(delta) // Calculate offset, y drag is applied to z axis to only get horizontal. // Rotate back into view space and calcualte new entity position let offset = lookRot.act(simd_float3(drag.x, 0, -(drag.y - drag.z))) * 0.001 let pos = refs.entityDragStartPos! + offset e.entity.setPosition(pos, relativeTo: nil) }.onEnded { e in refs.entityDragStartPos = nil })