RealityKit Skeleton Tracking: Hand positions were off quite a bit. Did I do something wrong?

I wrote/copied a code to show major body joint locations. The goal was to track hand location. The hand locations seem off quite a bit. Please let me know what I could change to make it better.

My initial impression was that the skeleton seemed not adjusted to a person's body type - in my case, I was shorter than standard skeleton model.


Device: iPad Pro (11-inch) (2nd generation)
Software Version: 14.0 (18A373)
XCode Version: 12.0(12A7209)
iOS Deployment Target: 14.0



I was standing there without obstruction and I was not moving for a couple of seconds when taking screenshot. I could not attach screenshot. But by my visual estimation, the hand joint locations were about 10-15cm away.

Here is how I coded it -

1> create an entity for each joint interested
2> they are all added to a "VisualSkeleton" (extension of Entity) object
3> Create an AnchorEntity and Place this Entity to the anchorEntity;
4> Refresh each ModelEntity's location based on corresponding joint's location

Configurating ...
Code Block swift
        // Run a body tracking configration.
        let configuration = ARBodyTrackingConfiguration()
        configuration.automaticImageScaleEstimationEnabled = true
        configuration.automaticSkeletonScaleEstimationEnabled = true
        arView.session.run(configuration)


Calculates joint positions
Code Block swift
    func update(with bodyAnchor: ARBodyAnchor) {
        let rootPosition = simd_make_float3(bodyAnchor.transform.columns.3)
        let skeleton = bodyAnchor.skeleton
        
        //rootAnchor.position = rootPosition
        //rootAnchor.orientation = Transform(matrix: bodyAnchor.transform).rotation
        
        for (jointName, jointEntity) in joints {
            if let jointTransform = skeleton.modelTransform(for: ARSkeleton.JointName.init(rawValue: jointName)) {
                let jointOffset = simd_make_float3(jointTransform.columns.3)
                jointEntity.position = rootPosition + jointOffset // rootPosition
                jointEntity.orientation = Transform(matrix: jointTransform).rotation
            }
        }
        
        if self.parent == nil {
            rootAnchor.addChild(self)
        }
    }


I will be happy to share more code if needed. Thank you so much!



Just want to add to the description: the direction of arms and hands looks accurate. But the skeleton model of arm/body length was off.

Hi mate. Have you been able to resolve that issue? If so, could you please share some more code? Thanks.

RealityKit Skeleton Tracking: Hand positions were off quite a bit. Did I do something wrong?
 
 
Q