Hand Tracking Palm towards face or not

Question

teezy_dev OP

Created Sep ’24

Replies 1

Boosts 0

Participants 2

Hi all, I’m quite new to XR development in general and need some guidance.

I want to create a function that simply tells me if my palm is facing me or not (returning a bool), but I honestly have no idea where to start. I saw an earlier Reddit post about 6 months that essentially wanted the same thing I need, but the only response was this:

Consider a triangle made up of the wrist, thumb knuckle, and little finger metacarpal (see here for the joints, and note that naming has changed slightly since this WWDC video): the orientation of this triangle (i.e., whether the front or back is visible) seen from the device location should be a very exact indication of whether the user’s palm is showing or not.

While I really like this solution, I genuinely have no idea how to code it, and no further code was provided. I’m not asking for the entire implementation, but rather just enough to get me on the right track.

Heres basically all I have so far (no idea if this is correct or not):

func isPalmFacingDevice(hand: HandSkeleton, devicePosition: SIMD3<Float>) -> Bool {
        // Get the wrist, thumb knuckle and little finger metacarpal positions as 3D vectors
        let wristPos = SIMD3<Float>(hand.joint(.wrist).anchorFromJointTransform.columns.3.x,
                                    hand.joint(.wrist).anchorFromJointTransform.columns.3.y,
                                    hand.joint(.wrist).anchorFromJointTransform.columns.3.z)

        let thumbKnucklePos = SIMD3<Float>(hand.joint(.thumbKnuckle).anchorFromJointTransform.columns.3.x,
                                           hand.joint(.thumbKnuckle).anchorFromJointTransform.columns.3.y,
                                           hand.joint(.thumbKnuckle).anchorFromJointTransform.columns.3.z)

        let littleFingerPos = SIMD3<Float>(hand.joint(.littleFingerMetacarpal).anchorFromJointTransform.columns.3.x,
                                           hand.joint(.littleFingerMetacarpal).anchorFromJointTransform.columns.3.y,
                                           hand.joint(.littleFingerMetacarpal).anchorFromJointTransform.columns.3.z)

}

Boost

Answer 1

Vision Pro Engineer OP

Apple

Sep ’24

Hi @teezy_dev

I think I understand the idea of your approach: create a triangle from three points, use the orientation of the triangle (the normal of the face created?) to check if its facing the user or not. This may get you there, but I think there is a simpler solution, although it involves a bit of matrix math.

Additionally, the coordinate systems of the left and right hand anchors have different chiralities. What this means is, if you imagine your hands resting on the keyboard in front of you, the local y axis of your left hand points toward the floor and the local y axis of your right hand points toward the ceiling. These mirrored coordinate spaces simplify workflows when creating animations for both hands, but it does add complexity we will need to account for here.

Here's my version of your isPalmFacingDevice function. Notice we need a simd_float4x4 matrix instead of a HandSkeleton. We also need information about chirality:

func isPalmFacingDevice(handAnchorMatrix: simd_float4x4, devicePosition: SIMD3<Float>, chirality: HandAnchor.Chirality) -> Bool {
    // convert the world-space position of the device into the local-space of the anchor
    let deviceLocalPos = simd_make_float4(devicePosition) * handAnchorMatrix
    switch chirality {
        case .right:
            return deviceLocalPos.y < 0
        case .left:
            return deviceLocalPos.y > 0
    }
}

The simd_float4x4 matrix is obtained from HandTrackingProvider anchorUpdates, for example:

for await update in handTrackingProvider.anchorUpdates {
    let handAnchorMatrix = update.anchor.originFromAnchorTransform
    // ...
}

Let me know if you have any questions about this solution! Matrix math can be tricky.

1