Motion capture in Scenekit

Is there ANY documentation or resource that explains how to implement motion capture in Scenekit?

Accepted Reply

You need to make use of the ARBodyAnchor that is returned in session(_:didUpdate:) during an ARBodyTrackingConfiguration. Specifically, you would use the jointModelTransforms of the ARSkeleton3D to update the positions of your geometry.


For example, the following snippet places an SCNSphere at every joint:


    fileprivate func addSphere(for jointTransform: simd_float4x4) {
        let node = SCNNode()
        let geometry = SCNSphere(radius: 0.01)
        geometry.firstMaterial?.diffuse.contents = UIColor(displayP3Red: .random(in: 0...1), green: .random(in: 0...1), blue: .random(in: 0...1), alpha: 1)
        node.geometry = geometry
        node.simdTransform = jointTransform
        bodyNode.addChildNode(node)
        jointNodes.append(node)
    }
   
    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        let bodyAnchors = anchors.compactMap { $0 as? ARBodyAnchor }
        guard let bodyAnchor = bodyAnchors.first else { return }
        print("body anchor")
        bodyNode = SCNNode()
        bodyNode.simdWorldTransform = bodyAnchor.transform
        bodyAnchor.skeleton.jointModelTransforms.forEach {
            addSphere(for: $0)
        }
        sceneView.scene.rootNode.addChildNode(bodyNode)
    }
   
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        let bodyAnchors = anchors.compactMap { $0 as? ARBodyAnchor }
        guard let bodyAnchor = bodyAnchors.first else { return }
        bodyNode.simdWorldTransform = bodyAnchor.transform
        bodyAnchor.skeleton.jointModelTransforms.enumerated().forEach {
            jointNodes[$0].simdTransform = $1
        }
    }

Replies

You need to make use of the ARBodyAnchor that is returned in session(_:didUpdate:) during an ARBodyTrackingConfiguration. Specifically, you would use the jointModelTransforms of the ARSkeleton3D to update the positions of your geometry.


For example, the following snippet places an SCNSphere at every joint:


    fileprivate func addSphere(for jointTransform: simd_float4x4) {
        let node = SCNNode()
        let geometry = SCNSphere(radius: 0.01)
        geometry.firstMaterial?.diffuse.contents = UIColor(displayP3Red: .random(in: 0...1), green: .random(in: 0...1), blue: .random(in: 0...1), alpha: 1)
        node.geometry = geometry
        node.simdTransform = jointTransform
        bodyNode.addChildNode(node)
        jointNodes.append(node)
    }
   
    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        let bodyAnchors = anchors.compactMap { $0 as? ARBodyAnchor }
        guard let bodyAnchor = bodyAnchors.first else { return }
        print("body anchor")
        bodyNode = SCNNode()
        bodyNode.simdWorldTransform = bodyAnchor.transform
        bodyAnchor.skeleton.jointModelTransforms.forEach {
            addSphere(for: $0)
        }
        sceneView.scene.rootNode.addChildNode(bodyNode)
    }
   
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        let bodyAnchors = anchors.compactMap { $0 as? ARBodyAnchor }
        guard let bodyAnchor = bodyAnchors.first else { return }
        bodyNode.simdWorldTransform = bodyAnchor.transform
        bodyAnchor.skeleton.jointModelTransforms.enumerated().forEach {
            jointNodes[$0].simdTransform = $1
        }
    }

Hey gchiste

Thanks for the replay. The sphere works just fine, but when I bring my 3D mesh in, the whole mesh just messed up. I guess the order of joints affects the transform? However, when I use jointLocalTransforms in session(_:didUpdate) it seems works fine. Could I get a hint of why it's not working with 3D models?

Hello GerryGGG,


It's not clear to me exactly what you mean by "when I bring my 3D mesh in, the whole mesh just messed up", but I recommend that you request technical support for this issue at https://developer.apple.com/support/technical/ since any solution for your issue is likely to be specific to your app, and may not be useful for other developers.