Is there ANY documentation or resource that explains how to implement motion capture in Scenekit?
Accepted Reply
You need to make use of the ARBodyAnchor that is returned in session(_:didUpdate:) during an ARBodyTrackingConfiguration. Specifically, you would use the jointModelTransforms of the ARSkeleton3D to update the positions of your geometry.
For example, the following snippet places an SCNSphere at every joint:
fileprivate func addSphere(for jointTransform: simd_float4x4) {
let node = SCNNode()
let geometry = SCNSphere(radius: 0.01)
geometry.firstMaterial?.diffuse.contents = UIColor(displayP3Red: .random(in: 0...1), green: .random(in: 0...1), blue: .random(in: 0...1), alpha: 1)
node.geometry = geometry
node.simdTransform = jointTransform
bodyNode.addChildNode(node)
jointNodes.append(node)
}
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
let bodyAnchors = anchors.compactMap { $0 as? ARBodyAnchor }
guard let bodyAnchor = bodyAnchors.first else { return }
print("body anchor")
bodyNode = SCNNode()
bodyNode.simdWorldTransform = bodyAnchor.transform
bodyAnchor.skeleton.jointModelTransforms.forEach {
addSphere(for: $0)
}
sceneView.scene.rootNode.addChildNode(bodyNode)
}
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
let bodyAnchors = anchors.compactMap { $0 as? ARBodyAnchor }
guard let bodyAnchor = bodyAnchors.first else { return }
bodyNode.simdWorldTransform = bodyAnchor.transform
bodyAnchor.skeleton.jointModelTransforms.enumerated().forEach {
jointNodes[$0].simdTransform = $1
}
}
Replies
You need to make use of the ARBodyAnchor that is returned in session(_:didUpdate:) during an ARBodyTrackingConfiguration. Specifically, you would use the jointModelTransforms of the ARSkeleton3D to update the positions of your geometry.
For example, the following snippet places an SCNSphere at every joint:
fileprivate func addSphere(for jointTransform: simd_float4x4) {
let node = SCNNode()
let geometry = SCNSphere(radius: 0.01)
geometry.firstMaterial?.diffuse.contents = UIColor(displayP3Red: .random(in: 0...1), green: .random(in: 0...1), blue: .random(in: 0...1), alpha: 1)
node.geometry = geometry
node.simdTransform = jointTransform
bodyNode.addChildNode(node)
jointNodes.append(node)
}
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
let bodyAnchors = anchors.compactMap { $0 as? ARBodyAnchor }
guard let bodyAnchor = bodyAnchors.first else { return }
print("body anchor")
bodyNode = SCNNode()
bodyNode.simdWorldTransform = bodyAnchor.transform
bodyAnchor.skeleton.jointModelTransforms.forEach {
addSphere(for: $0)
}
sceneView.scene.rootNode.addChildNode(bodyNode)
}
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
let bodyAnchors = anchors.compactMap { $0 as? ARBodyAnchor }
guard let bodyAnchor = bodyAnchors.first else { return }
bodyNode.simdWorldTransform = bodyAnchor.transform
bodyAnchor.skeleton.jointModelTransforms.enumerated().forEach {
jointNodes[$0].simdTransform = $1
}
}
Hey gchiste
Thanks for the replay. The sphere works just fine, but when I bring my 3D mesh in, the whole mesh just messed up. I guess the order of joints affects the transform? However, when I use jointLocalTransforms in session(_:didUpdate) it seems works fine. Could I get a hint of why it's not working with 3D models?
Hello GerryGGG,
It's not clear to me exactly what you mean by "when I bring my 3D mesh in, the whole mesh just messed up", but I recommend that you request technical support for this issue at https://developer.apple.com/support/technical/ since any solution for your issue is likely to be specific to your app, and may not be useful for other developers.