How can I get all joints XYZ coordinates in ARKit 3 motion capture with swift

I am trying to write to get all joint XYZ coordinates in ARKit 3 motion capture to get the XYZ to find all angles of body. but I can't find the XYZ coordinates so I can't do the next step.

I am trying to use

character!.jointTransforms
to get the joint data translation maybe is the XYZ coordinates but I tried to track a few motion but the numbers didn't change.


This is my code :

import UIKit
import RealityKit
import ARKit
import Combine

class ViewController: UIViewController, ARSessionDelegate {

  @IBOutlet var arView: ARView!

  var character: BodyTrackedEntity?
  let characterOffset: SIMD3 = [0.0, 0, 0]
  let characterAnchor = AnchorEntity()

  override func viewDidAppear(_ animated: Bool) {
  super.viewDidAppear(animated)
  arView.session.delegate = self

  guard ARBodyTrackingConfiguration.isSupported else {
  fatalError("This feature is only supported on devices with an A12 chip")
  }

  let configuration = ARBodyTrackingConfiguration()
  arView.session.run(configuration)

  arView.scene.addAnchor(characterAnchor)

  // Asynchronously load the 3D character.
  var cancellable: AnyCancellable? = nil
  cancellable = Entity.loadBodyTrackedAsync(named: "character/robot").sink(
  receiveCompletion: { completion in
  if case let .failure(error) = completion {
  print("Error: Unable to load model: \(error.localizedDescription)")
  }
  cancellable?.cancel()
  }, receiveValue: { (character: Entity) in
  if let character = character as? BodyTrackedEntity {
  // Scale the character to human size
  character.scale = [1.0, 1.0, 1.0]
  self.character = character
  cancellable?.cancel()
  } else {
  print("Error: Unable to load model as BodyTrackedEntity")
  }
  })
  }

  func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
  for anchor in anchors {
  guard let bodyAnchor = anchor as? ARBodyAnchor else { continue }

  let bodyPosition = simd_make_float3(bodyAnchor.transform.columns.3)
  characterAnchor.position = bodyPosition + characterOffset
  characterAnchor.orientation = Transform(matrix: bodyAnchor.transform).rotation

  if let character = character, character.parent == nil {
  characterAnchor.addChild(character)
  }

  let XYZArray = character!.jointTransforms.map({ (transform: RealityKit.Transform) -> SIMD3 in
  transform.translation
  })

  print(XYZArray)
  }
  }
}


The code is reference by :

https://developer.apple.com/documentation/arkit/capturing_body_motion_in_3d

Replies

Hello,


You can use the approach mentioned in this post: https://forums.developer.apple.com/thread/129766


The joint data is available on the body anchor's skeleton.

Thank you for your reply!


let right_hand_joint = bodyAnchor.skeleton.modelTransform(for: ARSkeleton.JointName(rawValue: "right_hand_joint"))


I am using this code to get right hand joint data but I got another question.

What is the meaning of these data and how can I Transform the data to XYZ coordinates?


This is the right hand joint data I got :

simd_float4x4([[-0.43996024, -0.03083223, -0.8974876, 0.0], [-0.78324467, 0.50204724, 0.36670947, 0.0], [0.43927482, 0.86429024, -0.24503005, 0.0], [-0.21202339, 0.31473792, 0.28043795, 1.0]])

Hello,


I recommend that you do some research on the web about 4x4 transform matrices. Understanding 4x4 transforms is essential to working in 3D.


The last column in the matrix contains the translation data (i.e. xyz coordinates). So, in the example output you posted, the coordinates are as follows:


x = -0.26797074

y = -4.768371e-09

z = -3.814697e-08


Keep in mind that the units for this data is in meters.

Hello


Thank you so much for your reply. However, I have edited the post (putting the 4x4 matrix in model transformation rather than that in local transformation) before you replied. I am planning to use the model transformation (i.e. getting the coordinates in relation to the hip joint)


Would you mind to have a look, in this set of data which is in model transformation, how should I do to convert this data into xyz coordinates?


simd_float4x4([[-0.43996024, -0.03083223, -0.8974876, 0.0], [-0.78324467, 0.50204724, 0.36670947, 0.0], [0.43927482, 0.86429024, -0.24503005, 0.0], [-0.21202339, 0.31473792, 0.28043795, 1.0]])


Is this correct?

X: -0.21202339

Y: 0.31473792

Z: 0.28043795


Thank you for your suggestions. I will definitely read some more on 4x4 transformation matrices.

Yes, that is the xyz of your joint relative to the hip joint.

we may be working on similar output for different intent you interested in collaborating?