RealityKit: Update joints' tranform in BodyTrackedEntity

Question

Created Jan ’21

Replies 2

Boosts 0

Views 1.2k

Participants 2

Is there any way to "manually" update the joint's transform in BrodyTrackedEntity? It seems that current api did this automatically.

More context:
I am trying to replay the robot using the ARBodyAnchor stored from previous session. I get the information of joint's transform from the ARBodyAnchor and update the corresponding joints' transform in "BodyTrackedEntity.jointTransforms". However, this doesn't work. The joints are all over the place.

Any suggestions or possible direction I could look for is helpful.

Answered by mhwu in 660147022

[UPDATE] I finally figure this out and think this is a bug in RealityKit.

I will walk through my solution below.

Thinking process:
I found two properties in "BodyTrackedEntity": jointNames and jointTransforms. Therefore, to change jointTransform's position, what we need is to get the up-to-date joint's transforms from ARBodyAnchor and update the corresponding jointTransform in "BodyTrackedEntity". See below.

Code Block SWIFTextension BodyTrackedEntity {
  func updateJointsTransform(with anchor: ARBodyAnchor) {
    for (idx, jointName) in self.jointNames.enumerated() {
      guard let name = jointName.split(separator: "/").last else { print("Invalid name"); return }
      guard let anchorTransform = anchor.skeleton.localTransform(for: ARSkeleton.JointName(rawValue: String(name))) else { print("can't find"); return}
      self.jointTransforms[idx] = Transform(matrix: anchorTransform)
    }
}

Based on the official documentation. We should use call jointNames to determine the order of the joints, and this is what I did in the solution above. However, as mentioned in previous post, the robot's joint's are all over the place.

  /// - Remark:
  ///   Call jointNames to determine the name and order of the joints.
  /// Note that active animations may override the joint transforms set using this function.*/
  public var jointTransforms: [Transform]

After several trial-and-errors, I foud that the bug is that the joint order in jointNames is WRONG. Therefore, the joints do not appear in the correct location when I use the ARBodyAnchor to update it's transform. The solution could be as simpe as follow.

Code Block SWIFTextension BodyTrackedEntity {
	func updateJointsTransform(with anchor: ARBodyAnchor) {
	  self.jointTransforms = anchor.skeleton.jointLocalTransforms.map { Transform(matrix:$0) }
	}
}

Note: To use this function, you have to set BodyTrackedEntity.bodyTracking.isPaused = true to stop the auto updates.

Boost

Answer 1

mhwu OP

Feb ’21

Accepted Answer

[UPDATE] I finally figure this out and think this is a bug in RealityKit.

I will walk through my solution below.

Thinking process:
I found two properties in "BodyTrackedEntity": jointNames and jointTransforms. Therefore, to change jointTransform's position, what we need is to get the up-to-date joint's transforms from ARBodyAnchor and update the corresponding jointTransform in "BodyTrackedEntity". See below.

Code Block SWIFTextension BodyTrackedEntity {
  func updateJointsTransform(with anchor: ARBodyAnchor) {
    for (idx, jointName) in self.jointNames.enumerated() {
      guard let name = jointName.split(separator: "/").last else { print("Invalid name"); return }
      guard let anchorTransform = anchor.skeleton.localTransform(for: ARSkeleton.JointName(rawValue: String(name))) else { print("can't find"); return}
      self.jointTransforms[idx] = Transform(matrix: anchorTransform)
    }
}

Based on the official documentation. We should use call jointNames to determine the order of the joints, and this is what I did in the solution above. However, as mentioned in previous post, the robot's joint's are all over the place.

  /// - Remark:
  ///   Call jointNames to determine the name and order of the joints.
  /// Note that active animations may override the joint transforms set using this function.*/
  public var jointTransforms: [Transform]

After several trial-and-errors, I foud that the bug is that the joint order in jointNames is WRONG. Therefore, the joints do not appear in the correct location when I use the ARBodyAnchor to update it's transform. The solution could be as simpe as follow.

Code Block SWIFTextension BodyTrackedEntity {
	func updateJointsTransform(with anchor: ARBodyAnchor) {
	  self.jointTransforms = anchor.skeleton.jointLocalTransforms.map { Transform(matrix:$0) }
	}
}

Note: To use this function, you have to set BodyTrackedEntity.bodyTracking.isPaused = true to stop the auto updates.

1

Answer 2

kimate OP

Mar ’21

I am wondering if the same thing is possible for hand joints in a non-skeleton or any joints that are not bodyanchors? I will try this same approach and let you know if it works. If you know how to update a joint already I will gladly pay the courtesy forward.

0