Simpler hierarchy for capturing body motion in 3D

I am looking into this example of capturing body motion in 3D:

https://developer.apple.com/documentation/arkit/capturing_body_motion_in_3d


The example given is using hi-fidelity skeletons hierarchy as USDZ. So I give it a look at the skeleton inside XCode, and trying to replicate the skeleton hierarchy hoping that my MUCH SIMPLER 3D character skeleton will just work. I have maybe just 20-30% of all the skeletons. I omit some nodes.


But my character does not work, it seems to give error like could not find skeleton X.


2019-07-15 22:36:43.985210+1000 BodyDetection[4434:181726] [API] Created entity character does not contain supplied joint name: spine_4_joint.

2019-07-15 22:36:43.985257+1000 BodyDetection[4434:181726] [API] Cannot get joint count for non-character entity.

Error: Unable to load model: The operation couldn’t be completed. (RealityKit.Entity.LoadError error 4.)


Is it actually possible to supply a USDZ character with less skeleton, or mayb just from chest and above, or maybe even just left hand and right hand?


Thanks!

Replies

My guess is that with Apple example, the whole skeleton hierarchy is loaded, but I only want a few of them, and trying to debunk it ...
UPDATE:
I manage to "extract" the Apple Robot.usdz getting some bone hierarchy etc. Brought it to Blender, skinning my own mesh roughly matching the bones, export my own USDZ, put it back inside the app. But issues: scale is wrong, bone rotation is wrong, orientation is wrong, and the resulting AR is out of whack...

How did you extract the usdz into blender? Did you then export it as fbx file and convert it to usdz? I tried exporting a file from Unity as a usd file but something is wrong with the skeleton naming because when running in Xcode it complains it can't find the skeleton.

Hello! I am trying to accomplish the same thing and finding it hard to extract from usdz to Blender. I tried going from usdz to scn to dae but have not had any success.


Could you please provide insight as to how you extracted the usdz information and brought it in Blender?

Has anyone had success in bringing in any custom-made 3D rigged model into Xcode for use with ARBodyTracking? I've yet to find any documentation that details the exact skeletal structure necessary for this technology, though I imagine something in ARKit must "map" the detected human structure to a 3D model, and therefore, there must be some relevant naming convention.

Also following — is there an accepted workaround for creating/importing a custom model to use with ARKit 3 BodyTracking?

How did you extract the robot.usdz, can you please give us the steps.

Steps to create a model to be used as a BodyTrackedEntity:


  1. Identify a 3d content creation software package that supports the authoring and editing of linear blend skinning on polygonal meshes, and supports the import of usdz, usda, or usdc files. Searching the internet for ‘USD 3rd Party Plugins’ will yield several results.
  2. Import the robot.usdz model (from https://developer.apple.com/documentation/arkit/capturing_body_motion_in_3d) into the software package you are using. If your 3d content creation software does not support the import of usdz files, but does support the import of usdc or usda files you can simply change the name of the robot.usdz file to robot.zip and unzip the file to access the embedded usdc file and the associated textures.
  3. Remove the old geometry (“ace_PLY”), but not the skeleton!
  4. Create your new geometry, pose it in the same T-pose that the original model was posed in (doesn’t have to be posed exactly the same, but should cover all joints).
  5. Do not rotate the joints in the imported skeleton and do not translate the y and z axis of the joints in the imported skeleton.
  6. Parent your new geometry under the node named ‘geom’. Zero the geometry pivots to the origin 0,0,0, and ensure that all scale values for geometry are at scaleX=1, scaleY=1, scaleZ=1.
  7. Bind your new geometry to the skeleton using linear blend skinning, make sure to select all the joints when binding. Before you export your newly bound character, verify that you only have one bind pose present in your scene. Otherwise, delete all existing bind pose nodes in your scene, select every joint, and recreate the bind pose.
  8. Export as a USDZ file (or some other format, and then use Reality Converter to convert to USDZ). Make sure that you specify Meters in your export options! .obj, .gltf, and .fbx are supported by Reality Converter.

If you follow all of these steps correctly, then your model will be properly loaded as a BodyTrackedEntity.

Hi gchiste.

Which 3d content creation software do you suggest in this case?


Thank you for you answer and the steps for creating a custom model here.

Please have a look at this new article https://developer.apple.com/documentation/arkit/rigging_a_model_for_motion_capture, which suggests a few software packages.

Just a note for others - if you do go down

Maya Autodesk 2020 - route


They have prebuilt binaries for USD / and Maya plugins here / ( they're currently not included).

https://github.com/Autodesk/maya-usd/releases/


It doesn't seem like Modo supports usdz out of the box either.

I tried opening ("importing") the robot FBX file in blender, removed ("unlinked") the cube that surrounded the robot, exported as gLTF, opened in Reality Converter (downloadable mac app), exported it as USDZ and imported it to XCode.

Everything works, only thing is that that the robot needs a 90 degree rotation.

Next step is to edit the robot mesh into other characters.. We'll see how that goes :)
  • Hey, how did rotate the robot, from blender or Xcode?

Add a Comment

Hi to all! I have a problem: I run the code as it comes from the zip file, Xcode makes the built, no error, and sends the app to my iphone. The problem is that nothing appears in the Ar view, I can't see the robot.usdz, inside the appledeveloper code. I'm using iPhone 12 pro, iOs 15 beta 3 and X code 13 beta. I've updated everything, so that may be the problem? Is lidar on as default? I don't see any code about lidar inside the code from the zip. Why doesn't the robot even appear to me? Apologise for all that question maybe almost off topic, but the situation It's really frustrating, so any help is very appreciated :)