Hand interaction with ARKit Scene.

What is the best way to interact with hands with ARKit scene. I decided to use Vision's hand detection to identify coordinates and then do raycasting. is there any other solution to do that. I want to subscribe to collisions of hand and 3d models.
Up vote post of elzhanerkeaccount Down vote post of elzhanerkeaccount
2.4k views
Posted by
elzhanerkeaccount
Reply
Add a Comment

Accepted Reply

In order to interact with 3D objects, the hand would likely need to be represented in 3D. However, Vision's hand pose estimation provides landmarks in 2D only.
Posted by
Frameworks Engineer
Up vote reply of Frameworks Engineer Down vote reply of Frameworks Engineer
Post marked as solved
Add a Comment

Replies

In order to interact with 3D objects, the hand would likely need to be represented in 3D. However, Vision's hand pose estimation provides landmarks in 2D only.
Posted by
Frameworks Engineer
Up vote reply of Frameworks Engineer Down vote reply of Frameworks Engineer
Post marked as solved
Add a Comment
By combining some API, you may be able to get exactly what you're after.

You would first use the x and y coordinates from the 2D landmarks in Vision's hand pose observations. Then get an estimated z coordinate for each point by analyzing the depth texture in the frame.

If you're using the front facing camera, you can get this through the TrueDepth data. If you're using a new iPad Pro with LiDAR, you can get at the depth data through the new API available in ARKit4 as noted in these examples:

Visualizing a Point Cloud Using Scene Depth
Creating a Fog Effect Using Scene Depth

Once you have the 3D points, you can associate some spherical colliders onto them and check for collision with other virtual objects in the scene.
Posted by
Technology Evangelist
Post not yet marked as solved Up vote reply of Technology Evangelist Down vote reply of Technology Evangelist
Add a Comment