What is the best way to interact with hands with ARKit scene. I decided to use Vision's hand detection to identify coordinates and then do raycasting. is there any other solution to do that. I want to subscribe to collisions of hand and 3d models.
Hand interaction with ARKit Scene.
In order to interact with 3D objects, the hand would likely need to be represented in 3D. However, Vision's hand pose estimation provides landmarks in 2D only.
By combining some API, you may be able to get exactly what you're after.
You would first use the x and y coordinates from the 2D landmarks in Vision's hand pose observations. Then get an estimated z coordinate for each point by analyzing the depth texture in the frame.
If you're using the front facing camera, you can get this through the TrueDepth data. If you're using a new iPad Pro with LiDAR, you can get at the depth data through the new API available in ARKit4 as noted in these examples:
Visualizing a Point Cloud Using Scene Depth
Creating a Fog Effect Using Scene Depth
Once you have the 3D points, you can associate some spherical colliders onto them and check for collision with other virtual objects in the scene.
You would first use the x and y coordinates from the 2D landmarks in Vision's hand pose observations. Then get an estimated z coordinate for each point by analyzing the depth texture in the frame.
If you're using the front facing camera, you can get this through the TrueDepth data. If you're using a new iPad Pro with LiDAR, you can get at the depth data through the new API available in ARKit4 as noted in these examples:
Visualizing a Point Cloud Using Scene Depth
Creating a Fog Effect Using Scene Depth
Once you have the 3D points, you can associate some spherical colliders onto them and check for collision with other virtual objects in the scene.