Retrieve Normal Vector of Tap location? RayCast from device(head)?

Question

The Location3D that is returned by a SpatialTapGesture does not return normal vector information. This can make it difficult to orient an object that's placed at that location.

Am I misusing this gesture or is this indeed the case?

As an alternative I was thinking I could manually raycast toward the location the user tapped, but to do that, I need two points. One of those points needs to be the location of the device / user's head in world space and I'm not familiar how to get that information.

Has anyone achieved something like this?

visionOS

566

Posted by

J0hn

Reply

Add a Comment

Answer 1

Alright. Good riddance. This worked for me:

NOTE: There's a BIT of oddness with raycasting to a tap gesture's location. Sometimes it fails, which is confusing to me given the tap succeeded. Maybe I'm not converting the locations correctly? Maybe it works better on device?

In a tap gesture handler, get the tap location on a collision shape with:

let worldPosition: SIMD3<Float> = value.convert(value.location3D, from: .local, to: .scene)

With a running WorldTrackingProvider you can get the current device pose with

worldTracking.queryDeviceAnchor(atTimestamp: CACurrentMediaTime())

Then process it like so to get it world-space:

let transform = Transform(matrix: pose.originFromAnchorTransform)
let locationOfDevice = transform.translation

You can then do a raycast to a tap location in world-coordinate-space like so:

let raycastResult = scene.raycast(from: locationOfDevice, to: worldPosition)

If successful, an entry in the raycast result will have normal information. Here I grab the first one

guard let result = raycastResult.first else {
    print("NO RAYCAST HITS?????")
}

let normal = result.normal

Make a quaternion to rotate from identity to the normal vector's angle:

// Calculate the rotation quaternion to align the forward axis with the normal vector
let rotation = simd_quatf(from: SIMD3<Float>(0, 1, 0), to: normal)

Apply it to an entity:

cylinder.transform.rotation = rotation

Posted by

J0hn

Add a Comment

Answer 2

The normal vector of the mesh is fundamentally sensitive to measurement errors. Therefore, if the measurement error is large, the spatial range for calculating the average normal vector at the tap location must be increased.

Additionally, real-life objects have sizes. The size can be approximately expressed as the radius of surface curvature. For example, the radius of a cylinder. Therefore, the spatial extent for calculating the normal vector at the tap location must be increased proportional to the radius of curvature of the object surface. The radius of curvature of a plane is very large.

Currently, it appears to take a normal vector of one vertex at the tap location. This is quite unstable information.

In summary,

The normal vector calculation range near the tap location must be expanded in proportion to the measurement error.
The normal vector calculation range must be expanded in proportion to the radius of curvature of the target object surface at the tap location.
In summary, one solution may be to take the average value of the normal vector of several vertices near the tap location.

App developers must implement it themselves. More preferably, Apple, which has information about measurement errors, should provide a solution.

Posted by

JoonAhn

How did you raycast from the device’s position when you did your tests?

—
J0hn

Add a Comment

Answer 3

The ray casting implemented in a CurvSurf FindSurface demo app is as follows.

The basic information needed for ray casting is:

Ray origin
Ray direction
Ray casting target domain.

In the CurvSurf FindSurface demo app:

Ray origin: Current position of device
Ray direction: Center of device screen (has eventually 6-DOF)
Ray casting target domain: 3D measurement point (point cloud, or vertex points of mesh).

pickPoint() sets up a viewing cone with the ray direction as its axis, and selects the point closest to the device among the points inside the viewing cone. If there are no point inside the viewing cone, the point closest to the viewing cone is selected.

https://github.com/CurvSurf/FindSurface-GUIDemo-iOS/blob/main/ARKitFindSurfaceDemo/ViewController.swift

https://github.com/CurvSurf/FindSurface-GUIDemo-iOS/blob/main/ARKitFindSurfaceDemo/Helper.swift

Posted by

JoonAhn

Wow. There's a lot I can learn from all that ARKit code. On VisionOS, though, what are we able to use in place of the "Camera transform" that you use as the ray origin?

—
J0hn

Add a Comment

Answer 4

let cameraTransform = camera.transform // Right-Handed

let rayDirection = -simd_make_float3( cameraTransform.columns.2 )

let rayOrigin = simd_make_float3( cameraTransform.columns.3 )

There is definitely no alternative.

camera.transform is the most accurate real-time info about camera's 6DOF under ARKit.

Posted by

JoonAhn

Add a Comment

Answer 5

3D point cloud processing is a really difficult problem.

It truly requires a lot of specialized knowledge; Linear algebra, algebra, geometry, differential geometry, probability and statistics, optimization, 3D measurement practice, software practice, ...

This is a problem that cannot be solved with expertise in just one field.

I have expressed my views as I thought about them since the fall of 2018.

https://laserscanningforum.com/forum/viewtopic.php?t=13716

It will be helpful if you read it carefully until the end.

Posted by

JoonAhn

Add a Comment

Answer 6

Alright. Good riddance. This worked for me:

NOTE: There's a BIT of oddness with raycasting to a tap gesture's location. Sometimes it fails, which is confusing to me given the tap succeeded. Maybe I'm not converting the locations correctly? Maybe it works better on device?

In a tap gesture handler, get the tap location on a collision shape with:

let worldPosition: SIMD3<Float> = value.convert(value.location3D, from: .local, to: .scene)

With a running WorldTrackingProvider you can get the current device pose with

worldTracking.queryDeviceAnchor(atTimestamp: CACurrentMediaTime())

Then process it like so to get it world-space:

let transform = Transform(matrix: pose.originFromAnchorTransform)
let locationOfDevice = transform.translation

You can then do a raycast to a tap location in world-coordinate-space like so:

let raycastResult = scene.raycast(from: locationOfDevice, to: worldPosition)

If successful, an entry in the raycast result will have normal information. Here I grab the first one

guard let result = raycastResult.first else {
    print("NO RAYCAST HITS?????")
}

let normal = result.normal

Make a quaternion to rotate from identity to the normal vector's angle:

// Calculate the rotation quaternion to align the forward axis with the normal vector
let rotation = simd_quatf(from: SIMD3<Float>(0, 1, 0), to: normal)

Apply it to an entity:

cylinder.transform.rotation = rotation

Posted by

J0hn

Add a Comment

Retrieve Normal Vector of Tap location? RayCast from device(head)?

Accepted Reply

Replies