Retrieve Normal Vector of Tap location? RayCast from device(head)?

The normal vector of the mesh is fundamentally sensitive to measurement errors. Therefore, if the measurement error is large, the spatial range for calculating the average normal vector at the tap location must be increased.

Additionally, real-life objects have sizes. The size can be approximately expressed as the radius of surface curvature. For example, the radius of a cylinder. Therefore, the spatial extent for calculating the normal vector at the tap location must be increased proportional to the radius of curvature of the object surface. The radius of curvature of a plane is very large.

Currently, it appears to take a normal vector of one vertex at the tap location. This is quite unstable information.

In summary,

• The normal vector calculation range near the tap location must be expanded in proportion to the measurement error.
• The normal vector calculation range must be expanded in proportion to the radius of curvature of the target object surface at the tap location.
• In summary, one solution may be to take the average value of the normal vector of several vertices near the tap location.

App developers must implement it themselves. More preferably, Apple, which has information about measurement errors, should provide a solution.

The ray casting implemented in a CurvSurf FindSurface demo app is as follows.

The basic information needed for ray casting is:

• Ray origin
• Ray direction
• Ray casting target domain.

In the CurvSurf FindSurface demo app:

• Ray origin: Current position of device
• Ray direction: Center of device screen (has eventually 6-DOF)
• Ray casting target domain: 3D measurement point (point cloud, or vertex points of mesh).

pickPoint() sets up a viewing cone with the ray direction as its axis, and selects the point closest to the device among the points inside the viewing cone. If there are no point inside the viewing cone, the point closest to the viewing cone is selected.

https://github.com/CurvSurf/FindSurface-GUIDemo-iOS/blob/main/ARKitFindSurfaceDemo/ViewController.swift

https://github.com/CurvSurf/FindSurface-GUIDemo-iOS/blob/main/ARKitFindSurfaceDemo/Helper.swift

let cameraTransform = camera.transform // Right-Handed

let rayDirection = -simd_make_float3( cameraTransform.columns.2 )

let rayOrigin = simd_make_float3( cameraTransform.columns.3 )

There is definitely no alternative.

camera.transform is the most accurate real-time info about camera's 6DOF under ARKit.

3D point cloud processing is a really difficult problem.

It truly requires a lot of specialized knowledge; Linear algebra, algebra, geometry, differential geometry, probability and statistics, optimization, 3D measurement practice, software practice, ...

This is a problem that cannot be solved with expertise in just one field.

I have expressed my views as I thought about them since the fall of 2018.

https://laserscanningforum.com/forum/viewtopic.php?t=13716

It will be helpful if you read it carefully until the end.

Alright. Good riddance. This worked for me:

NOTE: There's a BIT of oddness with raycasting to a tap gesture's location. Sometimes it fails, which is confusing to me given the tap succeeded. Maybe I'm not converting the locations correctly? Maybe it works better on device?

In a tap gesture handler, get the tap location on a collision shape with:

let worldPosition: SIMD3<Float> = value.convert(value.location3D, from: .local, to: .scene)

With a running WorldTrackingProvider you can get the current device pose with

worldTracking.queryDeviceAnchor(atTimestamp: CACurrentMediaTime())

Then process it like so to get it world-space:

let transform = Transform(matrix: pose.originFromAnchorTransform)
let locationOfDevice = transform.translation

You can then do a raycast to a tap location in world-coordinate-space like so:

let raycastResult = scene.raycast(from: locationOfDevice, to: worldPosition)

If successful, an entry in the raycast result will have normal information. Here I grab the first one

guard let result = raycastResult.first else {
    print("NO RAYCAST HITS?????")
}

let normal = result.normal

Make a quaternion to rotate from identity to the normal vector's angle:

// Calculate the rotation quaternion to align the forward axis with the normal vector
let rotation = simd_quatf(from: SIMD3<Float>(0, 1, 0), to: normal)

Apply it to an entity:

cylinder.transform.rotation = rotation