The set of points list[] is a collection of 3D points from: ARPointCloud https://youtu.be/4U4FlavRKa4 Apple ARKit: AR Based on Curved Object Surfaces. vertices of ARMeshAnchor https://youtu.be/JSNXB3zc4mo Fast Cylinder Fitting - Apple iPad Pro LiDAR. ARDepthData https://youtu.be/zc6GQOtgS7M Real Time Ball Tracking in depthMap - Apple iPad Pro LiDAR.

Hi TSHKS, I summarize the picking a 3D point in 2D screen as below. How to Pick a 3D Point in 2D Screen (https://github.com/CurvSurf/ARKitDepthFindSurfaceWeb): You aim your device at object surface. Your touch point is in the screen center. The probe radius [pixel] on screen is converted to physical radius at unit distance from ray_pos. unitRadius represents the half vertex angle of the “view cone”. Squared slope length: UR_SQ_PLUS_ONE = unitRadius * unitRadius + 1.0 . len1 = distance of list[idx] from ray_pos in the direction of ray_dir . Squared slope length at distance len1 is UR_SQ_PLUS_ONE * (len1 * len1). If the squared distance of list[idx] from ray_pos is smaller than the squared slope length at len1, the point list[idx] is viewed inside the “view cone”. Orthographic version “view cylinder (pipe)” is used by https://developers.curvsurf.com/WebDemo/ Hope this helps....

Hello, I hope I help you. The overall process will be: LiDAR data collection -> Touch a screen point -> Find the point closest to the touchscreen point from the LiDAR data. The unit of measurement for LiDAR depth data is meters, i.e. [m]. The relatively reliable distance range for which data accuracy is guaranteed is between about 0.3 and 4 meters. function _pickPoint() from GitHub/CurvSurf

3D point cloud processing is a really difficult problem. It truly requires a lot of specialized knowledge; Linear algebra, algebra, geometry, differential geometry, probability and statistics, optimization, 3D measurement practice, software practice, ... This is a problem that cannot be solved with expertise in just one field. I have expressed my views as I thought about them since the fall of 2018. https://laserscanningforum.com/forum/viewtopic.php?t=13716 It will be helpful if you read it carefully until the end.

let cameraTransform = camera.transform // Right-Handed let rayDirection = -simd_make_float3( cameraTransform.columns.2 ) let rayOrigin = simd_make_float3( cameraTransform.columns.3 ) There is definitely no alternative. camera.transform is the most accurate real-time info about camera's 6DOF under ARKit.

The ray casting implemented in a CurvSurf FindSurface demo app is as follows. The basic information needed for ray casting is: Ray origin Ray direction Ray casting target domain. In the CurvSurf FindSurface demo app: Ray origin: Current position of device Ray direction: Center of device screen (has eventually 6-DOF) Ray casting target domain: 3D measurement point (point cloud, or vertex points of mesh). pickPoint() sets up a viewing cone with the ray direction as its axis, and selects the point closest to the device among the points inside the viewing cone. If there are no point inside the viewing cone, the point closest to the viewing cone is selected. https://github.com/CurvSurf/FindSurface-GUIDemo-iOS/blob/main/ARKitFindSurfaceDemo/ViewController.swift https://github.com/CurvSurf/FindSurface-GUIDemo-iOS/blob/main/ARKitFindSurfaceDemo/Helper.swift

The normal vector of the mesh is fundamentally sensitive to measurement errors. Therefore, if the measurement error is large, the spatial range for calculating the average normal vector at the tap location must be increased. Additionally, real-life objects have sizes. The size can be approximately expressed as the radius of surface curvature. For example, the radius of a cylinder. Therefore, the spatial extent for calculating the normal vector at the tap location must be increased proportional to the radius of curvature of the object surface. The radius of curvature of a plane is very large. Currently, it appears to take a normal vector of one vertex at the tap location. This is quite unstable information. In summary, The normal vector calculation range near the tap location must be expanded in proportion to the measurement error. The normal vector calculation range must be expanded in proportion to the radius of curvature of the target object surface at the tap location. In summary, one solution may be to take the average value of the normal vector of several vertices near the tap location. App developers must implement it themselves. More preferably, Apple, which has information about measurement errors, should provide a solution.

You can use the set of vertices of ARMeshAnchor as a point cloud. https://developer.apple.com/documentation/arkit/armeshgeometry/3516924-vertices Then, there will be methods for determining the bounding box of the point cloud.

An item (model, or virtual object) has shape, size, position, and orientation. There is the world reference coordinate system. And, your device has moving device coordinate system. A real moving/stationary object has own model (shape, size, position, and orientation) coordinate system. And finally, You like to attach your item onto/around a real object surface inside your device screen. It's a real time AR example. Really difficult unsolved problem yet. It's a hard problem. But, there is already a solution. First, you created/designed an item (model, or virtual object) with shape, size on the model coordinate system. Then, the world reference coordinate system was set as you ran your App. The device's coordinate system referenced to the world reference coordinate system is determined in real time by motion tracking of ARKit. What information is lacked at this moment? What unknown is the shape, size, position, and orientation of the real object surface of your interest (e.g., your real object plane). Our FindSurface runtime library solves your last problem: FindSurface Web Demo https://developers.curvsurf.com/WebDemo/ Virtual ads inside Chungmuro station Line 3 - iPhone Pro 12 https://youtu.be/BmKNmZCiMkw

Apple's current privacy policy doesn't allow this. This is especially true given the failure of Google Glass in 2013. Imagine the person in front of you wearing Vision Pro, recognizing your face, tracking your body movements, and having fun creating virtual, comical transformations of your face and body. Also imagine that it tracks your ID and analyzes your body movement-related health status in real time. Your private living room can be scanned and misused somewhere at some point.

The FindSurface SDK software library has a binary file size of about 300 KB for Windows 64-bit DLL or 1 MB (due to the module name texts) for iOS/iPadOS/macOS/Linux/Android framework. It is a really tiny little runtime library that contains many algorithms for region growing, segmentation, model fitting (orthogonal distance fitting), optimization, geometry, curvature analysis, statistics and probability theory without any commercial libraries or ML/DL algorithms. It is a simple container with optimized abstract mathematical formulas and strategies.

What is clear is that if you can solve a problem with software, you can always save on hardware costs. Even if the data quality is low, if the desired results can be achieved through software, inexpensive hardware should be applied. That is why the role of mathematics stands out. Here is an example of processing 3D face point cloud data (2019) by CurvSurf FindSurface. Nose extraction - iPhone X, SR300 https://youtu.be/eCmVYl3GIEY

Night vision enables the visually impaired to see. In other words, it provides information about the shape, size, position and posture of objects in front of a visually impaired person. FindSurface can provide real-time information about this.

The ARPlaneAnchor and ARMeshAnchor (plus Scene understanding) spatial information will be the only secondary information provided even after the physical primary sensor information is processed internally by visionOS. Currently, no primary sensor information is provided in the visionOS 'simulator'. Although the primary physical sensors, the RGB camera and the LiDAR 3D camera, are installed in the commercially available Vision Pro, only ARPlaneAnchor and ARMeshAnchor appear to be made available to developers via visionOS ARKit to protect personal data. It seems that information such as RGB stream, LiDAR depth map, facial recognition and human body contours are not provided. There is absolutely no reason why Apple would allow the development of apps that allow users to attach Vision Pro to their heads and secretly alter other people's faces and bodies.

By processing LiDAR's 576 points in real time, you can determine the shape, size, location, and orientation of a real-world object's surface front of you, even in the total darkness. Not only the objects but also you are allowed to move.