How the depth-map is aligned to rgb image ?

Question

Created Aug ’21

Replies 6

Boosts 2

Views 2.3k

Participants 6

I want to know are depth map and RGB image are perfectly aligned(do both have the same principle point)? If yes then how the depth- map is created.

The depth map on iphone12 has 256x192 resolution as opposed to an RGB image (1920x1440). I am interested in exact pixel-wise depth. Is it possible to get the raw depth map of 1920x1440 resolution ?

How is the depth-map is created at 256 x 192 resolution? Behind the scenes does the pipeline captures it at 1920 x1440 resolution and then resize it to 256x192?

I have so many questions as there are no intrinsic, extrinsic, and calibration data given regarding the lidar.

I would greatly appreciate it if someone can explain the steps from a computer-vision perspective.

Many Thanks

Boost

Answer 1

CurvSurf OP

Sep ’21

The number of the current raw laser points is 576 (squared 24x24 = 4x16x3x3. iPad Pro LiDAR, iPhone 12 Pro, iPhone 12 Pro Max).

The advanced ML algorithm of Apple interpolating the 576 depth points together with RGB image generates the 256x192 depth map, a decision of compromise.

The number 576 and 256x192 may be increased in the future.

You can enjoy the current performance of 256x192 depth map of 60 Hz.

Super fast real-time detection of planes / spheres / cylinders / cones / tori in DepthMap

0

Answer 2

ale_ex_ OP

Apr ’24

Hi vaneesh, did you find any solution to get a higher resolution depth map? Or what was your workaround? I am facing the same issue here.

0

Answer 3

yzx OP

Jul ’24

I also want to match depth-map and rgb-image, but I have no idea, do you have any matching idea?

0

Answer 4

bill6155 OP

Sep ’24

It seems pretty fundamental to want to know how the depth pixels align with the image pixels. Is curvsurf right that the raw LiDAR sensor array is only 24x24, and we only have access to the result of some black box fusion between the image and the 24x24 depth?

Regardless, it would be really helpful to know how the LiDAR pixels align to the image...

0

Answer 5

JoonAhn OP

Sep ’24

It's up to Apple. We developers use the functionality that Apple visionOS ARKit offers.

YouTube: https://youtu.be/Mep1w5nvu0Y

GitHub: https://github.com/CurvSurf/FindSurface-RealityKit-visionOS-Response-to-Request

0

Answer 6

JoonAhn OP

Sep ’24

CurvSurf also wants to get access to the 576 laser distance points. CurvSurf's FindSurface framework could determine the shape (plane, sphere, cylinder, cone or ring), size and 6DoF pose of the object in front of iPhone/iPad/Vision Pro in real time at up to 300 Hz from the 576 points, even in total darkness.

0