ARKit Front facing depth camera intrinsics bug on XS/XR/XS Max?

Question

Created Sep ’20

Replies 3

Boosts 1

Views 1.9k

Participants 3

I've been using the front facing camera's depth data from the ARFrame's during an ARKit session, and on the iPhone XS/XR/XS Max the resultant point clouds are 'wider' or more spread out than the 11 Pro or the X, and they are different to the results from the standard foundation AV depth data captured on the same device.

In reviewing the data, the depth values themselves seem correct, it more appears that the intrinsics matrix and reference image size to produce values are 5-6% too spread out.

Is there a known issue with XS/XR/XS Max and these values in ARKit? It appears that the focal length in the ARKit intrinsics matrix is the same as in AV depth data's intrinsics matrix, but the ARKit reference image size is larger (by 5.1% coincidence?).

I think if I were to scale the focal length's by the difference in reference image sizes then this would come close to correcting the issue, BUT this is a 'hack' and implies that the OS is not providing the correct values for us to trust/use, and if it is a bug then it could be corrected and any 'hack' would then over compensation and distort the data.

Can anyone help?

Thanks in advance,

Peter Myerscough-Jackopson

Ps. (I think there may be a smaller issue with the X, but I am not confident enough with my analysis and the number of devices I have available is not significant enough to suggest it.)

Boost

Answer 1

KMT OP

Sep ’20

if it is a bug

I'd file one, and see what comes back, being sure to add your report numbers to your thread for ref.

As well, you might consider burning a support ticket w/DTS via the Member Center, having a project that demos the issue ready to hand off.

Good luck.

0

Answer 2

peter_m-j OP

Sep ’20

During testing we grabbed these values from three difference devices:

iPhone XR*
(ARKit)
Intrinsic Matrix (AVDepthData.cameraCalibrationData?.intrinsicMatrix): Optional(simdfloat3x3([[2897.0186, 0.0, 0.0], [0.0, 2897.0186, 0.0], [1617.3761, 1220.1245, 1.0]]))
Intrinsic Matrix Reference Dimensions (AVDepthData.cameraCalibrationData?.intrinsicMatrixReferenceDimensions): Optional((3248.0, 2436.0))
(AV system)
Intrinsic Matrix (AVDepthData.cameraCalibrationData?.intrinsicMatrix): Optional(simdfloat3x3([[2897.0186, 0.0, 0.0], [0.0, 2897.0186, 0.0], [1530.7522, 1151.7511, 1.0]]))
Intrinsic Matrix Reference Dimensions (AVDepthData.cameraCalibrationData?.intrinsicMatrixReferenceDimensions): Optional ((3088.0, 2316.0))

iPhone 11 Pro
(ARKit)
(AVDepthData.cameraCalibrationData?.intrinsicMatrix): Optional(simdfloat3x3([[2737.6152, 0.0, 0.0], [0.0, 2737.6152, 0.0], [2014.8547, 1509.0757, 1.0]]))
Intrinsic Matrix Reference Dimensions (AVDepthData.cameraCalibrationData?.intrinsicMatrixReferenceDimensions): Optional((4032.0, 3024.0))
(AV system)
Intrinsic Matrix (AVDepthData.cameraCalibrationData?.intrinsicMatrix): Optional(simdfloat3x3([[2737.6152, 0.0, 0.0], [0.0, 2737.6152, 0.0], [2013.7095, 1517.8486, 1.0]]))
Intrinsic Matrix Reference Dimensions (AVDepthData.cameraCalibrationData?.intrinsicMatrixReferenceDimensions): Optional((4032.0, 3024.0))

iPhone X
(ARKit)
Intrinsic Matrix (AVDepthData.cameraCalibrationData?.intrinsicMatrix): Optional(simdfloat3x3([[2878.297, 0.0, 0.0], [0.0, 2878.297, 0.0], [1918.1182, 1085.2241, 1.0]]))
Intrinsic Matrix Reference Dimensions (AVDepthData.cameraCalibrationData?.intrinsicMatrixReferenceDimensions): Optional((3840.0, 2160.0))
(AV system)
Intrinsic Matrix (AVDepthData.cameraCalibrationData?.intrinsicMatrix): Optional(simdfloat3x3([[2878.297, 0.0, 0.0], [0.0, 2878.297, 0.0], [1542.1182, 1154.2759, 1.0]]))
Intrinsic Matrix Reference Dimensions (AVDepthData.cameraCalibrationData?.intrinsicMatrixReferenceDimensions): Optional((3088.0, 2316.0))

The reference image sizes change on AR kit, for the XR, but not on the 11. On X there change aspect ratio too, but the X has 360x640 and 480x640 depth images so it is a little clear how to compare the changes in the X.

Using the AV system's reference image size in ARKit would 'correct' our issue when creating 3D points from the depth data.

0

Answer 3

Ozda OP

Nov ’20

Hi, I have exactly the same issue here. Did you find any solution for this issue ?

0