Does arkit consider radial distortion issue?

Looks like arkit updates the intrinsic parameters of the camera (ARCamera->intrinsics) from frame to frame. I am wondering if it also takes "radial distortion" into consideration, and rectify the video frames? Or, it just uses a consistent parameter for iphone camera regardless of the manufacture defects? To my knowledge, the radial distortion greatly affects the 6 DOF pose estimation accuracy.

Replies

Although it's not explicitly stated, I'm certain that ARKit is correcing for non-linear lens distortion. Lens distortion (and inverse distortion) lookup tables exist in iOS11 and are available via AVCameraCalibrationData, but they are not exposed by ARKit, presumably because there is no need for them, since you're already working with rectified coordinates.


Whether or not the distortion model parameters are the same for each device model (i.e. exact same values for each iPhone 7) is an interesting question. I don't have access to multiple phones of the same model, but this shouldn't be hard to figure out for someone who does.

>uses a consistent parameter for iphone camera regardless of the manufacture defects?


Fixed lookup values are consistently applied to live camera data. What ARKit does, I'd presume to be downstream, rather than direct.


>radial distortion greatly affects the 6 DOF pose estimation accuracy.


Correct. I'd expect the iPhone 7's dual cameras to help with that. Aren't we lucky 😉

Thanks for the helpful reply! I dont have two iPhone 7 with IOS 11 on hand but I will test later.


A more interesting finding is that the intrinsic parameters (output by https://developer.apple.com/documentation/arkit/arcamera/2875730-intrinsics) change as I rotate the iphone.


Based on the Arkit demo program, I plotted the data intrinsics of every frame, and found that the values of focal length and principal point changes as I rotate the phone. For example, the value of focal length becomes lowest (~1030) when the camera is downwards, and highest (~1050) upwards. Seems like this is affected by the gyroscope in the IMU. However, this confuses me a lot. To the best of my knowledge, the prime-len camera should have a consistent focal length value all the time, and shouldn't be affected by the rotation of the camera.


Now I am wondering if I am going to trust the camera intrinsics provided by the ArKit API or not 😝

You're right - the intrinsics do indeed change upon rotating the iPhone. On top of that, I'm also getting different intrinsic values for the initial 20 (or so) frames after starting the tracking session, even though my device is perfectly still (mounted on a tripod). The focal length variations are minor - in the range of 1% - but still, this is definitely NOT the expected behavior.