When isCameraIntrinsicMatrixDeliverySupported?

I'm trying to obtain the intrinsic matrix for each video frame of AVCaptureSession (the same intrisic matrix as ARKit provides), however the isCameraIntrinsicMatrixDeliverySupported property of AVCaptureConnection is false in my use-case.

The documentation of the property says "This property's value is

true
only if both the connection's input device format and output class support delivery of camera intrinsics.
"

How do I know which device formats support delivery of intrinsic matrix? What do I need to do to be able to enable the intrinsic matrix delivery?


Simple code to illustrate my problem:


import UIKit
import AVFoundation

class ViewController: UIViewController {

    var sess: AVCaptureSession!
    var sessOut: AVCaptureVideoDataOutput!
    var prevLayer: AVCaptureVideoPreviewLayer!

    override func viewDidLoad() {
        super.viewDidLoad()
   
        sess = AVCaptureSession()
   
        let device = AVCaptureDevice.default(.builtInWideAngleCamera, for: AVMediaType.video, position: .back)
        let input  = try! AVCaptureDeviceInput(device: device!)
        sess.addInput(input)
   
        sessOut = AVCaptureVideoDataOutput()
        sess.addOutput(sessOut)
        sessOut.connections.first?.videoOrientation = .landscapeRight
        sessOut.connections.first?.preferredVideoStabilizationMode = .cinematic
        print(sessOut.connections.first?.isCameraIntrinsicMatrixDeliverySupported) // <-- false - why?
   
        prevLayer = AVCaptureVideoPreviewLayer(session: sess)
        prevLayer.frame = self.view.frame
        prevLayer.videoGravity = .resizeAspectFill
        prevLayer.connection?.videoOrientation = .landscapeRight
        self.view.layer.addSublayer(prevLayer)
   
        sess.startRunning()
    }
}

Replies

Bump.


So nobody knows the answer (even the Apple Staff)?

Where is the proper place to ask this question?

Re-bumping. I have run into the same problem – I can't find documentation anywhere on what devices are supported, and how to get this working.

Hi Marek,

The reason you're not getting intrinsics is because you're setting the video stabilization mode to something other than None. Video stabilization moves the pixels around in the frame such that the values in the intrinsic matrix can no longer be relied upon.

This makes sense, however it's not true.

I'm getting intrinsics regardless of stabilization settings (tried off, auto and cinematic - works every time).

Tried it on an iPhone X and on an iPhone 7 plus.

Does not work on an iPad Pro.

The question was which devices/video formats are supported, and it is still unanswered.

In case anyone comes across this thread and was having the same problem I was:

When capturing from both constituent devices in builtInDualCamera, I had to enable intrinsics delivery on each connection after both connections were added to the session. Before that, isCameraIntrinsicMatrixDeliverySupported would return false for both connections.

Not sure how/if this insight applies to a single-cam case.
  • Yes, that's it!!! Works for three camera session on the iPhone 13 as well. Before all connections are added, it's always false. When all three are there, magically this becomes true.

    Thank you!!!!!!!!!

Add a Comment