Encoder Performance Unable to Achieve 4k 120fps on iPhone 16 Pro

Question

Created 2d

Replies 1

Boosts 0

Participants 2

Hello, I am trying to get the new iPhone 16 pro to achieve 4k 120fps encoding when we are getting the video feed from the default, wide angle camera on the back. We are using the apple API to capture the individual frames from the camera as they are processed and we get them in this callback:

// this is the main callback function to handle video frames captured
    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

We are then taking these frames as they come in and encoding them using VideoToolBox. After they are encoded, they are added to a ring buffer so we can access them after they have been encoded.

The problem is that when we are encoding these frames on an iPhone 16 Pro, we are only reaching 80-90fps instead of 120fps. We have removed as much processing as we can. We get some small attributes about the frame when it comes in, encode the frame, and then add it to our ring buffer.

I have attached a sample project that is broken down as much as possible to the basic task of encoding 4k 120fps footage. Inside the sample app, there is an fps and pps display showing how many frames we are encoding per second. FPS represents how many frames we are coming in per second from the camera, and PPS represents how many frames we are processing (encoding) per second.

Link to sample project: https://github.com/jake-fishtech/EncoderPerformance

Thanks you for any help or suggestions.

Boost

Answer 1

Media Engineer OP

Apple

1d

Hi j-fishman,

I see a couple suspect settings in your sample project.

You are asking for BGRA video output instead of 420v or x422. This will definitely impact performance, as it requires a pixel transfer from the native yuv format to the much bigger BGRA. And then a second transfer back to yuv in the encoder. Please keep things in the YUV domain.
In your compression settings, you are setting a keyframe interval of 20. This may be impacting performance as well, since it's forcing a much higher data rate 6x per second instead of 1x per second. We suggest you use the AVCaptureVideoDataOutput's

(nullable NSDictionary<NSString *, id> *)recommendedVideoSettingsForVideoCodecType:(AVVideoCodecType)videoCodecType assetWriterOutputFileType:(AVFileType)outputFileType

which will give you Apple's recommended setting for encoding. These are the same settings used by Apple's own Camera.app when recording 4K120.

Please let us know if these relieve the frame drops.

0