`AVCaptureVideoDataOutput` consumes three times more memory than `AVCaptureMovieFileOutput`

Issue

I'm using AVFoundation to implement a Camera that is able to record videos while running special AI processing.

Having an AVCaptureMovieFileOutput (for video recording) and a AVCaptureVideoDataOutput (for processing AI) running at the same time is not supported (see https://stackoverflow.com/q/4944083/5281431), so I have decided to use a single AVCaptureVideoDataOutput which is able to record videos to a file while running the AI processing in the same captureOutput(...) callback.

To my surprise, doing that drastically increases RAM usage from 58 MB to 187 MB (!!!), and CPU from 3-5% to 7-12% while idle. While actually recording, the RAM goes up even more (260 MB!).

I am wondering what I did wrong here, since I disabled all the AI processing and just compared the differences between AVCaptureMovieFileOutput and AVCaptureVideoDataOutput.

My code:

AVCaptureMovieFileOutput

Setup

Code Block swift
    if let movieOutput = self.movieOutput {
      captureSession.removeOutput(movieOutput)
    }
    movieOutput = AVCaptureMovieFileOutput()
    captureSession.addOutput(movieOutput!)

Delegate

(well there is none, AVCaptureMovieFileOutput handles all that internally)

Benchmark

When idle, so not recording at all:

• RAM: 56 MB

• CPU: 3-5%

When recording using AVCaptureMovieFileOutput.startRecording:

• RAM: 56 MB (how???)

• CPU: 20-30%

AVCaptureVideoDataOutput

Setup

Code Block swift
    // Video
    if let videoOutput = self.videoOutput {
      captureSession.removeOutput(videoOutput)
      self.videoOutput = nil
    }
    videoOutput = AVCaptureVideoDataOutput()
    videoOutput!.setSampleBufferDelegate(self, queue: videoQueue)
    videoOutput!.alwaysDiscardsLateVideoFrames = true
    captureSession.addOutput(videoOutput!)
    // Audio
    if let audioOutput = self.audioOutput {
      captureSession.removeOutput(audioOutput)
      self.audioOutput = nil
    }
    audioOutput = AVCaptureAudioDataOutput()
    audioOutput!.setSampleBufferDelegate(self, queue: audioQueue)
    captureSession.addOutput(audioOutput!)

Delegate

Code Block swift
extension CameraView: AVCaptureVideoDataOutputSampleBufferDelegate,
                      AVCaptureAudioDataOutputSampleBufferDelegate {
  public final func captureOutput(_ captureOutput: AVCaptureOutput,
                                  didOutput sampleBuffer: CMSampleBuffer,
                                  from _: AVCaptureConnection) {
    // empty
  }
  public final func captureOutput(_ captureOutput: AVCaptureOutput,
                                  didDrop buffer: CMSampleBuffer,
                                  from _: AVCaptureConnection) {
    // empty
  }
}

yes, they are literally empty methods. My RAM and CPU usage is still that high without doing any work here.

Benchmark

When idle, so not recording at all:

• RAM: 151-187 MB

• CPU: 7-12%

When recording using a custom AVAssetWriter:

• RAM: 260 MB

• CPU: 64%

Why is the AVCaptureMovieFileOutput so much more efficient than an empty AVCaptureVideoDataOutput? Also, why does it's RAM not go up at all when recording, compared to how my AVAssetWriter implementation alone consumes 80 MB?

Here's my custom AVAssetWriter implementation: [RecordingSession.swift](https://github.com/cuvent/react-native-vision-camera/blob/frame-processors/ios/RecordingSession.swift), and heres where I call it.

Any help appreciated!