Issue
I'm using AVFoundation to implement a Camera that is able to record videos while running special AI processing.
Having an AVCaptureMovieFileOutput (for video recording) and a AVCaptureVideoDataOutput (for processing AI) running at the same time is not supported (see https://stackoverflow.com/q/4944083/5281431), so I have decided to use a single AVCaptureVideoDataOutput which is able to record videos to a file while running the AI processing in the same captureOutput(...) callback.
To my surprise, doing that drastically increases RAM usage from 58 MB to 187 MB (!!!), and CPU from 3-5% to 7-12% while idle. While actually recording, the RAM goes up even more (260 MB!).
I am wondering what I did wrong here, since I disabled all the AI processing and just compared the differences between AVCaptureMovieFileOutput and AVCaptureVideoDataOutput.
My code:
AVCaptureMovieFileOutput
Setup
Code Block swift if let movieOutput = self.movieOutput { captureSession.removeOutput(movieOutput) } movieOutput = AVCaptureMovieFileOutput() captureSession.addOutput(movieOutput!)
Delegate
(well there is none, AVCaptureMovieFileOutput handles all that internally)
Benchmark
When idle, so not recording at all:
RAM: 56 MB
CPU: 3-5%
RAM: 56 MB (how???)
CPU: 20-30%
AVCaptureVideoDataOutput
Setup
Code Block swift // Video if let videoOutput = self.videoOutput { captureSession.removeOutput(videoOutput) self.videoOutput = nil } videoOutput = AVCaptureVideoDataOutput() videoOutput!.setSampleBufferDelegate(self, queue: videoQueue) videoOutput!.alwaysDiscardsLateVideoFrames = true captureSession.addOutput(videoOutput!) // Audio if let audioOutput = self.audioOutput { captureSession.removeOutput(audioOutput) self.audioOutput = nil } audioOutput = AVCaptureAudioDataOutput() audioOutput!.setSampleBufferDelegate(self, queue: audioQueue) captureSession.addOutput(audioOutput!)
Delegate
Code Block swift extension CameraView: AVCaptureVideoDataOutputSampleBufferDelegate, AVCaptureAudioDataOutputSampleBufferDelegate { public final func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from _: AVCaptureConnection) { // empty } public final func captureOutput(_ captureOutput: AVCaptureOutput, didDrop buffer: CMSampleBuffer, from _: AVCaptureConnection) { // empty } }
yes, they are literally empty methods. My RAM and CPU usage is still that high without doing any work here.
Benchmark
When idle, so not recording at all:
RAM: 151-187 MB
CPU: 7-12%
RAM: 260 MB
CPU: 64%
Why is the AVCaptureMovieFileOutput so much more efficient than an empty AVCaptureVideoDataOutput? Also, why does it's RAM not go up at all when recording, compared to how my AVAssetWriter implementation alone consumes 80 MB?
Here's my custom AVAssetWriter implementation: [RecordingSession.swift](https://github.com/cuvent/react-native-vision-camera/blob/frame-processors/ios/RecordingSession.swift), and heres where I call it.
Any help appreciated!