We are using AVAssetWriter to write videos using both HEVC and H.264 encoding. Occasionally, we get reports of choppy footage in which frames appear out of order when played back on a Mac (QuickTime) or iOS device (stock Photos app). This occurs extremely unpredictably, often not starting until 20+ minutes of filming, but occasionally happening as soon as filming starts.
Interestingly, users have reported the issue goes away while editing or viewing on a different platform (e.g. Linux) or in the built-in Google Drive player, but comes back as soon as the video is exported or downloaded again. When this occurs in an HEVC file, converting to H.264 seems to resolve it. I haven't found a similar fix for H.264 files.
I suspect an AVAssetWriter encoding issue but haven't been able to uncover the source.
Running a stream analyzer on HEVC files with this issue reveals the following error:
Short-term reference picture with POC = [some number] seems to have been removed or not correctly decoded.
However, running a stream analyzer on H.264 files with the same playback issue seems to show nothing wrong.
At a high level, our video pipeline looks something like this:
- Grab a sample buffer in
captureOutput(_ captureOutput: AVCaptureOutput!, didOutputVideoSampleBuffer sampleBuffer: CMSampleBuffer!) - Perform some Metal rendering on that buffer
- Pass the resulting
CVPixelBufferto theAVAssetWriterInputPixelBufferAdaptorassociated with ourAVAssetWriter
Example files can be found here: https://drive.google.com/drive/folders/1OjDZ3XaC-ubD5hyDiNvMQGl2NVqZbWnR?usp=sharing
This includes a video file suffering this issue, the same file fixed after converting to mp4, and a screen recording of the distorted playback in QuickTime.
Can anyone help point me in the right direction to solving this issue? I can provide more details as necessary.
