Frames out of order using AVAssetWriter

Question

Created May ’23

Replies 4

Boosts 2

Views 1.4k

Participants 4

We are using AVAssetWriter to write videos using both HEVC and H.264 encoding. Occasionally, we get reports of choppy footage in which frames appear out of order when played back on a Mac (QuickTime) or iOS device (stock Photos app). This occurs extremely unpredictably, often not starting until 20+ minutes of filming, but occasionally happening as soon as filming starts.

Interestingly, users have reported the issue goes away while editing or viewing on a different platform (e.g. Linux) or in the built-in Google Drive player, but comes back as soon as the video is exported or downloaded again. When this occurs in an HEVC file, converting to H.264 seems to resolve it. I haven't found a similar fix for H.264 files.

I suspect an AVAssetWriter encoding issue but haven't been able to uncover the source.

Running a stream analyzer on HEVC files with this issue reveals the following error: Short-term reference picture with POC = [some number] seems to have been removed or not correctly decoded.

However, running a stream analyzer on H.264 files with the same playback issue seems to show nothing wrong.

At a high level, our video pipeline looks something like this:

Grab a sample buffer in captureOutput(_ captureOutput: AVCaptureOutput!, didOutputVideoSampleBuffer sampleBuffer: CMSampleBuffer!)
Perform some Metal rendering on that buffer
Pass the resulting CVPixelBuffer to the AVAssetWriterInputPixelBufferAdaptor associated with our AVAssetWriter

Example files can be found here: https://drive.google.com/drive/folders/1OjDZ3XaC-ubD5hyDiNvMQGl2NVqZbWnR?usp=sharing

This includes a video file suffering this issue, the same file fixed after converting to mp4, and a screen recording of the distorted playback in QuickTime.

Can anyone help point me in the right direction to solving this issue? I can provide more details as necessary.

Boost

Answer 1

galad87 OP

May ’23

Your original.mov has got crazy values in the "ctts" atom. HEVC and H.264 has got a frame type called B that can reference previous and future frames, so they are stored out of order. This means the presentation timestamp (pts) and the decode timestamp (dts) are different, and the offset between the two is stored in the mov/mp4 ctts atom. Some players ignore this atom and uses the info stored in the video bitstream, so they might not display the issue.

I would check the pts/dts of the sample buffers you are passing to AVAssetWriter.

1

Answer 2

tenuki OP

May ’23

Thank you for the context @galad87!

I did suspect issues with the presentation timestamp, but unfortunately have been unable to recreate the issue by artificially messing with the timestamps. The relevant method is append(_ pixelBuffer: CVPixelBuffer, withPresentationTime presentationTime: CMTime) on AVAssetWriterInputPixelBufferAdaptor.

From my experimentation, artificially delaying a timestamp just causes a delay in the footage (somewhat obvious), while setting timestamps in the past just causes the writer to fail since AVAssetWriter expects frames in order. I have not been able to find a combination of timestamps to force the reordering behavior described before. Something could definitely still be wrong with our implementation, but I am starting to wonder if something is going on with AVAssetWriter internally.

1

Answer 3

mrousavy OP

Nov ’23

Hey - did you find a solution for this? In my case, video and audio tracks are out of sync by almost a full second...

0

Answer 4

erdeszbalazs OP

Mar ’24

I'm seeing very similar problems. However I am manually putting together a video from photos in an album, so it's even weirder, given I have full control over the order of frames.

0