AV1 Hardware Decoding

It would be great if Apple could provide an example of how to decode AV1 video on an iPhone 15 Pro Max or one of the new M3 MacBook Pros, using VideoToolbox.

The information appears not to be anywhere.

Might these links be of use:

1. https://chromium.googlesource.com/chromium/src/+/master/media/gpu/mac/vt_video_decode_accelerator_mac.cc
2. https://chromium.googlesource.com/chromium/src/+/HEAD/media/gpu/mac/vt_config_util.mm

Specifically the function CreateVideoFormatAV1 in 1 and the function CreateFormatExtensions in 2.?

Must set 'SampleDescriptionExtensionAtoms' for 'extensions', 'SampleDescriptionExtensionAtoms' must contain 'av1C' for av1 extradata, example:

{
	mediaType:'vide' 
	mediaSubType:'av01' 
	mediaSpecific: {
		codecType: 'av01'		dimensions: 720 x 1280 
	} 
	extensions: {{
    BitsPerComponent = 8;
    CVFieldCount = 1;
    CVImageBufferChromaLocationBottomField = Left;
    CVImageBufferChromaLocationTopField = Left;
    CVImageBufferColorPrimaries = "ITU_R_709_2";
    CVImageBufferTransferFunction = "ITU_R_709_2";
    CVImageBufferYCbCrMatrix = "ITU_R_709_2";
    Depth = 24;
    FormatName = "'av01'";
    FullRangeVideo = 0;
    RevisionLevel = 0;
    SampleDescriptionExtensionAtoms =     {
        av1C = {length = 20, bytes = 0x81050c000a0e0000002cd59f3fddaf9901010104};
    };
    SpatialQuality = 0;
    TemporalQuality = 0;
    VerbatimISOSampleEntry = {length = 124, bytes = 0x0000007c 61763031 00000000 00000001 ... 000a6669 656c0100 };
    Version = 0;
}}
}

Take a look at this issue. They had implemented av1 hardware decoding on iPhone 15 Pro, using same method as Chromium.

https://github.com/moonlight-stream/moonlight-ios/issues/585

Just posting back here as I got all this working in the end.

In case it's useful, here are the stumbling blocks I encountered. Probably, these are just more a reflection of my lack of understanding but maybe it'll help someone.

To construct AV1 Codec Configuration Box outside of FFmpeg etc, then this describes the structure:

1. https://aomediacodec.github.io/av1-isobmff/#av1codecconfigurationbox-section

The information needed comes from parsing the Sequence Header OBU:

1. https://aomediacodec.github.io/av1-spec/#general-sequence-header-obu-syntax

If you're writing from scratch (i.e. not. using ffmpeg or whatever), then you need to write or find code to parse the sequence header OBU.

Once you've written the 4 bytes described in 1. then you also need to append the sequence header OBU data block to the end of the block. If you don't, then the decoder setup will fail.

This is then added to the extensions dictionary, along with all the other basic information needed to initialise the decoder (the Chrome references detail all this information).

You then create the video format description using CMVideoFormatDescriptionCreate, passing in the extensions.

I then got caught out with a decode error because I didn't realise that I also had to pass in the Sequence Header OBU with the first frame data I attempted to decode. It wasn't enough that I had already given the same Sequence Header OBU when creating the video format description (via the extensions).

After that it worked.

Decoding itself is slightly simpler than with HEVC, in that you don't need to parse the OBUs, you just pass the data straight to the decoder. With HEVC, you had to parse the NALUs and only pass in slice segments, while also doing some minor conversion of the way the NALU's length is presented to the decoder.

It would be helpful, Apple, if you could consider writing something like CMVideoFormatDescriptionCreateFromAV1SequenceHeaderOBU similar to the existing CMVideoFormatDescriptionCreateFromH264ParameterSets and CMVideoFormatDescriptionCreateFromHEVCParameterSets.

This would lower the bar a little to AV1 hardware decoding.

Sorry for the late reply I hadn't set up notifications (I have now).

1. If you don't append the sequence header OBU data to the end of the AV1CodecConfigurationRecord then decoder initialisation will fail with error -12911.

2. if you don't include the sequence header OBU data at the start of the first frame you decode, then you will get error -12909 inside the decompression callback.

In my scenario, I am in control of the encoder (NVENC) so I get the sequence header OBU when I initialise the encoder, and send this across as my "extra data" before I send any encoded frame data. On the decode side, I then use this in 1 and 2 above. My encoder then doesn't send any further sequence header OBUs at all as in my case the format doesn't change.

In a more general scenario, there might be a sequence header with every IDR frame, so you'd just need to make sure you wait until you get the first sequence header so you can initialise the decoder as per 1. above, and then since the sequence header is already part of the encoded packet data 2. would be taken care of anyway.

Does your decoder callback always receive -12911 or is that an occasional error? I've had issues with occasional and unexplained errors, and the only solution was to wait for (or in my case request) a key frame to get it back on track. It seemed to relate to low bitrate AV1 streams.

No idea about VP9, I don't use it.

i met the -12911 error which is the error code for kVTVideoDecoderMulfunctionErr, the subsequent logs shows it was caused by an err in allocating the memory for the ioSurface.

default	13:16:36.985714+0800	videocodecd	AppleAVD: AppleAVDWrapperFghrnDecoderStartSession: storage->miscPreferences 8
default	13:16:36.985774+0800	videocodecd	AppleAVD: AppleAVDWrapperFghrnDecoderStartSession() codecType: Fghrn, encryptionScheme 6, 7680 x 4320, tryAllFrames = 0, session : 0xc54f1c020, built 01:13:29 Aug  8 2024
default	13:16:37.011217+0800	videocodecd	AppleAVD: INFO: AppleAVDCreateDecodeDeviceInternal(): Allocating new AppleAVDCommandBuilder
default	13:16:37.011267+0800	videocodecd	AppleAVD: CAVDAvxDecoder(): set av1 dual vp mode as 0
default	13:16:37.011369+0800	videocodecd	AppleAVD: INFO: Av1 decoder does not support kVASetSliceHeaderThreshold, command has been ignored
default	13:16:37.011811+0800	videocodecd	AppleAVD: ERROR: mapAVDMemory(): failed error: -536870212
default	13:16:37.011850+0800	videocodecd	AppleAVD: VAMapPixelBuffer(): error allocating surface
default	13:16:37.011862+0800	videocodecd	AppleAVD: createDecoder(): VAMapPixelBuffer failed for input buf IOSurface, err=-536870212
default	13:16:37.011890+0800	videocodecd	AppleAVD: createDecoder(): error -536870212 and decoder is 0xc55400000

But that error is caused by decoding a 8K AV1 error. when decoding a 4K video the error is gone.