Recently I've been trying to play some AV1-encoded streams on my iPhone 15 Pro Max. First, I check for hardware support:
VTIsHardwareDecodeSupported(kCMVideoCodecType_AV1); // YES
Then I need to create a CMFormatDescription
in order to pass it into a VTDecompressionSession
. I've tried the following:
{
mediaType:'vide'
mediaSubType:'av01'
mediaSpecific: {
codecType: 'av01' dimensions: 394 x 852
}
extensions: {{
CVFieldCount = 1;
CVImageBufferChromaLocationBottomField = Left;
CVImageBufferChromaLocationTopField = Left;
CVPixelAspectRatio = {
HorizontalSpacing = 1;
VerticalSpacing = 1;
};
FullRangeVideo = 0;
}}
}
but VTDecompressionSessionCreate
gives me error -8971
(codecExtensionNotFoundErr, I assume).
So it has something to do with the extensions dictionary? I can't find anywhere which set of extensions is necessary for it to work 😿.
VideoToolbox has convenient functions for creating descriptions of AVC and HEVC streams (CMVideoFormatDescriptionCreateFromH264ParameterSets
and CMVideoFormatDescriptionCreateFromHEVCParameterSets
), but not for AV1.
As of today I am using XCode 15.0 with iOS 17.0.0 SDK.
Must set 'SampleDescriptionExtensionAtoms' for 'extensions', 'SampleDescriptionExtensionAtoms' must contain 'av1C' for av1 extradata, example:
{
mediaType:'vide'
mediaSubType:'av01'
mediaSpecific: {
codecType: 'av01' dimensions: 720 x 1280
}
extensions: {{
BitsPerComponent = 8;
CVFieldCount = 1;
CVImageBufferChromaLocationBottomField = Left;
CVImageBufferChromaLocationTopField = Left;
CVImageBufferColorPrimaries = "ITU_R_709_2";
CVImageBufferTransferFunction = "ITU_R_709_2";
CVImageBufferYCbCrMatrix = "ITU_R_709_2";
Depth = 24;
FormatName = "'av01'";
FullRangeVideo = 0;
RevisionLevel = 0;
SampleDescriptionExtensionAtoms = {
av1C = {length = 20, bytes = 0x81050c000a0e0000002cd59f3fddaf9901010104};
};
SpatialQuality = 0;
TemporalQuality = 0;
VerbatimISOSampleEntry = {length = 124, bytes = 0x0000007c 61763031 00000000 00000001 ... 000a6669 656c0100 };
Version = 0;
}}
}