AVCaptureSession use AVAssetWriterInputMetadataAdaptor appendTimedMetadataGroup with detected face failed

Question

Created Jun ’21

Replies 0

Boosts 0

Participants 1

I want to capture video with effects while saving video data with detected face info for post face beauty process in my project. Because AVCaptureVideoDataOutput doesn't work well with AVCaptureMovieFileOutput, I choose to use AVCaptureVideoDataOutput + AVAssetWriter, writing face metadata to assetWriterInput with AVAssetWriterInputMetadataAdaptor, reading back with [AVAssetReaderOutputMetadataAdaptor nextTimedMetadataGroup].

This is how I tried in writing process,

### setup metadata format for input

NSArray *specifications = @[@{
(__bridge NSString *)kCMMetadataFormatDescriptionMetadataSpecificationKey_Identifier : @"mdta/com.apple.quicktime.detected-face",
(__bridge NSString *)kCMMetadataFormatDescriptionMetadataSpecificationKey_DataType : @"com.apple.quicktime.detected-face",
    CMMetadataFormatDescriptionRef metadataFormatDescription = NULL;

CMMetadataFormatDescriptionCreateWithMetadataSpecifications(kCFAllocatorDefault, kCMMetadataFormatType_Boxed, (__bridge CFArrayRef)specifications, &metadataFormatDescription);

AVAssetWriterInput * assetWriterMetadataInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeMetadata outputSettings:nil sourceFormatHint:metadataFormatDescription];

AVAssetWriterInputMetadataAdaptor *assetWriterMetadataAdaptor = [[AVAssetWriterInputMetadataAdaptor alloc]initWithAssetWriterInput:assetWriterMetadataInput];

[assetWriter addInput:assetWriterMetadataInput];
...

### write face metadata when detected using AVCaptureMetadataOutput
- (void)appendMetadataFaceObjectItems:(NSArray<AVMetadataFaceObject *> *)faces frameTime:(CMTime)frameTime{
if (assetWriter.status == AVAssetWriterStatusWriting ) {

            NSMutableArray <AVMutableMetadataItem *>*metadataItems = [NSMutableArray<AVMutableMetadataItem *> new];

            for (AVMetadataFaceObject *face in faces) {

                AVMutableMetadataItem *metadataItem = [AVMutableMetadataItem metadataItem];
                metadataItem.identifier = AVMetadataIdentifierQuickTimeMetadataDetectedFace;

                metadataItem.dataType = @"com.apple.quicktime.detected-face";

                metadataItem.keySpace = @"mdta";

                metadataItem.key = @"com.apple.quicktime.detected-face";

                metadataItem.value = face;

               //other time duration setup

                [metadataItems addObject:metadataItem];
            }            
                AVMutableTimedMetadataGroup *group = [[AVMutableTimedMetadataGroup alloc]initWithItems:metadataItems timeRange:CMTimeRangeMake(frameTime, CMTimeMake(20, 600))];

                BOOL success = [assetWriterMetadataAdaptor appendTimedMetadataGroup:group];
        }
}

the app crashes on appendTimedMetadataGroup, saying:

Cannot write to file timed metadata group : Metadata value is an instance of AVMetadataItem, but format description does not properly describe face data'

I thought I have paired the input format description in setup with in writing, am I missing some other details?

Boost