CMIOExtensionStreamSource supported pixel formats

Question

mansourmoufid OP

Created Feb ’23

Replies 4

Boosts 0

Views 1.3k

Participants 2

Hello.

I'm working on a Core Media IO camera extension, including a source and sink stream. My application sends sample buffers to the sink stream, which are passed to the source stream.

The sample buffers' pixel format is 420v. This format is determined in the application (not the camera extension) by a VTDecompressionSession, which has the image buffer attribute requested: kCVPixelBufferIOSurfaceCoreAnimationCompatibilityKey; and a decoder specification that requests the key kVTVideoDecoderSpecification_EnableHardwareAcceleratedVideoDecoder.

However, no client can connect to the extension's source stream. Photo Booth shows a blank screen.

I noticed the second argument to the function CMVideoFormatDescriptionCreate, which has type CMVideoCodecType, doesn't explicitly support 420v. In fact there is no 4:2:0 format in the list at all.

Comparing the headers, the only format in common between CoreVideo and CoreMedia is 422YpCbCr8...

% sed -n -e 's/.*\(kCMVideoCodecType_[a-zA-Z0-9_]*\).*/\1/p' < /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreMedia.framework/Versions/A/Headers/CMFormatDescription.h | sort | uniq > kCMVideoCodecType.txt
% sed -n -e 's/.*\(kCVPixelFormatType_[a-zA-Z0-9_]*\).*/\1/p' < /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreVideo.framework/Versions/A/Headers/CVPixelBuffer.h | sort | uniq > kCVPixelFormatType.txt
% comm <(sed -e 's/kCMVideoCodecType_//' < kCMVideoCodecType.txt) <(sed -e 's/kCVPixelFormatType_//' < kCVPixelFormatType.txt)

What is the guidance for pixel formats in camera extensions? The sample camera extension code produced by Xcode uses kCVPixelFormatType_32BGRA. Should camera extensions all be using this format? That's an extra step of YUV→RGB conversion that I'd rather avoid if possible.

Boost

Answer 1

mansourmoufid OP

Feb ’23

To clarify, kCVPixelFormatType_32BGRA works fine but I'd rather stick to 420v because the conversion to 32BGRA doubles my app's CPU usage.

0

Answer 2

mansourmoufid OP

Feb ’23

Nevermind, the pixel format was not the issue. 420v works fine. Lots of reboots later I realized I wasn't setting the timing info right.

In case anyone else is doing a source-sink type of extension, make sure to update the timing info before sending the sample buffer to the source stream:

            CMSampleBufferRef sbuf = NULL;
            CMSampleTimingInfo timing_info = {
                .presentationTimeStamp = CMClockGetTime(CMClockGetHostTimeClock()),
            };
            OSStatus err = CMSampleBufferCreateCopyWithNewTiming(
                kCFAllocatorDefault,
                self->_sampleBuffer,
                1,
                (const CMSampleTimingInfo []) {timing_info},
                &sbuf
            );
            [self->_streamSource.stream sendSampleBuffer:sbuf ...];
            CFRelease(sbuf);

0

Answer 3

DrXibber OP

Mar ’23

Thanks for this tip - after reading this I looked at my code and I was propagating the sampleTimingInfo that originated in the client app all the way through because, well, that seemed like the smart thing to do. I, too, sometimes see strange behaviour and I'm hoping this will make it more robust.

0

Answer 4

mansourmoufid OP

Mar ’23

Funny... Photo Booth will accept either 420v or kCVPixelFormatType_32BGRA (maybe more?) in the stream source's formats property. You can even send it sample buffers with a format different from the one in .formats and it will work just fine.

But QuickTime only accepts kCVPixelFormatType_32BGRA in .formats. It too works even if you send it 420v sample buffers.

Lesson: declare kCVPixelFormatType_32BGRA in .formats, and send whatever you want. 🙂

0