Is it possible to save AVCaptureDepthDataOutput to movie as individuals mediaType for applying post effect

I want to record the TrueDepth or Dual camera's depth data output when recording the video data. I have already managed to get the AVCaptureDepthDataOutput object and displayed it in realtime, but I also need the depth to be recorded as an individual track of AVMediaTypeVideo or AVMediaTypeMetadata in the movie, and read them back for post processing.

  • Compared to use AVCaptureMovieFileOutput, I use movieWriter and AVAssetWriterInputPixelBufferAdaptor to append pixel buffer. I have tried to append the streaming depth as normal AVAssetWriterInput with AVVideoCodecTypeH264, but failed.

Is it possible to append depth data buffer in the same way as video data for depth data, or with any other way of doing it?

Up vote post of kobunketsu Down vote post of kobunketsu
1.8k views
Posted by
kobunketsu
Reply
Add a Comment

Accepted Reply

Hello!

Compared to use AVCaptureMovieFileOutput, I use movieWriter and AVAssetWriterInputPixelBufferAdaptor to append pixel buffer. I have tried to append the streaming depth as normal AVAssetWriterInput with AVVideoCodecTypeH264, but failed.

Is it possible to append depth data buffer in the same way as video data for depth data, or with any other way of doing it?

Not exactly in the same way, video compression standards like h264 and h265 are not designed for depth data, and so trying to compress them as such is likely to yield strange results (or possibly will just error out when you attempt it).

It sounds like you simply want there to be a track in the video file (mov) which contains the depth data, in which case, I believe you can use a timed metadata track.

If you need more specific help about creating a timed metadata track, please request technical support.

Posted by
gchiste
Up vote reply of gchiste Down vote reply of gchiste
Post marked as solved

  • Finally, I end up with encoding depth buffer as additional video output, but the 8 bit precision is insufficient.

    —
    kobunketsu

  • Hello Kobunketu, can you please let me know how did you solve this issue? Because i am also getting error while saving depth data pixel buffer as video.

    —
    Abdul-iOS
Add a Comment

Replies

Hello!

Compared to use AVCaptureMovieFileOutput, I use movieWriter and AVAssetWriterInputPixelBufferAdaptor to append pixel buffer. I have tried to append the streaming depth as normal AVAssetWriterInput with AVVideoCodecTypeH264, but failed.

Is it possible to append depth data buffer in the same way as video data for depth data, or with any other way of doing it?

Not exactly in the same way, video compression standards like h264 and h265 are not designed for depth data, and so trying to compress them as such is likely to yield strange results (or possibly will just error out when you attempt it).

It sounds like you simply want there to be a track in the video file (mov) which contains the depth data, in which case, I believe you can use a timed metadata track.

If you need more specific help about creating a timed metadata track, please request technical support.

Posted by
gchiste
Up vote reply of gchiste Down vote reply of gchiste
Post marked as solved

  • Finally, I end up with encoding depth buffer as additional video output, but the 8 bit precision is insufficient.

    —
    kobunketsu

  • Hello Kobunketu, can you please let me know how did you solve this issue? Because i am also getting error while saving depth data pixel buffer as video.

    —
    Abdul-iOS
Add a Comment