AVVideoCompositionInstruction with custom compositor

Question

Created Sep ’20

Replies 3

Boosts 0

Participants 2

Dear AVFoundation Engineers & other AVFoundation developers,

In the context of a multilayer video editing timeline where there are 4 or more layers, I want to know if it is a problem to have just one AVVideoCompositionInstruction for the entire time range of the timeline? The parameter requiredSourceTrackIds will be all the tracks added to AVMutableComposition, containsTweening will be true, etc. Then at any frame time, the custom compositor could consult its own internal data structures and blend video frames of different tracks as required. I want to know if there is something wrong in this approach from the performance perspective, especially on new iOS devices(iPhone 7 or later)?

Boost

Answer 1

Media Engineer OP

Apple

Sep ’20

What you are doing should work, though you may get better performance if you specify only the tracks you need. If performance is critical, I recommend building it both ways and comparing the performance. Note you can also set the requiredSourceTrackIDs to nil, which means all source tracks will be considered required for composition.

1

Answer 2

Devgeek OP

Sep ’20

Thank you for the response. Well I can always implement and compare, but implementing the most optimal strategy and finding the most optimal set of instructions may be hard in a multilayer setup with so many visual element types. If we just keep black video(1 frame of size 480x320 stretched to fit timerange) in the below layers of passthrough tracks, will the decoder have any extra hard time decoding the black videos? In other words, passthrough track + black video vs passthrough track, is it any different?

0

Answer 3

Media Engineer OP

Apple

Sep ’20

Even though it’s just 1 frame, it still must be decoded and presented with every frame you are composing. It’s using extra memory at least. If you are sending all the tracks to the compositor, why not pick 1 track to be the main track instead of having the black frame track? Overall though, you may just have to try various things and evaluate whether the performance is acceptable for your use case.

1