AVVideoCompositionInstruction with custom compositor

Dear AVFoundation Engineers & other AVFoundation developers,

In the context of a multilayer video editing timeline where there are 4 or more layers, I want to know if it is a problem to have just one AVVideoCompositionInstruction for the entire time range of the timeline? The parameter requiredSourceTrackIds will be all the tracks added to AVMutableComposition, containsTweening will be true, etc. Then at any frame time, the custom compositor could consult its own internal data structures and blend video frames of different tracks as required. I want to know if there is something wrong in this approach from the performance perspective, especially on new iOS devices(iPhone 7 or later)?
Up vote post of Devgeek Down vote post of Devgeek
404 views
Posted by
Devgeek
Reply
Add a Comment

Replies

What you are doing should work, though you may get better performance if you specify only the tracks you need. If performance is critical, I recommend building it both ways and comparing the performance. Note you can also set the requiredSourceTrackIDs to nil, which means all source tracks will be considered required for composition.
Posted by
Media Engineer
Post not yet marked as solved Up vote reply of Media Engineer Down vote reply of Media Engineer
Add a Comment
Thank you for the response. Well I can always implement and compare, but implementing the most optimal strategy and finding the most optimal set of instructions may be hard in a multilayer setup with so many visual element types. If we just keep black video(1 frame of size 480x320 stretched to fit timerange) in the below layers of passthrough tracks, will the decoder have any extra hard time decoding the black videos? In other words, passthrough track + black video vs passthrough track, is it any different?
Posted by
Devgeek
Up vote reply of Devgeek Down vote reply of Devgeek
Add a Comment
Even though it’s just 1 frame, it still must be decoded and presented with every frame you are composing. It’s using extra memory at least. If you are sending all the tracks to the compositor, why not pick 1 track to be the main track instead of having the black frame track? Overall though, you may just have to try various things and evaluate whether the performance is acceptable for your use case.
Posted by
Media Engineer
Post not yet marked as solved Up vote reply of Media Engineer Down vote reply of Media Engineer
Add a Comment