Overlay two videos using AVMutableComposition?

The trick is that you have to specify the way the videos are composed using "instructions". See the "instructions" property of an AVMutableComposition, and the AVVideoCompositionInstruction and AVVideoCompositionLayerInstruction classes. These are kinda inconvenient to use, but once you get down to the layer instruction level, you'll be able to scale and position a video track over another one.

Note that the background color in the layer instructions may default to opaque black, which means by default the topmost layer will obscure all of the rest. Set the background color explicitly to clear if you run into that problem.

Thanks for the response. I'm attempting to work with these classes now.

The background color property on AVVideoCompositionInstruction is read-only. How do I set it?

Those classes have an "AVMutable…" variant that lets you set the properties.

Thanks. I'm making some progress.

I also need to crop my video inset such that it appears as a circle instead of a rectangle. Is this possible?

Yikes! I don't think this is possible with pre-existing AVFoundation components. I assume you can do it with a custom compositor for the AVMutableComposition (implementing the AVVideoCompositing protocol). This should be functionally straightforward, but you'll have to make sure the performance of your code is acceptable. Not an area I'm familiar with, so welcome to the Twilight Zone!

Well, I had something partially working a couple of hours ago, but now all I get is a video that contains a fraction of a second of my overlay at the beginning (on a black background) followed by the background video with no overlay.

I definitely had it at one point where I could see both videos for the full duration, and cannot for the life of me figure out what I changed that wrecked it.

I set the background color of my AVMutableVideoCompositionInstruction to the clear color. Is this the only place I need to do this?

Any idea what else could be going wrong? I put the smaller video (the one I want overlaid) into my composition first, so it should appear above the other video, but it doesn't appear at all.

It seems to **** up every time I try to take a step toward what I want.

For example, I can get the overlaid image to appear over the top of the main image, but it is rotated in the wrong direction.

I tried to rotate it by applying a rotation transform, and instead of rotating, it just disappeared.

It's so annoying. I could make progress if, for example, I saw that it rotated the wrong amount or in the wrong direction, but instead it just totally stops working. How can this be?

"The origin of the crop rectangle is the top-left corner of the buffer clean aperture rectangle".

What does this mean?

What is a "buffer clean aperture rectangle"?

I created a square crop rectangle (same width and height) but my output is a rectangular picutre. Perhaps I'm not understanding how a crop rectangle works. How do I crop a square area out of a rectangular video?

I don't know all the answers here. According to lurkertech.com/lg/video-systems/#aperture:

"The clean aperture, which is co-centric with the production aperture and the picture center, and contained within the production aperture, is that sub-rectangle of the video which has the system's standard picture aspect ratio (e.g. 4:3 for standard def)."

(where the production aperture is the actual frame size).

If a rotation transform is applied to a coordinate system with the origin in the top left corner (of whatever), rotations of 90 degrees or more (in either direction) will rotate the video out of the frame. You'll likely need a translation as well. Try using a small rotation (like 30 degrees) instead, until you see what's going on geometrically.

I don't know about the square crop. Are the pixels square? Again, I would try a small change from the original until you can see what's going on.

Thanks for all the input.

The goal of this task is to shoot a square video with the front-facing camera and overlay it on another video at a particular location.

I realized that the video shot from the camera ends up in landscape orientation, so I had to rotate it 90 degrees, then scale it down, and finally crop it. I managed to do these things and end up with a square video frame of the right size and orientation.

Where I'm stuck right now is figuring out how to place that frame in the right location. I know the exact size of the background video, and I know the exact x,y coordinates where I want the overlay to appear, but I can't get it to appear there (using CGAffineTransformMakeTranslation). I tried swapping x and y figuring maybe they were wrong because of the rotation, but that didn't help. And the amount that I'm off is not an obvious amount, like a factor of 2 or something. It's just wrong.

One difficulty I always run into is the order of the elementary transformations when one of them is translation, because then they're no longer commutative and the order matters. Apple isn't very clear about the order (there are equivalent pre- and post-multiplication forms of the transform arithmetic, and you need to know which you're doing). On top of that, there may be a invisible coordinate flip transform (depending on whether the y coordinates increase from the top or the bottom), as well as a possible origin translation due to the semantics of the video frame geometry.

Try doing just a single translation (ignoring size and rotation) and see if you can get rational placement. If so, trying adding in a small second transform again, and see if it's clear how it interacts with the translation.

It ain't fun.

Their is a tecnical note that may help with Debugging AVFoundation Compositions, Video Compositions.

Technical Note TN2447

Debugging AVFoundation Compositions, Video Compositions, and Audio Mixes

https://developer.apple.com/library/content/technotes/tn2447/_index.html

You're right, doing the rotation in smaller steps helped me see what was happening. The rotation was occuring not around the center of the view, as I expected, but around the upper-left hand corner of the original view (even though I cropped it, it was still rotating as if it were not cropped). Armed with that information I was able to compute the correct offset that caused the visible portion of the overlay to fall into the right position after it was rotated.

The last thing I want to do is round off the corners of the overlay so it appears as a circle instead of a square. Is there any concept of transparency in video composition? In other words, could I put the overlay behind the main video and have it partially appear through a transparent area?

Frank

Yes, I need an AVVideoComposting class to do this. The good news is, I found an example that does almost exactly what I want. The bad news is, I have to recreate everything I spent two days doing using AVVideoCompositionInstructions, because the custom video composter doesn't honor any of that code, it expects me to scale, rotate and draw the overlay myself. That's a bummer, but I guess it can be done.