Need to get the frames of an animation for streaming in 3rd party video format (macOS)

Question

Created Jun ’24

Replies 3

Boosts 0

Participants 2

I have a macos app that creates a sports scoreboard and streams this out in a third party streaming format. I need to use a library for this, which expects a CVPixelBuffer e.g. every 0.04 secs (25 fps). The scoreboard is a view with some subviews (scores, timers, etc) in an offscreen window. When a timer triggers to create a new frame, my code does:

• create a bitmap-representation for the view

• make a view.cacheDisplay(...)

• create a CIImage from the bitmap-representation

• create a CVPixelBuffer using CVPixelBufferCreate (kCVPixelFormatType_32BGRA format)

• render the CIImage into the CVPixelBuffer using ciContext.render(ciImage, to: pixelBufferOut)

• and finally hand over the CVPixelBuffer to the video library.

Works, the app is in production.

Now I come to the point where I need someone to tell me in which direction I have to think. The next feature requested is to do some animation for the scoreboard graphics , e.g. a simple "fade in" and "fade out". Later I would like to do more complex things, but let's just look at that. When I add a CABasicAnimation(keyPath: #keyPath(CALayer.opacity)) for fade in / fade out to the scoreboard base view (all views with layer), I can see it smoothly fading after I moved my offscreen to onscreen, so the animation works on screen. But in the video stream there is no animation, no fading, the content is shown with opacity 0.0 at one frame and in the next frame 1.0 (fade in) and the other way round for fade out. I have a not-animated, always visible text field displaying a timecode in the video screen and that shows every timecode in the video, so the frames are in fact generated, but contains no scoreboard view with an opacity other than 0 or 1.

How can I get the rendered frames from an animation? Is it somehow possible with Core Animation? Do I have to dig into Metal? I searched for hours to find something, but "animation" leads to discussions about animations on screen, while "video" is always about AVFoundation, but there is this library I have to use.

Thanks for any hints,

Nina

Answered by Graphics and Games Engineer in 793172022

You should probably change your approach to use ScreenCaptureKit, which gives you a stream of CMSampleBuffers taken directly from the window server. The approach you’re currently using cannot reproduce animations or certain other CoreAnimation-based effects.

Boost

Answer 1

Graphics and Games Engineer OP

Apple

Jun ’24

Accepted Answer

You should probably change your approach to use ScreenCaptureKit, which gives you a stream of CMSampleBuffers taken directly from the window server. The approach you’re currently using cannot reproduce animations or certain other CoreAnimation-based effects.

0

Answer 2

Ninakadin OP

Jun ’24

I implemented your suggested approach with ScreenCaptureKit and reached good results. I had some doubts whether this would deliver the alpha channel as needed (for keying in the switcher), but it does. Now I have nice animations in the stream. Just one detail, I can live with it, but I wonder if there is a solution: Since I want 50 frames per second I set the minimumFrameInterval in the SCStreamConfiguration to

CMTime(value: 1, timescale: CMTimeScale(50))

What I receive is around 47 frames. Is there way to get a more precise interval? With requesting 25 fps I receive 23 or 24 frames per second. The interval is most of time slightly above the calculated value (1-4 microseconds) and this sums up. I understand that the parameter says MINIMUM frame interval.

Anyway, thank you for the answer, I was able to get things done!

0

Answer 3

Ninakadin OP

Jun ’24

When optimising the first proof of concept, a major problem shows: It works fine as long as the content to capture is onscreen. As I said, I'm streaming, I'm creating content that is not meant to be shown on the Mac screen. There is just a window controlling the scoreboard. As soon the capture window is offscreen, the output is stopped. macOS does not want to waste computing power rendering invisible screens. The interface of

SCShareableContent.excludingDesktopWindows(true, onScreenWindowsOnly: false)

is misleading here. It means "partially offscreen windows". Windows completely offscreen will not even be offered as possible content.

All I can do is keeping a pixel of the output window onscreen and change it frequently, but first of all this is tinkering around and secondly the animations get jerky.

Is there no way to prevent macOS from "optimising" the rendering this way?

And if not, what would be another approach? Forget about the macOS animations and do everything yourself?

0