HEVC Video with Alpha in SwiftUI

Question

Created Jul ’22

Replies 3

Boosts 0

Views 1.7k

Participants 2

After stumbling upon this video from WWDC '19, I have been trying to get transparent HEVC (H.265) videos to play in SwiftUI. Unfortunately I have been unable to get it working. Is this truly supported in iOS and tvOS 13+?

Here is my code:


import AVKit

import SceneKit

import SpriteKit



struct VideoView: View {

    @State var videoPlayer = AVPlayer(url:  Bundle.main.url(forResource: "fl", withExtension: "mov")!)

    @State var mainScene = SKScene(size: CGSize(width: 500, height: 500))

    

    var body: some View {

        HStack {

            Spacer()

            

            VStack {

                

                Spacer()

                SpriteView(scene: mainScene)

                

                Spacer()

                

            }.onAppear {

                guard let scene = SKScene(fileNamed: "backgroundScene") else {

                    print ("Could not create a background scene")

                    return

                }

                

                scene.scaleMode = .aspectFill

                scene.backgroundColor = .blue

                scene.view?.allowsTransparency = true

                guard let alphaMovieURL = Bundle.main.url(forResource: "output", withExtension: "mov") else {

                    print("Failed to overlay alpha movie on the background")

                    return

                }

                

                

                videoPlayer = AVPlayer(url: alphaMovieURL)

                

                let video = SKVideoNode(avPlayer: videoPlayer)

                video.size = CGSize(width: 500, height: 500)

                scene.addChild(video)

                videoPlayer.play()

                mainScene = scene

            }

            Spacer()

        }.background(Color.red)

    }

}

The file I have been using can be found here:

https://firebasestorage.googleapis.com/v0/b/leaderboard-d5992.appspot.com/o/fl.mov?alt=media&token=c939cdfc-5047-4c86-a38d-89d5a0f3c459

It very well could be that the encoding is wrong, although I have checked it many times and it definitely contains an alpha layer.

Any direction or pointers would be greatly appreciated!

Boost

Answer 1

nparcher24 OP

Jul ’22

Also, for clarification, this works but contains a black background. The desired output would have a transparent background :)

0

Answer 2

DTS Engineer OP

Apple

Jul ’22

This appears to be working for me on macOS and iOS at least (didn't check tvOS, but suspect it works).

I slightly modified your code:

struct VideoView: View {

    @State var mainScene = SKScene()

    var body: some View {

        HStack {
            Spacer()
            VStack {
                Spacer()
                SpriteView(scene: mainScene)
                Spacer()
            }.onAppear {
                let scene = SKScene(size: .init(width: 500, height: 500))
                scene.scaleMode = .aspectFit
                scene.backgroundColor = .blue
                scene.view?.allowsTransparency = true
                
                let player = AVPlayer(url:  Bundle.main.url(forResource: "output", withExtension: "mov")!)


                let video = SKVideoNode(avPlayer: player)
                video.size = CGSize(width: 500, height: 500)
                video.anchorPoint = .init(x: 0, y: 0)
                scene.addChild(video)
                player.play()
                mainScene = scene
            }
            Spacer()
        }.background(Color.red)
    }
}

I end up seeing the video play, and the background color is whatever the scene's background color is (in this case, blue), which is the expected behavior.

0

Answer 3

nparcher24 OP

Jul ’22

Thanks for the quick reply. It appears there are other issues at play here in my system.

The above code runs on the tvOS simulator but has a black background still. iOS and macOS both spit strange errors (of which I'm researching at the moment):

'-[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5252: failed assertion `Draw Errors Validation

Fragment Function(FastSingle_FragFunc): missing sampler binding at index 0 for u_texture_sampler[0].'

For simplicity sake I made a GitHub repo here:

https://github.com/nparcher24/test-transparency.git

Does anyone else see something missing here? Is it my machine having this issue?

0