MetalKit in SwiftUI

Hi,

Same question here :

I have successfully displayed a SceneKit View in SwifUI (aka SCNView) using UIViewRepresentable, but failed to do this with MetalKit View (aka MTKView).

Thank you in advance if someone as an idea how to do that.

Here's a condensed implementation without doing that much other than rendering CIFilter on images. I have another MTKView that's actually running a compute pipeline and I banged my head a bit figuring out how to organize the code. Mainly the pattern sorta expect you to instantiate everything and leaving the Coordinator to handle actual delegation. However, for an MTKView, you are stuck either instantiating everything within the Coordinator itself or writing an additional Renderer Coordinator class to instantiate everything within. I didn't feel like going down the separate renderer route, so I just passed MTKView straight into the Coordinator and do the typical setup there.

struct MTKMapView: UIViewRepresentable {
    typealias UIViewType = MTKView
    var mtkView: MTKView
            
    func makeCoordinator() -> Coordinator {
        Coordinator(self, mtkView: mtkView)
    }
    
    func makeUIView(context: UIViewRepresentableContext<MetalMapView>) -> MTKView {
        mtkView.delegate = context.coordinator
        mtkView.preferredFramesPerSecond = 60
        mtkView.backgroundColor = context.environment.colorScheme == .dark ? UIColor.white : UIColor.white
        mtkView.isOpaque = true
        mtkView.enableSetNeedsDisplay = true
        return mtkView
    }
    
    func updateUIView(_ uiView: MTKView, context: UIViewRepresentableContext<MetalMapView>) {
        
    }
    
    class Coordinator : NSObject, MTKViewDelegate {
        var parent: MetalMapView
        var ciContext: CIContext!
        var metalDevice: MTLDevice!

        var metalCommandQueue: MTLCommandQueue!
        var mtlTexture: MTLTexture!
                
        var startTime: Date!
        init(_ parent: MetalMapView, mtkView: MTKView) {
            self.parent = parent
            if let metalDevice = MTLCreateSystemDefaultDevice() {
                mtkView.device = metalDevice
                self.metalDevice = metalDevice
            }
            self.ciContext = CIContext(mtlDevice: metalDevice)
            self.metalCommandQueue = metalDevice.makeCommandQueue()!
            
            super.init()
            mtkView.framebufferOnly = false
            mtkView.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0)
            mtkView.drawableSize = mtkView.frame.size
            mtkView.enableSetNeedsDisplay = true
        }

        func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
            
        }
        
        func draw(in view: MTKView) {
            guard let drawable = view.currentDrawable else {
                return
            }
            let commandBuffer = metalCommandQueue.makeCommandBuffer()
            let inputImage = CIImage(mtlTexture: mtlTexture)!
            var size = view.bounds
            size.size = view.drawableSize
            size = AVMakeRect(aspectRatio: inputImage.extent.size, insideRect: size)
            let filteredImage = inputImage.transformed(by: CGAffineTransform(
                scaleX: size.size.width/inputImage.extent.size.width,
                y: size.size.height/inputImage.extent.size.height))
            let x = -size.origin.x
            let y = -size.origin.y
            
            
            self.mtlTexture = drawable.texture
            ciContext.render(filteredImage,
                to: drawable.texture,
                commandBuffer: commandBuffer,
                bounds: CGRect(origin:CGPoint(x:x, y:y), size: view.drawableSize),
                colorSpace: CGColorSpaceCreateDeviceRGB())

            commandBuffer?.present(drawable)
            commandBuffer?.commit()
        }
        
        func getUIImage(texture: MTLTexture, context: CIContext) -> UIImage?{
            let kciOptions = [CIImageOption.colorSpace: CGColorSpaceCreateDeviceRGB(),
                              CIContextOption.outputPremultiplied: true,
                              CIContextOption.useSoftwareRenderer: false] as! [CIImageOption : Any]
            
            if let ciImageFromTexture = CIImage(mtlTexture: texture, options: kciOptions) {
                if let cgImage = context.createCGImage(ciImageFromTexture, from: ciImageFromTexture.extent) {
                    let uiImage = UIImage(cgImage: cgImage, scale: 1.0, orientation: .downMirrored)
                    return uiImage
                }else{
                    return nil
                }
            }else{
                return nil
            }
        }
    }
}

Same thing here. It would be great if metal kit was more naturally integrated with SwiftUI, without having to go through Coordinators and the likes

Can you please share the MetalMapView code? Is it some kind of SwiftUI View with a UIHostingController?..

Or is it a typo and you meant MTKMapView instead of MetalMapView ?

Any updates of Metal working with SwiftUI this year?

For completeness, here is a standalone example that also creates its own MTKView. To use this view, call it in ContentView's body like so:

Code Block swift
    var body: some View {
        MetalView()
    }

Here is the [NS/UI]ViewRepresentable class:

Code Block swift
import MetalKit
struct MetalView: NSViewRepresentable {
    func makeCoordinator() -> Coordinator {
        Coordinator(self)
    }
    func makeNSView(context: NSViewRepresentableContext<MetalView>) -> MTKView {
        let mtkView = MTKView()
        mtkView.delegate = context.coordinator
        mtkView.preferredFramesPerSecond = 60
        mtkView.enableSetNeedsDisplay = true
        if let metalDevice = MTLCreateSystemDefaultDevice() {
            mtkView.device = metalDevice
        }
        mtkView.framebufferOnly = false
        mtkView.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0)
        mtkView.drawableSize = mtkView.frame.size
        mtkView.enableSetNeedsDisplay = true
        return mtkView
    }
    func updateNSView(_ nsView: MTKView, context: NSViewRepresentableContext<MetalView>) {
    }
    class Coordinator : NSObject, MTKViewDelegate {
        var parent: MetalView
        var metalDevice: MTLDevice!
        var metalCommandQueue: MTLCommandQueue!
        
        init(_ parent: MetalView) {
            self.parent = parent
            if let metalDevice = MTLCreateSystemDefaultDevice() {
                self.metalDevice = metalDevice
            }
            self.metalCommandQueue = metalDevice.makeCommandQueue()!
            super.init()
        }
        func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
        }
        func draw(in view: MTKView) {
            guard let drawable = view.currentDrawable else {
                return
            }
            let commandBuffer = metalCommandQueue.makeCommandBuffer()
            let rpd = view.currentRenderPassDescriptor
            rpd?.colorAttachments[0].clearColor = MTLClearColorMake(0, 1, 0, 1)
            rpd?.colorAttachments[0].loadAction = .clear
            rpd?.colorAttachments[0].storeAction = .store
            let re = commandBuffer?.makeRenderCommandEncoder(descriptor: rpd!)
            re?.endEncoding()
            commandBuffer?.present(drawable)
            commandBuffer?.commit()
        }
    }
}

I recommend watching (https://developer.apple.com/videos/play/wwdc2019/231/) to understand how this works.

The suggested code works great, except I can’t get it to update continuously. It will load perfectly, call draw once, then not call it again unless I’m resizing the window. Is there anyway to call draw continuously?

Nevermind... I just added this in makeNSView and it worked great:

Code Block 
mtkView.isPaused =false

I thought it was false by default.

Thanks for sharing the code. But the code is not working for me, got errors like "Cannot find type NSViewRepresentable in scope".

Here is another resource regarding to UIKit Integration (available in YouTube) into SwiftUI by Stanford.

In response to songyeah's note: "Thanks for sharing the code. But the code is not working for me, got errors like "Cannot find type NSViewRepresentable in scope".

I just tried the posted MetalView struct, and it ran fine with the macOS simulator. However, switching to the iOS simulator, I get the same error that you described. I then changed all of the NSView... commands to instead be UIView... commands, and it now runs with the iOS simulator.

(By the way, this code is very helpful for me!) Thanks.

Hi rthart, thanks for your response. I had exactly the same experience like you, and now it is working. Cheers!

Replying to @crawforb: "The suggested code works great, except I can’t get it to update continuously. It will load perfectly, call draw once, then not call it again unless I’m resizing the window. Is there anyway to call draw continuously?"

I had the same issue. The draw() function in coordinator was only called once. I am wondering if you have solved this problem. Thanks.

oops, solved the problem by adding mtkView.isPaused = false, and commenting other settings.

I am curious how to interface with Metal shaders. The above MetalView code provided by the Graphics & Games engineer works, and works when I implement the Coordinator class in a separate file.

The issue I'm having seems to be when we set the mtkView delegate:

Code Block 
func makeUIView(context: UIViewRepresentableContext<MetalMapView>) -> MTKView {
	mtkView.delegate = context.coordinator
	...
}

because (see MakeCoordinator() https://developer.apple.com/documentation/swiftui/nsviewrepresentable/makecoordinator()-9e4i4 )

SwiftUI calls this method before calling the makeUIView(context:) method.

(of course the same is also true in the case of NSViewRepresentable, makeNSView )

My Coordinator class is just a renderer of sorts, and during initialization requires an MTKView. This is where I am having trouble. I am unsure of how (or where?) to create an MTKView for passing to my Coordinator / MTKViewDelegate. For example, in my Coordinator class, I attempt to obtain the aspect ratio by

Code Block 
var aspect = Float(mtkView.drawableSize.width / mtkView.drawableSize.height)

which returns NaN or some equivalent nonsense. I guess I do not have a properly initialized MTKView or something, I am not entirely sure how or where to do this (disclaimer: I am still fairly new to Metal and Swift programming). Furthermore, I do not know where I should be storing the MTKView -- should it be as @State or @Environment? Should it not be stored or initialized inside the MetalView struct?

I do not have issues when I do the same thing in a non-SwiftUI setup, e.g. in

Code Block 
class ViewController: UIViewController {
    var mtkView: MTKView!
	...
    override func viewDidLoad() {
        super.viewDidLoad()
        guard let mtkViewTemp = self.view as? MTKView else {...} 
        mtkView = mtkViewTemp
		...
}

and as far as I know making the (naive, speaking for myself) switch from UIViewRepresentable to UIViewControllerRepresentable is not a fix. As I stated a few lines above, the issue seems to be with how I am attempting to create, store, and use the MTKView.

Thanks everyone for posting your experiences.

I'm building a photo filter app (hobby project) from scratch for macOS. and got here when researching why my sliders started to become too slow to use and what to do about it.

So I added the example by Graphics and Games Engineer  to my project to replace my "ImageView" and display my CI / CG images directly without converting them to NSImage.

So far I got a working view, which shows a green background, but I am unable to figure out how to display my image on this, what to pass as context to view as it wants an NSViewRepresentableContext and the only thing I have is a CIContext...

I think I want to be able to display filter.outputImage... here's an example of how I make this:

Code Block Swift
func ciExposure (inputImage: CIImage, inputEV: Double) -> CIImage {
    let filter = CIFilter(name: "CIExposureAdjust")!
    filter.setValue(inputImage, forKey: kCIInputImageKey)
    filter.setValue(inputEV, forKey: kCIInputEVKey)
    return filter.outputImage!
}

Can someone give me a hint about how to do this? I don't mind doing research but it starts to feel like I've hit a wall here...

Thanks in advance!