MTKView Drawing Performance

What I am Trying to Do

I am trying to show filters on a camera feed by using a Metal view:

MTKView

What I Have So Far

Following code works great (mainly interpreted from above-mentioned sample code) :

class MetalObject: NSObject, MTKViewDelegate {
    
            private var metalBufferView         : MTKView?
            private var metalDevice             = MTLCreateSystemDefaultDevice()
            private var metalCommandQueue       : MTLCommandQueue!
    
            private var ciContext               : CIContext!
            private let colorSpace              = CGColorSpaceCreateDeviceRGB()
    
            private var videoPixelBuffer        : CVPixelBuffer?

            private let syncQueue               = DispatchQueue(label: "Preview View Sync Queue", qos: .userInitiated, attributes: [], autoreleaseFrequency: .workItem)
    
            private var textureWidth            : Int             = 0
            private var textureHeight           : Int             = 0
            private var textureMirroring        = false
            private var sampler                 : MTLSamplerState!
            private var renderPipelineState     : MTLRenderPipelineState!
            private var vertexCoordBuffer       : MTLBuffer!
            private var textCoordBuffer         : MTLBuffer!
            private var internalBounds          : CGRect!
            private var textureTranform         : CGAffineTransform?
    
            private var previewImage            : CIImage?
    
    init(with frame: CGRect) {
        super.init()
        
        self.metalBufferView = MTKView(frame: frame, device: self.metalDevice)
        self.metalBufferView!.contentScaleFactor = UIScreen.main.nativeScale
        self.metalBufferView!.framebufferOnly = true
        self.metalBufferView!.colorPixelFormat = .bgra8Unorm
        self.metalBufferView!.isPaused = true
        self.metalBufferView!.enableSetNeedsDisplay = false
        self.metalBufferView!.delegate = self
        
        self.metalCommandQueue = self.metalDevice!.makeCommandQueue()

        self.ciContext = CIContext(mtlDevice: self.metalDevice!)

        
        //Configure Metal
        let defaultLibrary = self.metalDevice!.makeDefaultLibrary()!
        let pipelineDescriptor = MTLRenderPipelineDescriptor()
        pipelineDescriptor.colorAttachments[0].pixelFormat = .bgra8Unorm
        pipelineDescriptor.vertexFunction = defaultLibrary.makeFunction(name: "vertexPassThrough")
        pipelineDescriptor.fragmentFunction = defaultLibrary.makeFunction(name: "fragmentPassThrough")
        
        // To determine how our textures are sampled, we create a sampler descriptor, which
        // will be used to ask for a sampler state object from our device below.
        let samplerDescriptor = MTLSamplerDescriptor()
        samplerDescriptor.sAddressMode = .clampToEdge
        samplerDescriptor.tAddressMode = .clampToEdge
        samplerDescriptor.minFilter = .linear
        samplerDescriptor.magFilter = .linear
        
        sampler = self.metalDevice!.makeSamplerState(descriptor: samplerDescriptor)
        
        do {
            renderPipelineState = try self.metalDevice!.makeRenderPipelineState(descriptor: pipelineDescriptor)
        } catch {
            fatalError("Unable to create preview Metal view pipeline state. (\(error))")
        }
        
    }
    
    

    
    final func update (newVideoPixelBuffer: CVPixelBuffer?) {
        
        self.syncQueue.async {
            
            var filteredImage : CIImage
            
            self.videoPixelBuffer = newVideoPixelBuffer
            
            //---------
            //Core image filters
            //Strictly CIFilters, chained together
            //---------
            
            self.previewImage = filteredImage
    
            //Ask Metal View to draw
            self.metalBufferView?.draw()
            
        }
    }
    
    
    
    //MARK: - Metal View Delegate
    final func draw(in view: MTKView) {
        
        print (Thread.current)
    
        guard let drawable = self.metalBufferView!.currentDrawable,
            let currentRenderPassDescriptor = self.metalBufferView!.currentRenderPassDescriptor,
            let previewImage = self.previewImage else {
                return
        }
        
        
        // create a texture for the CI image to render to
        let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor(
            pixelFormat: .bgra8Unorm,
            width: Int(previewImage.extent.width),
            height: Int(previewImage.extent.height),
            mipmapped: false)
        textureDescriptor.usage = [.shaderWrite, .shaderRead]
        
        let texture = self.metalDevice!.makeTexture(descriptor: textureDescriptor)!
        
        if texture.width != textureWidth ||
            texture.height != textureHeight ||
            self.metalBufferView!.bounds != internalBounds {
            setupTransform(width: texture.width, height: texture.height, mirroring: mirroring, rotation: rotation)
        }
        
        // Set up command buffer and encoder
        guard let commandQueue = self.metalCommandQueue else {
            print("Failed to create Metal command queue")
            return
        }
        
        guard let commandBuffer = commandQueue.makeCommandBuffer() else {
            print("Failed to create Metal command buffer")
            return
        }
        
        // add rendering of the image to the command buffer
        ciContext.render(previewImage,
                         to: texture,
                         commandBuffer: commandBuffer,
                         bounds: previewImage.extent,
                         colorSpace: self.colorSpace)
        
        guard let commandEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: currentRenderPassDescriptor) else {
            print("Failed to create Metal command encoder")
            return
        }
        
        // add vertex and fragment shaders to the command buffer
        commandEncoder.label = "Preview display"
        commandEncoder.setRenderPipelineState(renderPipelineState!)
        commandEncoder.setVertexBuffer(vertexCoordBuffer, offset: 0, index: 0)
        commandEncoder.setVertexBuffer(textCoordBuffer, offset: 0, index: 1)
        commandEncoder.setFragmentTexture(texture, index: 0)
        commandEncoder.setFragmentSamplerState(sampler, index: 0)
        commandEncoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4)
        commandEncoder.endEncoding()
        
        commandBuffer.present(drawable) // Draw to the screen
        commandBuffer.commit()
        
    }
    
    
    final func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
        
    }
    
}

Notes

• The reason

  MTKViewDelegate

  is used instead of subclassing

  MTKView

  is that when it was subclassed, the draw call was called on the main thread. With the delegate method shown above, it seems to be a different metal related thread call each loop. Above method seem to give much better performance.
• Details on

  CIFilter

  usage on update method above had to be redacted. All it is a heavy chain of

  CIFilters

  stacked.

Issue

Above code seems to slow down the main thread a lot, causing rest of the app UI to be choppy. For example, scrolling a

UIScrollview

Goal

Tweak Metal view to ease up on CPU and go easy on the main thread to leave enough juice for rest of the UI.

Preparation of command buffer is all done in CPU until presented and committed(?). Is there a way to offload that from CPU?

Any hints, feedback, tips, etc to improve the drawing efficiency would be appreciated.