Lagging Video Feed Using VNGeneratePersonSegmentationRequest in macOS Camera Extension App

I'm developing a macOS application using Swift and a camera extension. I'm utilizing the Vision framework's VNGeneratePersonSegmentationRequest to apply a background blur effect. However, I'm experiencing significant lag in the video feed. I've tried optimizing the request, but the issue persists. Could anyone provide insights or suggestions on how to resolve this lagging issue? Details:

Platform: macOS
Language: Swift
Framework: Vision

code snippet I am using are below `class ViewController: NSViewController, AVCaptureVideoDataOutputSampleBufferDelegate { var frameCounter = 0 let frameSkipRate = 2 private let visionQueue = DispatchQueue(label: "com.example.visionQueue")

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    frameCounter += 1
    if frameCounter % frameSkipRate != 0 {
        return
    }
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
    let ciImage = CIImage(cvPixelBuffer: pixelBuffer)
    performPersonSegmentation(on: ciImage) { [self] mask in
        guard let mask = mask else { return }
        let blurredBackground = self.applyBlur(to: ciImage)
        let resultImage = self.composeImage(with: blurredBackground, mask: mask, original: ciImage)
        let nsImage = ciImageToNSImage(ciImage: resultImage)
       
        DispatchQueue.main.async { [self] in
            // Update your NSImageView or other UI elements with the composite image
            
            if needToStream {
                if (enqueued == false || readyToEnqueue == true), let queue = self.sinkQueue {
                    enqueued = true
                    readyToEnqueue = false
                    if let _ = image, let cgImage = nsImage.cgImage(forProposedRect: nil, context: nil, hints: nil) {
                        enqueue(queue, cgImage)
                    }
                }
            }
        }
    }
}

private func performPersonSegmentation(on image: CIImage, completion: @escaping (CIImage?) -> Void) {
    let request = VNGeneratePersonSegmentationRequest()
    request.qualityLevel = .fast // Adjust quality level as needed
    request.outputPixelFormat = kCVPixelFormatType_OneComponent8
    let handler = VNImageRequestHandler(ciImage: image, options: [:])
    visionQueue.async {
        do {
            try handler.perform([request])
            guard let result = request.results?.first as? VNPixelBufferObservation else {
                completion(nil)
                return
            }
            let maskPixelBuffer = result.pixelBuffer
            let maskImage = CIImage(cvPixelBuffer: maskPixelBuffer)
            completion(maskImage)
        } catch {
            print("Error performing segmentation: \(error)")
            completion(nil)
        }
    }
}

private func composeImage(with blurredBackground: CIImage, mask: CIImage, original: CIImage) -> CIImage {
    // Invert the mask to blur the background
    let invertedMask = mask.applyingFilter("CIColorInvert")

    // Ensure mask is correctly resized to match original image
    let resizedMask = invertedMask.transformed(by: CGAffineTransform(scaleX: original.extent.width / invertedMask.extent.width, y: original.extent.height / invertedMask.extent.height))
    
    // Blend the images using the mask
    let blendFilter = CIFilter(name: "CIBlendWithMask")!
    blendFilter.setValue(blurredBackground, forKey: kCIInputImageKey)
    blendFilter.setValue(original, forKey: kCIInputBackgroundImageKey)
    blendFilter.setValue(resizedMask, forKey: kCIInputMaskImageKey)

    return blendFilter.outputImage ?? original
}
 
private func ciImageToNSImage(ciImage: CIImage) -> NSImage {
    let cgImage = context.createCGImage(ciImage, from: ciImage.extent)!
    return NSImage(cgImage: cgImage, size: ciImage.extent.size)
}

private func applyBlur(to image: CIImage) -> CIImage {
    let blurFilter = CIFilter.gaussianBlur()
    blurFilter.inputImage = image
    blurFilter.radius = 7.0 // Adjust the blur radius as needed
    return blurFilter.outputImage ?? image
}

}`

Hello @yogesh25,

I definitely see a few opportunities to optimize the performance of this code, however, the first step to performance optimization would be to measure your app with Instruments, using the Time Profiler instrument.

Time Profiler will help you identify which lines of code are the most expensive, and that is where you should focus your optimization efforts.

After profiling, if you aren't sure how you can improve the performance, please provide a link to your instruments trace.

Best regards,

Greg

Hi @Greg, Thanks for sharing this i have already use time instrument when i remove below code it is working fine VNGeneratePersonSegmentationRequest framework taking time let request = VNGeneratePersonSegmentationRequest() request.qualityLevel = .fast // Adjust quality level as needed request.outputPixelFormat = kCVPixelFormatType_OneComponent8 let handler = VNImageRequestHandler(ciImage: image, options: [:])

Hello, @yogesh25,

Thanks for sharing this i have already use time instrument when i remove below code it is working fine VNGeneratePersonSegmentationRequest framework taking time let request = VNGeneratePersonSegmentationRequest() request.qualityLevel = .fast // Adjust quality level as needed request.outputPixelFormat = kCVPixelFormatType_OneComponent8 let handler = VNImageRequestHandler(ciImage: image, options: [:])

Ok, but person segmentation is the feature you are trying to build, so removing it is not the answer to this performance optimization problem. Can you provide a link to your Instruments trace?

Best regards,

Greg

Lagging Video Feed Using VNGeneratePersonSegmentationRequest in macOS Camera Extension App
 
 
Q