Using MPSImageLaplacianPyramid for image processing

Hello there,


I've been going through attempting to implement some image processing techniques that require the use of creating a Laplacian Pyramid. Now I'm not exactly sure that MPSImageLaplacianPyramid is completly useable yet as the documentation for the page is not filled out, but I've found places that it requires taking the input of a gaussian pyramid. I am, however, getting very strange outputs that don't even look anything like what you would expect out of the level of one of the pyramids, further I believe that there may be some sort of pixel format restriction, but I am not sure.


I would like to take advantage of the speed of mipmapping to create the pyramids, as I'm sure anything I come up with is going to be far slower. Some things that I have noticed is that the pixel format seems to matter. When creating the Laplacian Pyramid, you take the difference of an upsampled lower level of the guassian with another level, hence negative values possibly could get clamped to 0, and so we get weird looking outputs. I can not for the life of me figure out if the filter is working as intended or not. Below is my code:


import Cocoa
import Metal
import MetalPerformanceShaders
import MetalKit


class ViewController: NSViewController, MTKViewDelegate {
    func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {}
    func draw(in view: MTKView) {}
    
    override func viewDidLoad() {
        super.viewDidLoad()
        //Metal Setup
        let device = MTLCreateSystemDefaultDevice()!
        let commandQueue = device.makeCommandQueue()!
        let textureLoader = MTKTextureLoader(device: device)
        let commandBuffer = commandQueue.makeCommandBuffer()!
        let gauss = MPSImageGaussianPyramid(device: device, centerWeight: 0.375)
        let lap = MPSImageLaplacianPyramid(device: device)
        
        
        //Extract Image into MTLTexture
        let url = Bundle.main.url(forResource: "flower", withExtension: "png")!
        let options = [MTKTextureLoader.Option.textureUsage : NSNumber(value:MTLTextureUsage.shaderRead.rawValue |
                       MTLTextureUsage.shaderWrite.rawValue | MTLTextureUsage.pixelFormatView.rawValue),
                       MTKTextureLoader.Option.SRGB: false, MTKTextureLoader.Option.allocateMipmaps : true]
        var texIn = try! textureLoader.newTexture(URL: url, options: options)
        
        
        //View Setup
        let metalView = MTKView(frame: CGRect(x: 0, y: 0, width: texIn.width, height: texIn.height))
        metalView.framebufferOnly = false
        metalView.device = device
        metalView.isPaused = true
        metalView.colorPixelFormat = .bgra8Unorm
        view.addSubview(metalView)
        

        //Intermediate Texture for Pyramid
        var intD = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .bgra8Unorm, 
                   width: Int(texIn.width), height: Int(texIn.height), mipmapped: true)
        intD.usage = MTLTextureUsage(rawValue: ( MTLTextureUsage.shaderWrite.rawValue |
                                                 MTLTextureUsage.shaderRead.rawValue |
                                                 MTLTextureUsage.pixelFormatView.rawValue))
        var intT = device.makeTexture(descriptor: intD)!

        
        //Image to Pyramid, Laplacian requires a Guassian Pyramid as input
        gauss.encode(commandBuffer: commandBuffer, inPlaceTexture: &texIn, fallbackCopyAllocator: nil)
        lap.encode(commandBuffer: commandBuffer, sourceTexture: texIn, destinationTexture: intT)

        //Grab a particular level for viewing
        let blitEncoder = commandBuffer.makeBlitCommandEncoder()!

        blitEncoder.copy(from: intT, sourceSlice: 0, sourceLevel: 0, 
                         sourceOrigin: MTLOrigin(x: 0, y: 0, z: 0),
                         sourceSize: MTLSize(width: texIn.width, height: texIn.height, depth: 1),
                         to: (metalView.currentDrawable?.texture)!,
                         destinationSlice: 0, destinationLevel: 0, destinationOrigin: MTLOrigin(x: 0, y: 0, z: 0))
        blitEncoder.endEncoding()

        
        commandBuffer.present(metalView.currentDrawable!)
        commandBuffer.commit()
        commandBuffer.waitUntilCompleted()

        // Do any additional setup after loading the view.
    }

    override var representedObject: Any? {
        didSet {
        // Update the view, if already loaded.
        }
    }


}

Replies

I have had exactly the same problem with MPSImageLaplacianPyramid and believe it is a flaw with the way Apple are applying the EXPAND. The headers refer to the upsampling with 'zero-stuffing' then applying the interpolation. But this will reduce the energy in the upsampled image, the interpolation needs to have values that are x2 for both vertical and horizontal convolution. However, you cannot specify a 1x5 matrix for this kernel, only a center weight (which doesn't address the issue) or the full 5x5 matrix kernel. I have tried this with a 5x5 matrix with the x4 weights and now get the correct pyramid. The 5x5 matrix is created by multipling the 5x1 and 1x5 matrices of the form:

{1/4 - a/2, 1/4, a, 1/4, 1/4 - a/2} and ensure each element in this matrix is x2.

Great observation and thank you for the explanation, I was encoutering the same issue! This definitely seems to produce a correct Laplacian. A follow up problem I have when using the x2 gaussian:

1. Generate the Gaussian as described above: looks ok other than brighter mip map levels

2. Generate Laplacian pyramid from the above gaussian: looks ok

3. Collapse the above laplacian using MPSImageLaplacianAdd: this is the one causing issues! Ideally you should get the original gaussian at level 0 which is it does ONLY if the image size is smaller than a certain dimension! For bigger images I am getting black rendering. Is anyone able to reproduce this? This does NOT happen when using normal MPSImageGaussianPyramid without x2 weights (but then the laplacian pyramid is useless)!


The other question I had was: has anyone played with the laplacianBias and laplacianScale parameters available with MPSImageLaplacianPyramid ?