RealityKit visualize the virtual depth texture from post-process callback

Question

Created Feb ’24

Replies 0

Boosts 0

Participants 1

I am using RealityKit and ARView PostProcessContext to get the sourceDepthTexture of the current virtual scene in RealityKit, using .nonAR camera mode.

My experience with Metal is limited to RealityKit GeometryModifier and SurfaceShader for CustomMaterial, but I am excited to learn more! Having studied the Underwater sample code I have a general idea of how I want to explore the capabilities of a proper post processing pipeline in my RealityKit project, but right now I just want to visualize this MTLTexture to see what the virtual depth of the scene looks like.

Here’s my current approach, trying to create a depth UIImage from the context sourceDepthTexture:

func postProcess(context: ARView.PostProcessContext) {
    let depthTexture = context.sourceDepthTexture
    var uiImage: UIImage? // or cg/ci
    
    if processPost { 
       print("#P Process: Post Processs BLIT")

        // UIImage from MTLTexture
        uiImage = try createDepthUIImage(from: depthTexture)
       
        let blitEncoder = context.commandBuffer.makeBlitCommandEncoder()
        blitEncoder?.copy(from: context.sourceColorTexture, to: context.targetColorTexture)
        blitEncoder?.endEncoding()
        getPostProcessed()
    } 

    else {
        print("#P No Process: Pass-Through")

        let blitEncoder = context.commandBuffer.makeBlitCommandEncoder()
        blitEncoder?.copy(from: context.sourceColorTexture, to: context.targetColorTexture)
        blitEncoder?.endEncoding()
    }
}

func createUIImage(from metalTexture: MTLTexture) throws -> UIImage {
    guard let device = MTLCreateSystemDefaultDevice() else { throw CIMError.noDefaultDevice }
    
    let descriptor = MTLTextureDescriptor.texture2DDescriptor(
        pixelFormat: .depth32Float_stencil8,
        width: metalTexture.width,
        height: metalTexture.height,
        mipmapped: false)
    descriptor.usage = [.shaderWrite, .shaderRead]
    
    guard let texture = device.makeTexture(descriptor: descriptor) else {
        throw NSError(domain: "Failed to create Metal texture", code: -1, userInfo: nil)
    }
    
    // Blit!
    let commandQueue = device.makeCommandQueue()
    let commandBuffer = commandQueue?.makeCommandBuffer()
    let blitEncorder = commandBuffer?.makeBlitCommandEncoder()
    
    blitEncorder?.copy(from: metalTexture, to: texture)
    blitEncorder?.endEncoding()
    commandBuffer?.commit()
    
    // Raw pixel bytes
    let bytesPerRow = 4 * texture.width
    let dataSize = texture.height * bytesPerRow
    var bytes = [UInt8](repeating: 0, count: dataSize)
    //var depthData = [Float](repeating: 0, count: dataSize)
    
    bytes.withUnsafeMutableBytes { bytesPtr in
        texture.getBytes(
            bytesPtr.baseAddress!,
            bytesPerRow: bytesPerRow,
            from: .init(origin: .init(), size: .init(width: texture.width, height: texture.height, depth: 1)),
            mipmapLevel: 0
        )
    }
    
    // CGDataProvider from the raw bytes
    let dataProvider = CGDataProvider(data: Data(bytes: bytes, count: bytes.count) as CFData)
    
    // CGImage from the data provider
    let cgImage = CGImage(width: texture.width,
                          height: texture.height,
                          bitsPerComponent: 8,
                          bitsPerPixel: 32,
                          bytesPerRow: bytesPerRow,
                          space: CGColorSpaceCreateDeviceRGB(),
                          bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedLast.rawValue),
                          provider: dataProvider!,
                          decode: nil,
                          shouldInterpolate: true,
                          intent: .defaultIntent)
    
    // Return as UIImage
    return UIImage(cgImage: cgImage!)
}

I have hacked together the ‘createUIImage’ function with generative aid and online research to provide some visual feedback, but it looks like I am converting the depth values incorrectly — or somehow tapping into the stencil component of the pixels in the texture.

Either way I am out of my depth, and would love some help.

Ideally, I would like to produce a grayscale depth image, but really any guidance on how I can visualize the depth would be greatly appreciated.

As you can see from the magnified view on the right, there are some artifacts or pixels that are processed differently than the core stencil. The empty background is transparent in the image as expected.

Boost