Showing photo from PHPickerViewController in "HDR" mode

How to show photos from PHPickerViewController the way they are shown in Apple's Photos with "View Full HDR" enabled? I've found all EDR-related talks, rendering CIImage into MTKView already... nothing helps, image is same as in UIImageView. How?! :–)

What I do now:

  1. I get photo URL (copy) via provider.loadFileRepresentation(forTypeIdentifier: UTType.image.identifier
  2. I create MTKView with metalLayer.wantsExtendedDynamicRangeContent = true and other recommended settings
  3. I load CIImage from URL provided earlier.
  4. I render CIImage via CIContext backed with mtlCommandQueue with option .useSoftwareRenderer: false.

And I still get "normal" image.

Exact same image is being displayed in Photos app with much brighter whites, and this is exactly what I want to achieve.

Please help :)

Thanks!

Answered by bealex in 739406022

I've found it! You need to setup metalLayer.edrMetadata to something like this: metalLayer.edrMetadata = CAEDRMetadata.hdr10(minLuminance: 0, maxLuminance: 1000, opticalOutputScale: 1000)

I still have to figure out the parameters and how to use all that (I see problems after app switching for example), but it works!

If you are fetching assets using the “public.image” UTI, you can set preferredAssetRepresentationModeto .current to avoid image transcoding. Alternatively, you can request “public.heif” UTI directly.

Tried that. No luck :–(

Here's some more code. Maybe my mistake will be visible somewhere here.

PHPickerConfiguration configuration

var configuration = PHPickerConfiguration(photoLibrary: .shared())
configuration.filter = PHPickerFilter.images
configuration.preferredAssetRepresentationMode = .current
configuration.selection = .default
configuration.selectionLimit = 1
configuration.preselectedAssetIdentifiers = []

Here I get result from the Picker

let provider = pickerResult.itemProvider
guard provider.hasItemConformingToTypeIdentifier(UTType.image.identifier) else { throw Problem.noAssetForPickerResult }

provider.loadFileRepresentation(forTypeIdentifier: UTType.image.identifier, completionHandler: convertAndExtractEXIF)

Copying file to avoid any changes

// inside convertAndExtractEXIF
let originalUrl = Storage.documentsDirectoryUrl.appendingPathComponent("\(UUID().uuidString).heic", isDirectory: false)
try! FileManager.default.copyItem(at: url, to: originalUrl)

Configuring MTKView-based view

private func setup() {
    device = MTLCreateSystemDefaultDevice()
    clearColor = MTLClearColorMake(0.0, 0.0, 0.0, 0.0)
    preferredFramesPerSecond = 1
    framebufferOnly = false
    enableSetNeedsDisplay = true
    delegate = renderer

    if let metalLayer = layer as? CAMetalLayer {
        if #available(iOS 16, *) {
            metalLayer.wantsExtendedDynamicRangeContent = true
        }
        metalLayer.pixelFormat = .rgba16Float
        metalLayer.colorspace = CGColorSpace(name: CGColorSpace.extendedLinearDisplayP3)
    }
}

Drawing image from imageURL in the delegate:

func draw(in view: MTKView) {
    guard let imageUrl else { return }

    Task { @MainActor [imageUrl, view] in
        guard let commandQueue = view.device?.makeCommandQueue() else { return print("CommandQueue problem") }
        guard let commandBuffer = commandQueue.makeCommandBuffer() else { return print("CommandBuffer problem") }
        guard let drawable = view.currentDrawable else { return print("Drawable problem") }
        guard let originalImage = CIImage(data: try! Data(contentsOf: imageUrl)) else { return print("Original image problem") }

        let originalSize = originalImage.extent.size

        let drawableSize = view.drawableSize
        let scale = min(CGFloat(drawableSize.width / originalSize.width), CGFloat(drawableSize.height / originalSize.height))
        let ciImage = originalImage
            .transformed(by: CGAffineTransform(scaleX: scale, y: scale))

        let contextOptions: [CIContextOption: Any] = [
            .useSoftwareRenderer: false,
        ]
        let ciContext = CIContext(mtlCommandQueue: commandQueue, options: contextOptions)

        let destination = CIRenderDestination(
            width: Int(view.drawableSize.width),
            height: Int(view.drawableSize.height),
            pixelFormat: .bgra10_xr_srgb,
            commandBuffer: commandBuffer,
            mtlTextureProvider: { drawable.texture }
        )

        try! ciContext.startTask(toRender: ciImage, to: destination)
        commandBuffer.present(drawable)
        commandBuffer.commit()
    }
}

Tried all pixel formats. Can make photo "lighter" or "darker", but no HDR in any way. Even tried "hack from the internet" with AVPlayer, playing HDR video.

Next option is to convert image into video and try to play it instead :)

Accepted Answer

I've found it! You need to setup metalLayer.edrMetadata to something like this: metalLayer.edrMetadata = CAEDRMetadata.hdr10(minLuminance: 0, maxLuminance: 1000, opticalOutputScale: 1000)

I still have to figure out the parameters and how to use all that (I see problems after app switching for example), but it works!

Showing photo from PHPickerViewController in "HDR" mode
 
 
Q