How to save and load MTLTexture, getting back same RGB values?

Rob,

If you are always rendering using MTLTextures, you don't need all of that.

It should be much more optimal to call getBytes and replaceRegion on the MTLTexture without ever involving Core Graphics or Core Image.

As for the color space issue, you didn't say what your pixelFormat is. On a Metal Kit View, the default is MTLPixelFormatBGRA8Unorm_sRGB

It sounds like the sRGB difference is what's going on here.

Ah, yes - I should add that I'm using a new iPad Pro. That is probably why the pixel format is `.bgra8unorm' (no _srgb) on the MTKView (the device has a different kind of display). If I have a texture that is .bgra8unorm_srgb, then it does save and reload with the same pixel values, as desired. But I'll need to run some other experiments to see if I can then convert or display that properly on this iPad that is using .bgra8unorm by default.

As for saving and reloading, I know it's not required for normal rendering, but my app requires it for other reasons. I'm trying to save the results of a render to image files and be able to reload them later. So I need to somehow get the MTLTexture's pixel data into a compressed, lossless format like PNG. I searched the API and the only way I saw to do this was through CIImage and CGImage.

It definitely sounds like somewhere in the process, an sRGB color space is being assumed. You may want to check the colorspace of the png that is saved, and this will help determine if the problem is in the saving or loading stage.

For saving pngs from a MTLTexture, I don't use Core Image. Instead, I call getBytes and pass them into CGDataProviderCreateWithData.

For saving images for your app's own use, and not for exporting an image for the user, I would recommend saving the image data using the libcompression library:

https://developer.apple.com/documentation/compression/data_compression?language=objc

Split the image data into chunks and compress them yourself. This should give faster save/load times.

Btw, as a forewarning, when working with low level metal and image programming, it will be significantly faster and a lower memory footprint using C with Objective-C only where we are forced to use it. Swift is ironically slow, and as can be seen on the swift evolution mailing list, they intend to continue making the swift implementation slower with every update for the foreseeable future. For this type of app, you'll want the most optimized code possible. The syntax for working with data is much uglier in Swift anyway. I originally wrote my whole app in Swift and was unsatisfied with the performance and language syntax, after rewriting in C, everything was much better.

I can partly fix this if I call `makeTextureView()` and get a MTLTexture that views the same data but is marked as having format `.bgra8unorm_srgb`. Then when I turn into a CGImage for output it doesn't adjust the opaque pixels. It still multiplies the RGB channels by the alpha, which I don't want, but that's not a Metal issue.

Try the Core Graphics approach with CGImageCreate() and CGDataProviderCreateWithData(), by passing in the pixel data from MTLTexture's getBytes

You can specify whether or not you want to premultiply the alpha with the bitmapInfo parameter of CGImageCreate().

Are you sure about the new memory? The docs for makeTextureView say "return a new texture object that shares the same storage allocation as the source texture object. Because they share the same storage, any changes to the pixels of the new texture are reflected in the source texture, and vice versa."

Most of my saves are internal to the app, so I took your other advice on using libcompression, at least for now. But there is another case where I need to export a PNG so I still need to sort that out. I will try CGImageCreate and it's various flags.

Rob

Yes, your right, it does reuse the same memory space... which is an interesting approach on it's own. I'd be curious to know how it actually goes about remaping the byte array behind the scenes. For example, if you called replaceBytes on a remapped texture, it's not possible for it to do a memcpy() all at once, and instead it would have to loop through each pixel and replace them using the remapped byte offsets.

Topic is old, but just in case if anyone encounter the same issue in future. This line create CIImage, without options. By default it chooses wrong color space here.

let ciImage = CIImage(mtlTexture: metalTexture, options: [:])

Do it like this

let colorSpace = CGColorSpaceCreateDeviceRGB()
let options: [CIImageOption : Any] = [
            .colorSpace: colorSpace
]
let ciImage = CIImage(mtlTexture: metalTexture, options: options)

It fixed issue with too light images for me. Just in case, I'm not using these images for actual switch between image and texture in production, this is for debugging purpose, because metal debuggin not always provide proper texture or working at all on older devices/sims