Re: Support for image outputs

To use the helper you provided, "convert_multiarray_output_to_image", does the model output dataType need to be DOUBLE or INT32, or both?

Hey Michael, can I use this script to conver my MLMultiArray input to be an Image instead? I'm guessing that it needs to consider

spec.description.input

in this case.

Thanks for those details, it really helped! One question though:

How would you apply scale and biases to the image data before conversion? For instance, my network outputs in the range [-1, 1] and I need to convert that to [0, 255].

When using the Keras converter, the input image can thankfully be scaled and biased. Can that code be reused somehow?

I'm trying to convert outputs of mlmodel to UIImage, but it's not working...

outputs of mlmodel : Image(Grayscale width x height)

guard let results = request.results as? [VNPixelBufferObservation]  else{  fatalError("Fatal error")}
print(String(describing: type(of: results))
print(String(describing: type(of: results[0])))
let ciImage = CIImage(cvPixelBuffer: results[0].pixelBuffer)

Outputs:

Array<VNPixelBufferObservation>

VNPixelBufferObservation

Error occurs on line 04 :

Thread 1: EXC_BAD_ACCESS (code=1, address=0xe136dbec8)

-------------------------------------------------------------------------------------------

I'm trying to keep mlmodel MultiArray.

outputs of mlmodel : MultiArray (Double 1 x width x height)

guard let results = request.results as? [VNCoreMLFeatureValueObservation]  else{  fatalError("Fatal error")}

print(String(describing: type(of: results))
print(String(describing: type(of: results[0]))
print(String(describing: type(of: results[0].featureValue)))
print(results[0].featureValue)
print(results[0].featureValue.multiArrayValue)

let imageMultiArray = results.[0].featureValue.multiArrayValue
let imageData = UnsafeMutablePointer<Double>(OpaquePointer(imageMultiArray?.dataPointer))
let channelStride = imageMultiArray?.strides[0].intValue;
let yStride = imageMultiArray?.strides[1].intValue;
let xStride = imageMultiArray?.strides[2].intValue;

func pixelOffset(_ channel: Int, _ y: Int, _ x: Int) -> Int {
  return channel*channelStride! + y*yStride! + x*xStride!
}

let topLeftGreenPixel = Unit8(imageData![pixelOffset(1,0,0)])

Outputs:

Array<VNCoreMLFeatureValueObservation>

VNCoreMLFeatureValueObservation

Optional<MLFeatureValue>

Optional(MultiArray : Double 1 x width x height array)

Optional(Double 1 x width x height array)

Error occurs on line 19 :Use of unresolved identifier 'Unit8

shoud be replaced by Uint8 ? and how to convert to UIImage?

Thank you for your any help in advance!

Unlike others who have used this(forcing the model to output an image) method and gotten back CVPixelBuffers with alpha 0, I am getting back an alpha of 255 and r,g,b in [0,1]. Ignoring the alpha channel, PIL displays an image that is definitely related to the desired output. Could it have something to do with the image scale/rgb biases when i convert the model from keras? I am not really sure where to go with this, we're trying to convert the CVPixelBuffer output into a MetalTexture and want to avoid extra post-processing steps.

For clarity: I am converting from keras to coreML, and then calling "convert_multiarray_output_to_image(...)" on the coreML model. I have an image scale of 1/127.5 and rgb biases of -1 at the keras conversion step.