Better performances with CoreML on CPU than on GPU

Hello, I've build a CoreML model with Keras and converted it with coreml_tools


But when I run it on a device, prediction is faster when "usesCPUOnly" is true

It seems to be slower with GPU enabled.

Tested on iPhone SE and iPad Pro 13'

What do you think of that 😉 ?

Replies

What sort of model is it?

A simple one :


img_width = 48

img_height = 129

input = ( img_height, img_width, 3 )


model = Sequential()

model.add(Conv2D(32, (5, 5), input_shape=input, activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Dropout(0.5))

model.add(Flatten())

model.add(Dense(num_classes, activation='softmax'))


But any model I build is faster on CPU

  • Hi, How many is the cpu occupation of model inferencing?

Add a Comment

There is a certain overhead to using the GPU. For small models the CPU will be faster because: a) it's a fast CPU, and b) you're not doing a lot of things in parallel that make it worth running this on the GPU.


The devices you tried it in should be able to use the GPU with Core ML, and Core ML (in theory) should pick the fastest option.

Ok,


maybe if my model was more complex, GPU would be more usefull.


But, performances can be 5x faster on CPU which is strange.


And CoreML definitely not choose the fastest option ; I always have to set usesCPUOnly true or it is slower


😟

The GPU is great for accelerating tasks which can be parallelized like 3D rendering or training of a neural network. Inference often can’t be parallized in the same way so it runs faster on the CPU.

I thought that it was not possible to train with CoreML