When instantiating the CoreML model try to pass a MLModelConfiguration object. Play around with the config's computeUnits options. For example:
let config MLModelConfiguration()
config.computeUnits = .cpuOnly
let model = MLModel(contentsOf: localModelUrl, configuration: config)
In my case .cpuOnly worked best, but it's different for different models, try the other options as well.