Summation errors in average pooling of CoreML on M1 devices and Big Sur

Question

Created Jan ’21

Replies 2

Boosts 0

Participants 2

Hello, we have trouble with summation errors (differences) in the computation of an average pooling layer of a CNN by means of CoreML and on a M1 device with Big Sur. For several consecutive runs we get different results. This happens on GPU, CPU and with the Neural Engine (AI cores). So it's not only restricted to one computation method. The results are not stable.

In contrast to this, the results on an Apple Intel device running Big Sur are stable. There we get always the same constant result.

We build an easy CNN toy example, that allows for reproducing this effect on a M1 device and that we can provide.

Since we wanna provide our photo management software Excire Foto for M1 devices as fast as possible, we really appreciate to find a quick solution for this numerical problem. So please help us.

Best regards, your Excire Team

Boost

Answer 1

kerfuffle OP

Jan ’21

I don't know what the answer is, but I would try running the model in CPU-only or GPU mode to see if it happens there too. Perhaps it's an issue with the ANE.

In any case, these developer forums are pretty deserted. If this is an important issue for your business (and it sounds like it is), you should use one of your Tech Support Incidents or contact a developer evangelist from Apple.

0

Answer 2

Excire-Developer OP

Jan ’21

Thanks for your reply and this is what we already did. We conducted inference on GPU only, CPU only and on the M1 Tensor cores (Neural Engine) only, for all of these approaches we observed the same unstable results. We know, that floating point summation can cause some problems, especially if you conduct it in parallel.

And yes, we will try to contact an Apple developer directly, if there is a chance to reach one.

0