I'm having the same issue. I asked for help on StackOverflow, but it looks like nobody knows, so I'm guessing is not a common issue.
https://stackoverflow.com/questions/73639510/after-converting-tensorflow-model-to-coreml-the-model-doesnt-predict-correct
If anybody can point me in the right direction, is good enough
Thank you