A similar problem here with tensorflow.
Training of my CNN model (NVIDIA PilotNet) works fine on a standard python runtime environment but when using a virtual runtime with tensorflow-metal, it fails to converge (loss diverges).
[hardware] Macbook Air M1 (2020)
[os] 13.5.2 (22G91)
[runtime] virtual env python 3.11.5 + tensorflow-metal (as described in https://developer.apple.com/metal/tensorflow-plugin/)
Any solution?