I have the same problem. You can try to replace Adam optimizer to another one. This problem only cause on this optimizer, avoid to use it and the problems will be solved temporarily. I tested on tensorflow-macos 2.5.0 and 2.6.0 and hope it will be fix soon.
noahc1510
Users receive reputation points by contributing quality content to the forums.
Posts include post and replies published on the forums.
Post authors can mark replies as Accepted to indicate successful solutions.
Apple Developer Forums admins can mark replies as Apple Recommended to indicate an approved solution