I have the same problem with my M2 chip.
I also use the cleaning function of tensor flow and explicit garbage collection from python, but my memory usage grows completely uncontrolled and after 2000 iterations it crashes. This is super frustrating, my training interrupts and there is no way to make ti work, even if i try to optimize the code it does not work. I think apple should fix these compatibility issue with tensor flow :(