I have installed it successfully using the instructions from here.
https://github.com/jeffheaton/t81_558_deep_learning/blob/master/install/tensorflow-install-mac-metal-jul-2021.ipynb
It uses miniforge which is lightweight compared to the full Anaconda. I too do not like to have multiple package managers floating around but nothing else was working and it is
I did eventually get everything to compile correctly but my TensorFlow would never run. It always ended with an error "INTERNAL: platform is already registered with name: "METAL". I could not get this to work with my homebred'ed Python3.9.10 but it works flawlessly with Python3.9.10 from miniforge.