I got the same behavior on my macbook air with M1.
with tensorflow-metal (GPU)
Init Plugin
Init Graph Optimizer
Init Kernel
Num GPUs Available: 1
Metal device set to: Apple M1
2021-07-27 14:56:50.637472: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2021-07-27 14:56:50.639552: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2021-07-27 14:56:50.928184: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
Epoch 1/5
2021-07-27 14:56:51.027610: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
1875/1875 [==============================] - 16s 8ms/step - loss: 0.2980 - accuracy: 0.9137
Epoch 2/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.1446 - accuracy: 0.9563
Epoch 3/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.1079 - accuracy: 0.9676
Epoch 4/5
1875/1875 [==============================] - 15s 8ms/step - loss: 0.0894 - accuracy: 0.9722
Epoch 5/5
1875/1875 [==============================] - 16s 8ms/step - loss: 0.0746 - accuracy: 0.9767
2021-07-27 14:58:08.389344: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
313/313 - 3s - loss: 0.0763 - accuracy: 0.9766
with tensorflow-macos alone (CPU)
Num GPUs Available: 0
2021-07-27 15:40:12.555884: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
Epoch 1/5
1875/1875 [==============================] - 1s 481us/step - loss: 0.2988 - accuracy: 0.9139
Epoch 2/5
1875/1875 [==============================] - 1s 492us/step - loss: 0.1416 - accuracy: 0.9581
Epoch 3/5
1875/1875 [==============================] - 1s 484us/step - loss: 0.1072 - accuracy: 0.9674
Epoch 4/5
1875/1875 [==============================] - 1s 493us/step - loss: 0.0876 - accuracy: 0.9736
Epoch 5/5
1875/1875 [==============================] - 1s 492us/step - loss: 0.0734 - accuracy: 0.9779
313/313 - 0s - loss: 0.0721 - accuracy: 0.9782