Graph execution error

2023-01-05 12:04:45.782992: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x168ba5420 2023-01-05 12:04:45.910000: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x168ba5420 2023-01-05 12:04:45.910028: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x168ba5420 2023-01-05 12:04:45.910038: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x168ba5420 2023-01-05 12:04:45.910049: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x168ba5420 Traceback (most recent call last): File "/Users/mac.user/Downloads/test.py", line 13, in model.fit(x_train, y_train, epochs=5, batch_size=64) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler raise e.with_traceback(filtered_tb) from None File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.NotFoundError: Graph execution error:

Detected at node 'StatefulPartitionedCall_212' defined at (most recent call last): File "/Users/mac.user/Downloads/test.py", line 13, in model.fit(x_train, y_train, epochs=5, batch_size=64) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/engine/training.py", line 1650, in fit tmp_logs = self.train_function(iterator) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/engine/training.py", line 1249, in train_function return step_function(self, iterator) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/engine/training.py", line 1233, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/engine/training.py", line 1222, in run_step outputs = model.train_step(data) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/engine/training.py", line 1027, in train_step self.optimizer.minimize(loss, self.trainable_variables, tape=tape) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 527, in minimize self.apply_gradients(grads_and_vars) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1140, in apply_gradients return super().apply_gradients(grads_and_vars, name=name) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 634, in apply_gradients iteration = self._internal_apply_gradients(grads_and_vars) File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1166, in _internal_apply_gradients return tf.internal.distribute.interim.maybe_merge_call( File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1216, in _distributed_apply_gradients_fn distribution.extended.update( File "/Users/mac.user/Library/Python/3.9/lib/python/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1211, in apply_grad_to_update_var return self._update_step_xla(grad, var, id(self._var_key(var))) Node: 'StatefulPartitionedCall_212' could not find registered platform with id: 0x168ba5420 [[{{node StatefulPartitionedCall_212}}]] [Op:__inference_train_function_23355]

Hi @ptmp93,

Is this on the latest wheels with tensorflow-macos==2.11 and tensorflow-metal==0.7.0? In that case has to do with recent changes on tensorflow side for version 2.11 where a new optimizer API has been implemented where a default JIT compilation flag is set (https://blog.tensorflow.org/2022/11/whats-new-in-tensorflow-211.html). This is forcing the optimizer op to take an XLA path that the pluggable architecture has not implemented yet causing the inelegant crash as it cannot fall back to supported operations. Currently the workaround is to use the older API for optimizers that was used up to TF 2.10 by exporting it from the .legacy folder of optimizers. So more concretely by using Adam optimizer as an example one should change:

from tensorflow.keras.optimizers import Adam

to

from tensorflow.keras.optimizers.legacy import Adam

For those following the official setup here: https://developer.apple.com/metal/tensorflow-plugin/

And then trying to run the test code, you will need to make an instance of the legacy optimizer:

from tensorflow.keras.optimizers.legacy import Adam
adam = Adam()

And then use that created instance instead of the string "Adam":

model.compile(optimizer=adam, loss=loss_fn, metrics=["accuracy"])

Graph execution error
 
 
Q