Post

Replies

Boosts

Views

Activity

Reply to GPU cannot be assigned properly while NLP task.
Is there any update on this? Any ETA? I am seeing this error when training the tensorflowtts model on mac m1 chip. Metal device set to: Apple M1 Max ... systemMemory: 64.00 GB maxCacheSize: 21.33 GB Traceback (most recent call last): File "/Users/bemnet.merha/P4/TensorFlowTTS/./examples/tacotron2/train_tacotron2.py", line 528, in <module> main() File "/Users/bemnet.merha/P4/TensorFlowTTS/./examples/tacotron2/train_tacotron2.py", line 516, in main trainer.fit( File "/Users/bemnet.merha/P4/TensorFlowTTS/tensorflow_tts/trainers/base_trainer.py", line 1010, in fit self.run() File "/Users/bemnet.merha/P4/TensorFlowTTS/tensorflow_tts/trainers/base_trainer.py", line 104, in run self._train_epoch() File "/Users/bemnet.merha/P4/TensorFlowTTS/tensorflow_tts/trainers/base_trainer.py", line 126, in _train_epoch self._train_step(batch) File "/Users/bemnet.merha/P4/TensorFlowTTS/./examples/tacotron2/train_tacotron2.py", line 113, in _train_step self.one_step_forward(batch) File "/Users/bemnet.merha/miniforge3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "/Users/bemnet.merha/miniforge3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation gradients/tacotron2/decoder/while_grad/tacotron2/decoder/while/Placeholder_0/accumulator: Could not satisfy explicit device specification '/job:localhost/replica:0/task:0/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=1 requested_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' assigned_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' resource_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Merge: GPU CPU AddV2: GPU CPU
Jan ’22