Tensorflow-macos and tensorflow-metal still cause kernal panic

Question

raymondjiii OP

Created Mar ’22

Replies 2

Boosts 0

Views 2.7k

Participants 3

I'm running this on Intell MBPro

Metal device set to: AMD Radeon Pro 5600M 2022-03-29 11:46:55.397695: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.2 AVX AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-03-29 11:46:55.398290: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2022-03-29 11:46:55.398545: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: )

My model gets created and as soon as it starts the training on the first epoch the kernel dies.

Epoch 1/3000 2022-03-29 11:47:10.011230: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.

I would give almost anything to get this working. Tensorflow-metal has never worked for me other than saying I have a gpu.

Boost

Answer 1

dbl001 OP

Mar ’22

Are you using the 'Adam' optimizer?

0

Answer 2

raymondjiii OP

Mar ’22

I tried running my script outside of jupyter lab and the error is as follows:

2022-03-29 13:26:37.306914: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2022-03-29 13:26:37.307420: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: ) Model: "sequential"

Layer (type) Output Shape Param #

masking (Masking) (None, 1, 28) 0

layer1 (Bidirectional) (None, 1, 128) 47616

dropout (Dropout) (None, 1, 128) 0

layer2 (Bidirectional) (None, 1, 128) 98816

dropout_1 (Dropout) (None, 1, 128) 0

layer3 (Bidirectional) (None, 128) 98816

Output (Dense) (None, 1) 129

================================================================= Total params: 245,377 Trainable params: 245,377 Non-trainable params: 0

Epoch 1/3000 2022-03-29 13:26:54.863499: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-03-29 13:26:57.201 python[66880:3338950] -[MPSGraph adamUpdateWithLearningRateTensor:beta1Tensor:beta2Tensor:epsilonTensor:beta1PowerTensor:beta2PowerTensor:valuesTensor:momentumTensor:velocityTensor:gradientTensor:name:]: unrecognized selector sent to instance 0x600041707c60

zsh: segmentation fault python test.py

I was able to run another tensorflow based script on cmd line so there is something specific to this error that I am getting.

0