I am trying to profile a tensorflow 2.5 model with tensorflow-macos and tensorflow-metal.
I am getting this error:
AttributeError: module 'tensorflow.compat.v1.profiler' has no attribute 'experimental'
Here's a code snippet:
import tensorflow as tf
import numpy as np
from utils import *
tf.compat.v1.enable_v2_behavior()
from tensorflow.python.framework.ops import disable_eager_execution
disable_eager_execution()
options = tf.profiler.experimental.ProfilerOptions(host_tracer_level = 3,
python_tracer_level = 1,
device_tracer_level = 1)
tf.profiler.experimental.start('~/logdir', options=options)
...
tf.profiler.experimental.stop()
% pip list
Package Version
-------------------------- -------------------
absl-py 0.12.0
anyio 3.2.1
appnope 0.1.2
argon2-cffi 20.1.0
astunparse 1.6.3
async-generator 1.10
attrs 21.2.0
Babel 2.9.1
backcall 0.2.0
bleach 3.3.1
cachetools 4.2.2
certifi 2021.5.30
cffi 1.14.6
charset-normalizer 2.0.1
cloudpickle 1.6.0
cycler 0.10.0
Cython 0.29.24
debugpy 1.3.0
decorator 5.0.9
defusedxml 0.7.1
dill 0.3.4
dm-tree 0.1.6
dotmap 1.3.23
entrypoints 0.3
flatbuffers 1.12
future 0.18.2
gast 0.4.0
gensim 4.0.1
google-auth 1.32.1
google-auth-oauthlib 0.4.4
google-pasta 0.2.0
googleapis-common-protos 1.53.0
grpcio 1.34.1
gviz-api 1.9.0
gym 0.18.3
h5py 3.1.0
idna 3.2
importlib-resources 5.2.0
ipykernel 6.0.1
ipython 7.25.0
ipython-genutils 0.2.0
ipywidgets 7.6.3
jedi 0.18.0
Jinja2 3.0.1
json5 0.9.6
jsonschema 3.2.0
jupyter-client 6.1.12
jupyter-core 4.7.1
jupyter-server 1.9.0
jupyterlab 3.0.16
jupyterlab-pygments 0.1.2
jupyterlab-server 2.6.1
jupyterlab-widgets 1.0.0
keras-nightly 2.5.0.dev2021032900
Keras-Preprocessing 1.1.2
kiwisolver 1.3.1
Markdown 3.3.4
MarkupSafe 2.0.1
matplotlib 3.4.2
matplotlib-inline 0.1.2
memory-profiler 0.58.0
mistune 0.8.4
nbclassic 0.3.1
nbclient 0.5.3
nbconvert 6.1.0
nbformat 5.1.3
nest-asyncio 1.5.1
notebook 6.4.0
numpy 1.19.5
oauthlib 3.1.1
opt-einsum 3.3.0
packaging 21.0
pandas 1.3.0
pandocfilters 1.4.3
parso 0.8.2
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.2.0
pip 21.2.1
prometheus-client 0.11.0
promise 2.3
prompt-toolkit 3.0.19
protobuf 3.17.3
psutil 5.8.0
ptyprocess 0.7.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pybind11 2.6.2
pycparser 2.20
pyglet 1.5.15
Pygments 2.9.0
pyparsing 2.4.7
pyrsistent 0.18.0
python-dateutil 2.8.2
pytz 2021.1
pyzmq 22.1.0
requests 2.26.0
requests-oauthlib 1.3.0
requests-unixsocket 0.2.0
rsa 4.7.2
scipy 1.7.0
Send2Trash 1.7.1
setuptools 41.2.0
six 1.15.0
smart-open 5.1.0
sniffio 1.2.0
tensorboard 2.5.0
tensorboard-data-server 0.6.1
tensorboard-plugin-profile 2.4.0
tensorboard-plugin-wit 1.8.0
tensorflow-datasets 4.3.0
tensorflow-estimator 2.5.0
tensorflow-hub 0.12.0
tensorflow-macos 2.5.0
tensorflow-metadata 1.1.0
tensorflow-metal 0.1.1
tensorflow-probability 0.13.0
termcolor 1.1.0
terminado 0.10.1
testpath 0.5.0
tornado 6.1
tqdm 4.61.2
traitlets 5.0.5
typing-extensions 3.7.4.3
urllib3 1.26.6
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.1.0
Werkzeug 2.0.1
wheel 0.36.2
widgetsnbextension 3.5.1
wrapt 1.12.1
zipp 3.5.0
Post
Replies
Boosts
Views
Activity
Details here:
https://stackoverflow.com/questions/68551935/why-does-my-tensorflow-model-stop-training
I can run the VariationalDeepSemantic Hashing model in an Anaconda python 3.85 virtual environment with tensorflow 2.5 on CPUs.
If I run the same code in the tensorflow-metal virtual environment with python 3.82, accessing my AMD Radeon Pro 5700 XT GPU, the process stops training at epoch 5 on the 5200 batch.
I am running tensorflow-macos and tensorflow-metal on Big Sur.
I am getting this error:
AttributeError: module 'tensorflow.keras' has no attribute 'utils_dataset_from_directory'
https://github.com/keras-team/keras-io/issues/12
Can I install tf_nightly? Or does it conflict with tensorflow-macos?
from tensorflow import keras
from tensorflow.python.framework.ops import disable_eager_execution
dataset = keras.utils_dataset_from_directory(
"celeba_gan",
label_mode=None,
image_size=(64, 64),
batch_size=32,
smart_resize=True)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/var/folders/3n/56fpv14n4wj0c1l1sb106pzw0000gn/T/ipykernel_41859/2519466253.py in <module>
1 from tensorflow import keras
2 from tensorflow.python.framework.ops import disable_eager_execution
----> 3 dataset = keras.utils_dataset_from_directory(
4 "celeba_gan",
5 label_mode=None,
AttributeError: module 'tensorflow.keras' has no attribute 'utils_dataset_from_directory'
I have installed tensorflow-macos and tensorflow-metal on Big Sur on a iMac 27" with AMD Radeon Pro 5700 XT.
I am trying to run Keras code from Francios Challet's Deep Learning example:
E.g Chapter 11-part04_sequence-to-Sequence
https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/chapter11_part04_sequence-to-sequence-learning.ipynb
seq2seq_rnn.compile(
optimizer="rmsprop",
loss="sparse_categorical_crossentropy",
metrics=["accuracy"])
seq2seq_rnn.fit(train_ds, epochs=15, validation_data=val_ds)
2021-07-15 13:17:00.117869: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-07-15 13:17:01.403133: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at partitioned_function_ops.cc:114 : Invalid argument: No OpKernel was registered to support Op 'CudnnRNNV3' used by {{node cond_41/then/_0/cond/CudnnRNNV3}} with these attrs: [T=DT_FLOAT, input_mode="linear_input", direction="unidirectional", rnn_mode="gru", seed2=0, is_training=true, num_proj=0, time_major=false, seed=0, dropout=0]
Registered devices: [CPU, GPU]
Registered kernels:
<no registered kernels>
[[cond_41/then/_0/cond/CudnnRNNV3]]
2021-07-15 13:17:01.419061: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at partitioned_function_ops.cc:114 : Invalid argument: No OpKernel was registered to support Op 'CudnnRNNV3' used by {{node cond_41/then/_0/cond/CudnnRNNV3}} with these attrs: [time_major=false, dropout=0, seed=0, T=DT_FLOAT, input_mode="linear_input", direction="unidirectional", rnn_mode="gru", seed2=0, is_training=true, num_proj=0]
Registered devices: [CPU, GPU]
Registered kernels:
<no registered kernels>
[[cond_41/then/_0/cond/CudnnRNNV3]]
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
/var/folders/3n/56fpv14n4wj0c1l1sb106pzw0000gn/T/ipykernel_94493/3093225856.py in <module>
3 loss="sparse_categorical_crossentropy",
4 metrics=["accuracy"])
----> 5 seq2seq_rnn.fit(train_ds, epochs=15, validation_data=val_ds)
~/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
1181 _r=1):
1182 callbacks.on_train_batch_begin(step)
-> 1183 tmp_logs = self.train_function(iterator)
1184 if data_handler.should_sync:
1185 context.async_wait()
~/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
887
888 with OptionalXlaContext(self._jit_compile):
--> 889 result = self._call(*args, **kwds)
890
891 new_tracing_count = self.experimental_get_tracing_count()
~/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
948 # Lifting succeeded, so variables are initialized and we can run the
949 # stateless function.
--> 950 return self._stateless_fn(*args, **kwds)
951 else:
952 _, _, _, filtered_flat_args = \
~/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
3021 (graph_function,
3022 filtered_flat_args) = self._maybe_define_function(args, kwargs)
-> 3023 return graph_function._call_flat(
3024 filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access
3025
~/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1958 and executing_eagerly):
1959 # No tape is watching; skip to running the function.
-> 1960 return self._build_call_outputs(self._inference_function.call(
1961 ctx, args, cancellation_manager=cancellation_manager))
1962 forward_backward = self._select_forward_and_backward_functions(
~/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py in call(self, ctx, args, cancellation_manager)
589 with _InterpolateFunctionError(self):
590 if cancellation_manager is None:
--> 591 outputs = execute.execute(
592 str(self.signature.name),
593 num_outputs=self._num_outputs,
~/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
57 try:
58 ctx.ensure_initialized()
---> 59 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
60 inputs, attrs, num_outputs)
61 except core._NotOkStatusException as e:
InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: No OpKernel was registered to support Op 'CudnnRNNV3' used by {{node cond_41/then/_0/cond/CudnnRNNV3}} with these attrs: [T=DT_FLOAT, input_mode="linear_input", direction="unidirectional", rnn_mode="gru", seed2=0, is_training=true, num_proj=0, time_major=false, seed=0, dropout=0]
Registered devices: [CPU, GPU]
Registered kernels:
<no registered kernels>
[[cond_41/then/_0/cond/CudnnRNNV3]]
[[model/bidirectional/backward_gru/PartitionedCall]]
[[broadcast_weights_1/assert_broadcastable/is_valid_shape/else/_1/broadcast_weights_1/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/then/_53/broadcast_weights_1/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/concat/_66]]
(1) Invalid argument: No OpKernel was registered to support Op 'CudnnRNNV3' used by {{node cond_41/then/_0/cond/CudnnRNNV3}} with these attrs: [T=DT_FLOAT, input_mode="linear_input", direction="unidirectional", rnn_mode="gru", seed2=0, is_training=true, num_proj=0, time_major=false, seed=0, dropout=0]
Registered devices: [CPU, GPU]
Registered kernels:
<no registered kernels>
[[cond_41/then/_0/cond/CudnnRNNV3]]
[[model/bidirectional/backward_gru/PartitionedCall]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_520769]
Function call stack:
train_function -> train_function
I installed tensorflow-mac and tensorflow-metal on an iMac 2021 27" running Big Sur with an AMD Radeon Pro 5700 XT.
I am running Python 3.8.5
(tensorflow-metal) (base) davidlaxer@x86_64-apple-darwin13 ~ % pip list
Package Version Location
----------------------- ------------------- -------------------------
...
tensorboard 2.5.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.0
tensorflow 2.5.0
tensorflow-estimator 2.5.0
tensorflow-hub 0.12.0
tensorflow-macos 2.5.0
tensorflow-metal 0.1.1
tensorflow-text 2.5.0
When I try to import tensorflow I get this error:
% ipython
Python 3.8.5 (default, Sep 4 2020, 02:22:02)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.24.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import tensorflow
---------------------------------------------------------------------------
NotFoundError Traceback (most recent call last)
<ipython-input-1-d6579f534729> in <module>
----> 1 import tensorflow
~/tensorflow-metal/lib/python3.8/site-packages/tensorflow/__init__.py in <module>
447 _plugin_dir = _os.path.join(_s, 'tensorflow-plugins')
448 if _os.path.exists(_plugin_dir):
--> 449 _ll.load_library(_plugin_dir)
450 # Load Pluggable Device Library
451 _ll.load_pluggable_device_library(_plugin_dir)
~/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py in load_library(library_location)
152
153 for lib in kernel_libraries:
--> 154 py_tf.TF_LoadLibrary(lib)
155
156 else:
NotFoundError: dlopen(/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow-plugins/libmetal_plugin.dylib, 6): Symbol not found: _TF_AssignUpdateVariable
Referenced from: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow-plugins/libmetal_plugin.dylib
Expected in: flat namespace
I am trying to get the AMD Radeon Pro 5700 XT GPU on my iMac 27" 2021 running Big Sur 11.4 to work with tensorflow-macos.
If I disable eager execution I get an exception, if I don't, tensorflow-macos choses the CPU and not the GPU.
Here's a simple example which shows the exception:
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text
import numpy as np
from sklearn.preprocessing import normalize
from tensorflow.python.framework.ops import disable_eager_execution
disable_eager_execution()
m4 = hub.load("/Users/davidlaxer/Downloads/universal-sentence-encoder_4")
english_sentences = ["dog", "Puppies are nice.", "I enjoy taking long walks along the beach with my dog."]
r4 = np.array(m4(english_sentences))
print(r4)
print(m4)
type(r4)
type(m4)
---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
<ipython-input-4-8b0ba0e4c28c> in <module>
1 m4 = hub.load("/Users/davidlaxer/Downloads/universal-sentence-encoder_4")
2 english_sentences = ["dog", "Puppies are nice.", "I enjoy taking long walks along the beach with my dog."]
----> 3 r4 = np.array(m4(english_sentences))
4 print(r4)
5 print(m4)
~/anaconda3/envs/tensorflow_mac/lib/python3.8/site-packages/tensorflow/python/framework/ops.py in __array__(self)
850
851 def __array__(self):
--> 852 raise NotImplementedError(
853 "Cannot convert a symbolic Tensor ({}) to a numpy array."
854 " This error may indicate that you're trying to pass a Tensor to"
NotImplementedError: Cannot convert a symbolic Tensor (StatefulPartitionedCall_1:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
And commenting out disable_eager_execution():
from tensorflow.python.framework.ops import disable_eager_execution
#disable_eager_execution()
m4 = hub.load("/Users/davidlaxer/Downloads/universal-sentence-encoder_4")
english_sentences = ["dog", "Puppies are nice.", "I enjoy taking long walks along the beach with my dog."]
r4 = np.array(m4(english_sentences))
print(r4)
print(m4)
type(r4)
type(m4)
[-0.06334164 -0.01812314 0.03680531 ... -0.02809388 0.02786911
-0.04715428]
[ 0.01975714 -0.02284616 0.04316505 ... -0.01376714 -0.00614742
-0.00124967]
[-0.02169351 -0.003993 0.06716524 ... 0.05952153 0.02262796
0.03501643]]
<tensorflow.python.saved_model.load.Loader._recreate_base_user_object.<locals>._UserObject object at 0x7fbb3892a9a0>
[5]:
tensorflow.python.saved_model.load.Loader._recreate_base_user_object.<locals>._UserObject
The tensorflow group does not support this GPU and the tensorflow-mac repository is now read-only.
https://github.com/tensorflow/tensorflow/issues/50353
https://github.com/apple/tensorflow_macos
Any ideas?