Post

Replies

Boosts

Views

Activity

InvalidArgumentError: Cannot assign a device for operation agent/VerifyFinite/CheckNumerics
Hardware: MacBook Pro (13-inch, M1, 2020) macOS Monterey 12.1 Libs: tensorflow-macos 2.7.0 pypi_0 pypi tensorflow-metal 0.3.0 pypi_0 pypi tensorforce 0.6.5 dev_0 Validate tensorflow: import tensorflow as tf print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU'))) ... % python3 tensorflow_demo.py Num GPUs Available: 1 Install tensorforce from source: git clone https://github.com/tensorforce/tensorforce.git cd tensorforce # edit requirements.txt # -tensorflow == 2.7.0 # +tensorflow-macos == 2.7.0 pip3 install -e . Copy the Quickstart Example code to tensorforce_demo.py $ python3 tensorforce_demo.py Metal device set to: Apple M1 systemMemory: 16.00 GB maxCacheSize: 5.33 GB WARNING:root:Infinite min_value bound for state. Traceback (most recent call last): File "/Users/derek/Sites/ml/lottoenv/tensorforce_demo.py", line 31, in <module> actions = agent.act(states=states) File "/Users/derek/Sites/ml/tensorforce/tensorforce/agents/agent.py", line 415, in act return super().act( File "/Users/derek/Sites/ml/tensorforce/tensorforce/agents/recorder.py", line 262, in act actions, internals = self.fn_act( File "/Users/derek/Sites/ml/tensorforce/tensorforce/agents/agent.py", line 462, in fn_act actions, timesteps = self.model.act( File "/Users/derek/Sites/ml/tensorforce/tensorforce/core/module.py", line 136, in decorated output_args = function_graphs[str(graph_params)](*graph_args) File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation agent/VerifyFinite/CheckNumerics: Could not satisfy explicit device specification '' because the node {{colocation_node agent/VerifyFinite/CheckNumerics}} was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index_=1 requested_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' assigned_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' resource_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[] Identity: GPU CPU Switch: GPU CPU CheckNumerics: CPU _Arg: GPU CPU Colocation members, user-requested devices, and framework assigned devices, if any: args_0 (_Arg) framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0 agent/VerifyFinite/CheckNumerics (CheckNumerics) agent/VerifyFinite/control_dependency (Identity) agent/assert_greater_equal/Assert/AssertGuard/args_0/_16 (Switch) agent/assert_less_equal/Assert/AssertGuard/args_0/_26 (Switch) Func/agent/StatefulPartitionedCall/input/_80 (Identity) /job:localhost/replica:0/task:0/device:GPU:0 Func/agent/assert_greater_equal/Assert/AssertGuard/then/_10/input/_150 (Identity) Func/agent/assert_greater_equal/Assert/AssertGuard/else/_11/input/_156 (Identity) Func/agent/assert_less_equal/Assert/AssertGuard/then/_20/input/_162 (Identity) Func/agent/assert_less_equal/Assert/AssertGuard/else/_21/input/_168 (Identity) Func/agent/StatefulPartitionedCall/state_preprocessing/PartitionedCall/input/_257 (Identity) /job:localhost/replica:0/task:0/device:GPU:0 Func/agent/StatefulPartitionedCall/state_preprocessing/PartitionedCall/linear_normalization0/PartitionedCall/input/_350 (Identity) /job:localhost/replica:0/task:0/device:GPU:0 [[{{node agent/VerifyFinite/CheckNumerics}}]] [Op:__inference_act_1212] Run the code on CPU as follows: import tensorflow as tf with tf.device("/cpu:0"): #... indent the demo code above Returns no error % python3 tensorforce_demo.py Metal device set to: Apple M1 systemMemory: 16.00 GB maxCacheSize: 5.33 GB WARNING:root:Infinite min_value bound for state.
3
0
1.5k
Feb ’22