Hi, this error below was happening after I submit my result and lead to failed submission status, how do I fix it? Thank you
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation default_policy/fc_1/kernel/IsInitialized/VarIsInitializedOp: Could not satisfy explicit device specification '' because the node node default_policy/fc_1/kernel/IsInitialized/VarIsInitializedOp (defined at tmp/codalab/tmpYElp7V/run/input/res/policy.py:466) placed on device No device assignments were active during op 'default_policy/fc_1/kernel/IsInitialized/VarIsInitializedOp' creation. was colocated with a group of nodes that required incompatible device '/device:GPU:0'
Posted by: team_19 @ Nov. 15, 2019, 12:04 p.m.Hi, it looks like your model is being frozen with device names attached to TensorFlow ops. This is not letting TensorFlow re-adjust the computational graph when loading the model on a system that has different devices. (ie. the evaluation environment does not have a GPU available to it)
To get around this, you can modify the way in which you load the frozen model to discard any device annotations that may have been added when the model was frozen.
To do this, simply add the `clear_devices=True` flag when you load your model in your policy file.
```
self.base_model = tf.saved_model.load(self._sess,
export_dir=model_path,
tags=['serve'],
clear_devices=True) # remove any device tags that may have been added to TF Ops
```
Also, make sure to run the `run.py` script that ships along with the starter kit as a sanity check that your model is runnable outside of your training script
Posted by: HuaweiUK @ Nov. 19, 2019, 7:33 p.m.