Hello my submissions with the following timestamps have Failed without throwing an error, they terminated after a Warning text. Other submissions also threw this warning but did not Fail, similarly on my local system, the ingestion continues to finish despite the warnings, can you please look into this?
08/16/2020 10:02:39
08/17/2020 07:09:54
Thanks
Posted by: shmakn @ Aug. 17, 2020, 7:35 a.m.Apologies! I have deleted both comments.
It is quite late now for me. I will restart your submission and try to look into the code tomorrow.
Hi!
Are you sure it is running locally? I don't see any tf.function in your code, so tensorflow can't do any optimizations. Also, it is better to loop outside of the gradient tape if possible. For instance:
def bad_practice(model, batches):
...
with tf.GradientTape(persistent=True) as tape:
for b in batches:
...
g = tape.gradient(something, model.trainable_weights)
...
Should be replaced by:
def good_practice(model, batches):
@tf.function
def get_gradient(single_batch):
with tf.GradientTape() as tape:
...
return tape.gradient(something, model.trainable_weights)
all_gradients = [get_gradient(b) for b in batches]
...
Let me know it that helped!
Posted by: pforet @ Aug. 17, 2020, 11:04 p.m.The submission that I restarted for you yesterday is still running. We also tried to run it on a Colab notebook and it actually crashed the notebook.
It might be the case that there are some efficiency problems. Please refer to Pierre's solution above and let us know if the problem persists.
Hi, thanks for taking the time out to optimize the code and test it., really appreciate.