Predicting Generalization in Deep Learning Forum

Go back to competition Back to thread list Post in this thread

> Request to restart submissions

Hello my submissions with the following timestamps have Failed without throwing an error, they terminated after a Warning text. Other submissions also threw this warning but did not Fail, similarly on my local system, the ingestion continues to finish despite the warnings, can you please look into this?
08/16/2020 10:02:39
08/17/2020 07:09:54

Thanks

Posted by: shmakn @ Aug. 17, 2020, 7:35 a.m.

Apologies! I have deleted both comments.
It is quite late now for me. I will restart your submission and try to look into the code tomorrow.

Posted by: ydjiang @ Aug. 17, 2020, 8:31 a.m.

Hi!

Are you sure it is running locally? I don't see any tf.function in your code, so tensorflow can't do any optimizations. Also, it is better to loop outside of the gradient tape if possible. For instance:

def bad_practice(model, batches):
...
with tf.GradientTape(persistent=True) as tape:
for b in batches:
...
g = tape.gradient(something, model.trainable_weights)
...

Should be replaced by:

def good_practice(model, batches):
@tf.function
def get_gradient(single_batch):
with tf.GradientTape() as tape:
...
return tape.gradient(something, model.trainable_weights)
all_gradients = [get_gradient(b) for b in batches]
...

Let me know it that helped!

Posted by: pforet @ Aug. 17, 2020, 11:04 p.m.

The submission that I restarted for you yesterday is still running. We also tried to run it on a Colab notebook and it actually crashed the notebook.
It might be the case that there are some efficiency problems. Please refer to Pierre's solution above and let us know if the problem persists.

Posted by: ydjiang @ Aug. 18, 2020, 5:31 a.m.

Hi, thanks for taking the time out to optimize the code and test it., really appreciate.

Posted by: shmakn @ Aug. 18, 2020, 6:38 a.m.
Post in this thread