I submit the code which gives out the prediction about 5 hours ago (shown in 'ingestion output log') but it is still running now. Is the submission system still running?
Posted by: zhanyu @ July 18, 2020, 2:12 a.m.What is the the submission ID? From our end it seems like all of your submissions have finished.
Posted by: ydjiang @ July 18, 2020, 3:24 a.m.Thank you! It indeed finished now. The submission was much faster in my local environment so I thought something was wrong.
Anyway, is it normal that the scoring program (not including the ingestion part) would take 5-6 hours with the prediction given? It took less than one minute in my local environment (V100 16GB).
That does seems weird. Scoring should be pretty fast. I suspect it's some orchestration problems. Have you been experiencing the same issues again?
Posted by: ydjiang @ July 19, 2020, 7:26 p.m.Oh I haven't encountered this problem after the post. But one of my submissions 'Failed' (the most recent one) without any error output, and it also took much more time than what it took in my local environment (V100). I'm not sure what went wrong.
Posted by: zhanyu @ July 19, 2020, 8:57 p.m.I faced a similar problem; My submission #5 has been running for 30 hours...
I also face the same problem. I tested a pretty simple complexity function which takes less than 10 seconds in my local server (GTX1080) for scoring. But it takes approximately 5 hours in the submission system...
Posted by: knowzou @ July 20, 2020, 9:50 a.m.Hi everyone,
we were trying to figure out what is going to to these submissions but unfortunately we don't have a definitive answers yet.
If you guys are continuously running into the problem, please send us an email at pgdl.neurips@google.com with your submission id and we will manually terminate your submissions.
In the meantime, you can just create a new submission. The jobs though however is set to hard terminate after 24 hours so I am not sure why it would take 30 hours.
Sorry for the inconvenience!