My submission ran for >1.5hr and was killed in the middle of the 135th episode out of the 140 in the evaluation process. Is there a hard limit on execution time? Could you possibly relax that limit?
Posted by: Team34 @ Dec. 11, 2019, 8:33 a.m.Hi, Yes there is a 1 hour limit on the evaluation run-time, if you were seeing longer delays, it's probably due to queuing times while CodaLab is waiting for workers to be freed up.
1 hour should be plenty of time to complete the 7 test cases. There are two main reasons where you might see a slow down 1. your evaluation worker was scheduled on a machine with noisy neighbors, 2. your code might need some performance tune ups. Unfortunately we don't have much control over (1), as a test, if you re-submit successfully it suggests this might be the case. On the other hand, (2) is the more likely situation.
For diagnosing slow parts of your policy, look into a tool like this one here: https://github.com/asokoloski/python-flamegraph
Try running the run.py script that comes with the starter kit examples with a profiler like the one linked above and see if you can spot any inefficient code paths.
Hope that helps and best of luck
Posted by: HuaweiUK @ Dec. 12, 2019, 6:43 a.m.