The evaluation stopped before it was finished showing the following error message: Execution time limit exceeded!
Is there any way around this?
Many thanks in advance!
Attila
Hi Attila,
You can use the validation set, for which the ground truth has been made available, to debug locally the evaluation of your predictions. This will most likely help you to fix the issue you have on the test set too. The time limit for the execution of the evaluation script on each submission on the CodaLab server is 30 minutes. This should be enough, as the execution usually takes only 5 minutes in our machines.
In order to run evaluation on the validation set locally, you need to download all three packages that are made available on the Files page https://competitions.codalab.org/competitions/23553#participate-get_starting_kit :
1) the starting kit, containing the evaluation code (you may have to adjust slightly for your configuration),
2) the public data from the development phase, containing the RGB images of the validation set, and
3) the public data from the testing phase, containing the ground truth annotations of the validation set.
Best regards,
Christos Sakaridis
Hi Christos,
Thanks for the quick reply.
As per your suggestion I ran the evaluation code on the validation set and the script indeed finished in ~5 minutes. I then again uploaded the test set predictions to Codalab but unfortunately the evaluation has been stuck in "Submitted" for more than 3 hours. Not sure if you can check what exactly is going on, I guess Codalab is just handling a lot of CVPR traffic at the moment. ;)
Hi Attila,
I checked your new submission and the evaluation runs properly, but the time limit is reached before the evaluation is completed (just a few images are left). So, I increased the general time limit for the execution of the evaluation to 45 minutes. I am now re-running your submission and there should be enough time for the evaluation. Let me know if there is still a problem.
Best,
Christos
Hi Christos,
Thanks again for your efforts; unfortunately the evaluation timed out again near the end. Would it be possible to have all my submissions so far (all failed) deleted so that I can give it a couple of more tries?
Best,
Attila
Done!
Christos