Hi,
First of all, I would like to thank the organizers for organizing this competition. It has been mostly great, but I have a couple of criticisms and a request.
As you know by now, the submission system has been broken since the 30th of June (if not earlier).
This means that many of the participants have been unable to evaluate their models on the validation set even though they submitted their results well before the deadline for the first phase.
Moreover, you ignored the calls from multiple participants who were affected by this problem to postpone the deadline for the first phase (I understand that a deadline extension can be difficult because of the upcoming conference, but an acknowledgement would have been appreciated nonetheless).
As you can understand, this creates an unfair advantage for those who managed to submit their results before the submission system stopped working. Personally, I have been stuck for the last five days waiting to learn my validation performance to select/optimize the best model for the second phase (considering that training can take multiple days, I have been reluctant to retrain on another split of the training set).
So, I would like to ask you to release the validation ground truths for the very least so that we can do our own evaluations. I would really like to hear your feedback on this issue.
Thanks!
Posted by: spinpop56 @ July 4, 2016, 8:26 p.m.Dear Participant,
First of all, thank you very much for transmiting us your feelings and feedback.
We are aware of the submission problem, which is an internal error of Codalab that we did not expect to happen. In addition, the problem has appeared in the same dates of the CVPR conference, and hence it has been impossible to answer before since many members of the organization attended to the conference. However, we did not ignore the calls of those participants who suggested to postpone the deadline for the first phase (please see the answers to other multiple posts related to the same issue). The problem is that the schedule does not allow us to do that and cannot release the validation labels, so that we have been until now figuring out alternatives, as the competition is still running. In this sense, after several the meetings with the organization we decided that we are going to run all your submissions that you did for the learning phase and provide you the results in a new post soon. This will allow all participants to decide which submission they want to submit on the second test phase.
By the way, note that the test phase is blind, so we will compute the results of the last submissions of each participant, but we will provide the results when the competition ends (after verifying the code of the winners).
We sincerely apologize for the waiting time. Please note we are going to provide the results for the learning phase and we have a lot of submissions to compute, but we are going to do this as soon as possible.
Thank you for your comprehension.
Best regards,
Posted by: vponcel @ July 5, 2016, 11:21 a.m.