there are some problems about the leaderboard, the result with lowest Bias (positive pairs) is shown on the leaderboard, however, this result may not be the best ranking result,
will this be fixed at the final phase?
Dear participant,
The ranking shown on the leaderbord (based on average ranking position of the 3 variables) may generate some confusion when some submitted file obtain very low accuracy (in this case, they can also show very low bias scores, which is the case of the provided starting kit & sample submission). To solve this problem, submissions that do not improve the baseline in terms of accuracy will be ignored at the end of each phase. For this reason, the ranking shown on the leaderboard may not be considered as the final ranking. For a detailed explanation, please check the "Ranking strategy, Leaderboard and basic Rules" section in our webpage (http://chalearnlap.cvc.uab.es/challenge/38/description/#ranking).
Best
thanks.
but I mean another situation,
for example:
I have two submit results,
results 1 get rank 1 on Bias (positive pairs), rank 10 on Bias (negative pairs) and rank 20 on Accuracy and
results 2 get rank 3 on Bias (positive pairs), rank 1 on Bias (negative pairs) and rank 1 on Accuracy
it;s obvious that results 2 is better than results 1, but now, it's result 1 that shown on the leaderboard, which may cause some problem
Dear participant,
this is also mentioned in our Challenge webpage (http://chalearnlap.cvc.uab.es/challenge/38/description/) at "Making a submission" section, copied below for you easy check:
"Note, Codalab will keep on the leaderboard the last valid submission. This helps participants to receive real-time feedback on the submitted files. Participants are responsible to upload the file they believe will rank them in a better position as a last and valid submission."
I hope this answer your question.
Best
thanks again，
but now the system is different from the description
I've made a valid submission which can achieve a better ranking but it was not show on the leaderboard just because
the system won't update your submission on the leaderboard unless you achieve a lower Bias (positive pairs)
I see you made a submisison today (submission ID 624459) and for some reason it is not appearing in the leaderboard.
I will investigate what happened and will return to you. Thanks for pointing this out and sorry about that.
Best
Dear participant,
In the meantime I am waiting a feedback from Codalab team, can you submit your file again to make a simple test?
Thanks and best
Recently, the codalab server is get slower for scoring?
Posted by: zhaixingzi @ June 4, 2020, 9:26 a.m.Dear zhaixingzi,
sometimes the computation of the scores may take more time depending on the size or your file. For example, because of very large float precision. I have set Codalab to run the evaluation script to a maximum time (600s being more precise). Please le me know if you have any problem related to time out and I can try to increase it. Other possible reasons related to Codalab servers our out of our control but we can check with them in case we face any problem.
Thanks and best