Hello,
After scorer was updated I can not compare new submissions vs. previous ones. For instance, a submission that previously received 0.4967912355, is getting 0.4394035721, if uploaded anew. But the first one was not updated and remains 0.4967912355 on the list. So I have exactly same submissions with different scores and totally disoriented. Furthermore, it appears that leader board is mixed and does not reflect updated scores too, at least in part.
Please publish the scorer code.
Thank you.
Posted by: rakhlin @ April 22, 2018, 5:13 p.m.Q: For instance, a submission that previously received 0.4967912355, is getting 0.4394035721, if uploaded anew. But the first one was not updated and remains 0.4967912355 on the list. So I have exactly same submissions with different scores and totally disoriented.
A: Unfortunately this is how CodaLab works - the re-run would duplicate a submission that will be evaluated by the new evaluation script, while the old score remain there forever (but they don't affect the leaderboard).
Q: Furthermore, it appears that leader board is mixed and does not reflect updated scores too, at least in part.
A: We have made sure that all submissions on the leaderboard had been re-evaluated, and to be safe, now we've scheduled a re-run on all submissions. Note that CodaLab only shows the score for the latest submission, instead of the highest.
Posted by: jinghuang @ April 23, 2018, 5:48 p.m.