Diagnostic Questions - The NeurIPS 2020 Education Challenge Forum

Go back to competition Back to thread list Post in this thread

> Poll: Ability to hide top scores, do you like it or not?

Poll Question: Do you like the ability to hide your top scores from other participants?

My Vote: No

Please cast your vote if you have time.

----------------

Hello to all participants!

I have a huge concern about the ability to hide your top scores from the leaderboard. You can do it by going to one of your scores in the Results, and pressing the "Post on Leaderboard" link. This ability allows anyone to hide their top score.

I believe that this can cause issues at the end of the competition, because no one will know who how far they actually are from the top submission. Although a submission graph shows the top submission over time, rounded at the nearest 10th across all of the tasks. But I don't think it is quite helpful.

I emailed the organizers about this but haven't heard back.

Nirmal

Posted by: nirmalpatel @ Aug. 11, 2020, 6:43 p.m.

Hi Nirmal,

Thanks for raising this issue, the organizers will discuss this and get back to you shortly.

Thanks,
Angus

Posted by: anlam @ Aug. 12, 2020, 8:31 a.m.

My vote is: No too

I find that this makes leaderboards not relevant to the competition. One may check their model performance by submitting their code and checking their scores, and then immediately withdraw it from leaderboard to hide the their performance, which is a signal for other competitors to improve their model. This is almost same as checking their model performance locally with public test data. To prevent this, other participants may run a script that watches leaderboards over time to collect scores appearing for a short period, and that is an effort irrelevant to the aim of the competition in my opinion.

Posted by: xtalclr @ Aug. 13, 2020, 3:36 a.m.

yesterday, i'm the rank11, but now rank9 ??? execuse me??? i did nothing but sleeping.....

Posted by: zhuangyan @ Aug. 13, 2020, 8:52 a.m.

@zhuangyan It looks like someone pulled out their top score, and you got pushed up on the leaderboard...

Posted by: nirmalpatel @ Aug. 13, 2020, 1:55 p.m.

Agreed: No, we should not be able to hide public submissions.

Posted by: lukasolson @ Aug. 13, 2020, 2:04 p.m.

Agreed: No

Posted by: cfong32 @ Aug. 27, 2020, 8:07 a.m.

Thanks for pointing this out. I have contacted Codalab admin to figure out how to force best submission to show up in the leaderboard. They also suggest that it is good idea to always display participants' best submissions. I have changed the competition configuration so that now your best submission(s) will show up in the leaderboard and you cannot take it down.

Let me know if you have further questions or concerns on this, thanks!!

Posted by: moonlightlane @ Aug. 29, 2020, 2:53 a.m.
Post in this thread