@Lazada: Can you explain zero RMSE for clarity? Looks like someone has participated in the competition with the reference keys.
Posted by: chriswolg @ May 5, 2017, 7:50 p.m.Hi,
I'm a CIKM representative that organizes this CodaLab competition.
First of all, to be clear: this is not a case where someone submitted with the reference keys. Secondly, upon investigation, we found out that the submission was actually invalid, with illegal entries that caused the Clarity score and Overall score to be NaN. This NaN was previously interpreted by the table as a 0. We have now corrected this scoring issue, and the results table is now showing the latest results. We have also notified space51 about this issue.
Thank you for pointing this out. We appreciate your effort in helping us keep a watch on what's going on!
Posted by: hadylauw @ May 5, 2017, 11:45 p.m.