Forums

SemEval-2018 Task 11: Machine Comprehension using Commonsense Knowledge Forum

Go back to competition Back to thread list Post in this thread

> Test scores

Hello,

I was wondering if there is a bug in evaluation phase results, because all the participants seem to have a perfect 1.0 scores.
Could you please comment on that?
Regards,
Olga

Posted by: okovaleva @ Jan. 29, 2018, 5:51 p.m.

Hi,
no, this is not a bug. Evaluation is done offline and the evaluation script is just a dummy, outputting 1.0 or 0.0. the reason for this is that we found a bug in Codalab some days before the evaluation started, so we had to use a dummy script. The proper evaluation script will be uploaded soon.

Best
Simon

Posted by: simono @ Jan. 30, 2018, 10:05 a.m.
Post in this thread