SemEval-2018 Task 11: Machine Comprehension using Commonsense Knowledge Forum

Go back to competition Back to thread list Post in this thread

> Test scores


I was wondering if there is a bug in evaluation phase results, because all the participants seem to have a perfect 1.0 scores.
Could you please comment on that?

Posted by: okovaleva @ Jan. 29, 2018, 5:51 p.m.

no, this is not a bug. Evaluation is done offline and the evaluation script is just a dummy, outputting 1.0 or 0.0. the reason for this is that we found a bug in Codalab some days before the evaluation started, so we had to use a dummy script. The proper evaluation script will be uploaded soon.


Posted by: simono @ Jan. 30, 2018, 10:05 a.m.
Post in this thread