Diagnostic Questions - The NeurIPS 2020 Education Challenge Forum

Go back to competition Back to thread list Post in this thread

> Clarification - Task 3 Evaluation

Hi all,

We've had a couple of questions about the specifics of the evaluation procedure for Task 3. In addition to the description included in the competition guide, we are sharing here part of the code used for evaluating the submitted solutions in case it is helpful for you:

# extract ranking
left = list(truth.left)
right = list(truth.right)
if len(left) != len(right):
message = 'left and right lengths are not the same'

submission_left = []
submission_right = []
submission_preference = []
for idx in range(len(left)):
ranking_left = submission[submission.QuestionId==left[idx]].ranking.values[0]
ranking_right = submission[submission.QuestionId==right[idx]].ranking.values[0]
preference = 1 if ranking_left < ranking_right else 2

Hope this helps!

Posted by: anlam @ Oct. 2, 2020, 3:46 p.m.
Post in this thread