Hi all,
We've had a couple of questions about the specifics of the evaluation procedure for Task 3. In addition to the description included in the competition guide, we are sharing here part of the code used for evaluating the submitted solutions in case it is helpful for you:
# extract ranking
left = list(truth.left)
right = list(truth.right)
if len(left) != len(right):
message = 'left and right lengths are not the same'
sys.exit(message)
submission_left = []
submission_right = []
submission_preference = []
for idx in range(len(left)):
submission_left.append(left[idx])
submission_right.append(right[idx])
ranking_left = submission[submission.QuestionId==left[idx]].ranking.values[0]
ranking_right = submission[submission.QuestionId==right[idx]].ranking.values[0]
preference = 1 if ranking_left < ranking_right else 2
submission_preference.append(preference)
Hope this helps!
Angus