Given a user query and candidate passages corresponding to each, the task is to mark the most relevant passage which contains the answer to the user query.
In the data, there are 10 passages for each query out of which only one passage is correct. Therefore, only one passage is marked as label 1 and all other passages for that query are marked as label 0. Your goal is to rank the passages by scoring them such that the actual correct passage gets as high score as possible.
We provide three types of data sets to the participants -- i) the labelled train data for training your models and doing validations ii) the unlabelled eval1 data against which you submit your predictions during the contest and iii) the unlabelled eval2 data against which final predictions are submitted.
Result on eval2 dataset will be used to declare winners.
|1329 0.22 0.36 0.11 0.15 0.5 0.7 0.34 0.23 0.34 0.13|
|256 0.3 0.4 0.3 0.2 0.5 0.1 0.7 0.2 0.3 0.6|
Please note that all the columns should be tab separated and there should be no header.
We will evaluate using Mean Reciprocal Rank (MRR) which is defined as follows:
where ranki is the rank of the correct passage in the prediction file for the ith query across all Q queries.
Example Illustration: Suppose for a query q, you have given 10 scores for passages 0 to 9. Your scores are: 0.22 0.36 0.11 0.15 0.5 0.7 0.34 0.23 0.34 0.13. Now suppose the 8th passage is correct according to the labels. Now according to your scores, 8th passage has score of 0.23 and hence a rank of 6 (rank is 6 because you have given higher scores 0.36, 0.34, 0.34, 0.5 and 0.7 to other passages). Therefore the MRR you get for this query is 1/6 = 0.166666667. Final MRR will be the mean MRR across all queries.
The file you submit should be a zip file containg your predictions in a tsv which must be named answer.tsv
Our evaluation script comes as a part of the Starting Kit. You can use this script to evaluate your submissions offline before making submissions.
Step 1: Get Data Sets
To get the data set, please navigate to Participate -> Files. You have to download the zip file under Public Data. This zip file contains two tsv files: data.tsv and eval1_unlabelled.tsv. The details about this data is provided in the section Participate -> Get Data.
You can read the detailed rules, terms and conditions here
Should you have any queries, please write to the AI Challenge LinkedIn group. Discussions on the group help all the participants, and is preferable.
Alternatively, you can reach out to us at:
Start: Nov. 19, 2018, midnight
Description: The contest is over now. It is enabled now for learning and experimentation purposes only.
Start: June 30, 2019, 6:30 p.m.
Description: You will be allowed to run your submission on eval2. A maximum of 3 submissions will be allowed. The final winners will be chosen based on evaluation in this phase. All times are in UTC.
July 1, 2019, 6:31 p.m.
You must be logged in to participate in competitions.Sign In