Microsoft AI Challenge India 2018

Organized by microsoftaichallenge_organizer - Current server time: Dec. 15, 2018, 5:34 a.m. UTC

Current

Development
Nov. 19, 2018, midnight UTC

Next

Final Evaluation Starts
Dec. 22, 2018, 6:30 p.m. UTC

End

Competition Ends
Dec. 24, 2018, 6:30 p.m. UTC

Introduction

As search engines evolve to respond to speech inputs and as usage of ambient devices like speakers grow in the society etc. returning 10 blue links to a search query is not always desirable. At Bing.com, our aim is to serve answer to questions directly without users having to search through the 10 blue links. Try searching “what is cricket ball made of” in Bing and a direct answer pops up.
 In this challenge, we are introducing the following problem statement:
 

 Given a user query and candidate passages corresponding to each, the task is to mark the most relevant passage which contains the answer to the user query.
 

In the data, there are 10 passages for each query out of which only one passage is correct. Therefore, only one passage is marked as label and all other passages for that query are marked as label 0. Your goal is to rank the passages by scoring them such that the actual correct passage gets as high score as possible.

We provide three types of data sets to the participants -- i) the labelled train data for training your models and doing validations ii) the unlabelled eval1 data against which you submit your predictions during the contest and iii) the unlabelled eval2 data against which final predictions are submitted.

Result on eval2 dataset will be used to declare winners. 

Submission and Evaluation

 
Participants need to generate result for the eval1 set (and later the eval2 set) in a file, containing 11 columns. The first column shall contain the query_id and the remaining 10 columns shall contain the scores for each of the 10 passages following the same order as in dataset. A sample submission file should look like as following: 
 
1329    0.22    0.36    0.11    0.15    0.5    0.7    0.34    0.23    0.34    0.13
256      0.3    0.4    0.3    0.2    0.5    0.1    0.7    0.2    0.3       0.6

 

Please note that all the columns should be tab separated and there should be no header.

We will evaluate using Mean Reciprocal Rank (MRR) which is defined as follows:

where ranki is the rank of the correct passage in the prediction file for the ith query across all Q queries.

Example  Illustration: Suppose for a query q, you have given 10 scores for passages 0 to 9. Your scores are: 0.22    0.36    0.11    0.15    0.5    0.7    0.34    0.23    0.34    0.13. Now suppose the 8th passage is correct according to the labels. Now according to your scores, 8th passage has score of 0.23 and hence a rank of 6 (rank is 6 because you have given higher scores 0.36, 0.34, 0.34, 0.5 and 0.7 to other passages). Therefore the MRR you get for this query is 1/6 = 0.166666667. Final MRR will be the mean MRR across all queries.

The file you submit should be a zip file containg your predictions in a tsv which must be named answer.tsv

Our evaluation script comes as a part of the Starting Kit. You can use this script to evaluate your submissions offline before making submissions.

 

 

How To Get Started?

Step 1: Get Data Sets
To get the data set, please navigate to Participate -> Files. You have to download the zip file under Public Data. This zip file contains two tsv files: data.tsv and eval1_unlabelled.tsv. The details about this data is provided in the section Participate -> Get Data.

Step 2: Get Starting kit
To make it easier to get your first AI model trained, we have provided a starting kit in Section Participate -> Files
Please download it and get started. 
Staring kit contains scripts for baseline models, which you can run on data sets provided in Step 1 and it generates the final test set in the format which is accepted by Step 3.
 
Step 3: Submit the test set for Leaderboard score evaluation
To submit the test set for evaluation and leaderboard calculation, Please go to Section Participate -> Submit/ View Results

Terms and Conditions

You can read the detailed rules, terms and conditions here

 

Things to Remember

  1. You need to carefully read all the instructions related to data, submission and evaluation before making your submission.
  2. You can submit at most 10 times per day and 200 times through out the contest in Phase 1.
  3. Your submission will be evaluated on the eval1 set and reflected on the leaderboard during the contest in Phase 1.
  4. Approximately 250 top teams will go from Phase 1 to Phase 2 based on the leader board.
  5. In Phase 2, we will release another evaluation set eval2 and you will be allowed to make 3 submissions against the same.
  6. Submissions on eval2 will be final and winners will be declared based on evaluation on eval2 and system explanation that we will seek. 

Support:

Should you have any queries, please write to the AI Challenge LinkedIn group. Discussions on the group help all the participants, and is preferable. 

Alternatively, you can reach out to us at:

aichallenge@microsoft.com

 

Organizing committee:

  • Niranjan Nayak
  • Kedhar Nath Narahari
  • Harish Yenala
  • Rajarshee Mitra
  • Simerpreet Kaur
  • Daniel Campos
  • Puneet Agrawal

Development

Start: Nov. 19, 2018, midnight

Description: Contest starts. Submission window activated. Maximum of 10 submissions per day with a total submission allowance of 200 times through out Phase 1. All times are in UTC.

Final Evaluation Starts

Start: Dec. 22, 2018, 6:30 p.m.

Description: You will be allowed to run your submission on eval2. A maximum of 3 submissions will be allowed. The final winners will be chosen based on evaluation in this phase. All times are in UTC.

Competition Ends

Dec. 24, 2018, 6:30 p.m.

You must be logged in to participate in competitions.

Sign In
# Username Score
1 msaic_dhan_pawa_3186 0.7048
2 msaic_sasa_swet_9354 0.6958
3 msaic_tgur_4588 0.6948