Diagnostic Questions - The NeurIPS 2020 Education Challenge

Organized by moonlightlane - Current server time: Jan. 21, 2021, 3:52 p.m. UTC

First phase

Task 1 Public
July 15, 2020, midnight UTC


Competition Ends
Oct. 23, 2020, midnight UTC

Introducing the NeurIPS 2020 Education Challenge

Update Dec 11, 2020

The competition is now finished. You can still access competition material on this site but we have open sourced all data (including evaluation data), sctipts and some participants solutions (extended abstracts and, in some cases, code) on this website

Welcome to the NeurIPS 2020 Education Challenge! 

Digital technologies are becoming increasingly prevalent in education, enabling personalized, high quality education resources to be accessible by students across the world. Importantly, among these resources are diagnostic questions: the answers that the students give to these questions reveal key information about the specific nature of misconceptions that the students may hold. Analyzing the massive quantities of data stemming from students’ interactions with these diagnostic questions can help us more accurately understand the students’ learning status and thus allow us to automate learning curriculum recommendations.

In this competition, you will focus on the students’ answer records to these multiple-choice diagnostic questions, with the aim of 1) accurately predicting which answers the students provide; 2) accurately predicting which questions have high quality; and 3) determining a personalized sequence of questions for each student that best predicts the student’s answers. These tasks closely mimic the goals of a real-world educational platform and are highly representative of the educational challenges faced today.

We provide data from the last two school years (2018-2020) of students’ answers to mathematics questions from Eedi, a leading educational platform which millions of students interact with daily around the globe. By participating, you have a chance to make a lasting, real-world impact on the quality of personalized education for millions of students across the world.

More information, including instructions to get started and make submissions, can be found in the official competition guide here





  • Users may work in teams of up to 10 people.
  • For participants to be eligible for prizes, their code must be publicly available.
  • Any parties or individuals affiliated with Eedi, Microsoft Research, the Computational, Biological Learning Group@University of Cambridge and Baraniuk Group@Rice University are not eligible for the prizes, but may participate in the competition. In particular, participants who are affiliated with these organisations may still enter the competition, be ranked and present their solutions at the competition workshop, but are not eligible for receipt of the cash prize.
  • In order to be eligible for receipt of the cash prize, participants must be willing to open source the source code to their submission.


  • The use of supplementary existing open-source datasets for e.g. pre-training is permitted provided they are credited. The use of private, proprietary datasets is not permitted.
  • The questions images have been shared solely for the purpose of this competition and must not be used for any other purpose. The question images must not printed or shared with anyone outside of the competition.
  • Data other than the images may be used for non-commerical purposes, provided that the competition guide is cited in any resulting publications, using the following BibTeX: 
      title={Diagnostic Questions: The NeurIPS 2020 Education Challenge},
      author={Wang, Zichao and Lamb, Angus and Saveliev, Evgeny and Cameron, Pashmina and Zaykov, Yordan and Hern{\'a}ndez-Lobato, Jos{\'e} Miguel and Turner, Richard E and Baraniuk, Richard G and Barton, Craig and Jones, Simon Peyton and others},
      journal={arXiv preprint arXiv:2007.12061},


  • The submitted code must generate predictions using a machine learning model. Submissions making hard-coded predictions are forbidden.
  • For the 4th task that requires a code submission to be executed, your code must run on the provided compute environment within 30 minutes. This is to encourage computationally efficient solutions appropriate for real-time personalized education.
  • During the evaluation of the submitted code, the computation environment has no access to any external network. This is to avoid any information leak.
  • Users may make use of open source libraries given proper attribution. At the end of the competition, we encourage all code to be open-sourced so the results can be reproduced.
  • In order to prevent cheating, all the evaluation data will be kept completely inaccessible to the participants during the competition. The aforementioned rules on manual review of the submissions and lack of network access during code evaluation also aim to prevent cheating.

In addition, any instances of cheating, defined as violation of the rules above or any other attempt to circumvent the spirit and intent of the competition, as determined by the organizers, will result in the invalidation of all of the offending team’s submissionsand a disqualification from the competition.




Competition Tasks

There are four tasks of varying styles of this competition. Below is a brief description of each task.

Each task contains a public evaluation phase and a private evaluation phase. In the public evaluation phase, you can see the evaluation results of your submission in a public leaderboard compared to other participants. In the private evaluation phase, the evaluation results of the competition is not publicly viewable.

More Information on the tasks, including the evaluation metrics, can be found in the official competition guide here

Task 1

Predict whether or not students will answer questions correctly.

Real world impact: Enable recommending questions of an appropriate difficulty to a given student that best fit their background and learning status.

Task 2

The second task extends this to the prediction of which multiple-choice answer students choose for each question.

Real world impact: Enable discovering potential common misconceptions that students have by clustering of question-answer pairs which may indicate the same or related misconceptions.

Task 3

The third task is to devise a metric to measure the quality of the questions. This metric will be evaluated against the opinions of a number of domain experts.

Real world impact: Enable  feedback to be provided to authors of diagnostic questions so they can revise poor quality questions and to guide teachers to choose questions for their students.

Task 4

The fourth task is to acquire a limited set of answers from students in order to accurately predict student performance prediction on unseen questions. This requires personalized machine learning methods with the ability to estimate the value of information.

Real world impact: Enable personalized assessments for each student to improve learning outcomes.






Official Competition Guide

We have provided a document that contains more detailed information about the competition including dataset description, detailed task descriptions and evaluation metrics, submission instructions and how to get started. The document can be downloaded here.

Get Data

We provide training data for all tasks along with metadata on students, questions and answers. All data can be downloaded from this website. Please go to "Participate" tab, then click the "Public Data" button to download the data.

A detailed description of each data file can be found in the official competition guide

Starter Kit

We prepare useful files and scripts in a starter kit to help you get started on the competition. The starter kit contains 1) useful scripts to help you get started on the competition and 2) submission templates to help you prepare submission files for the competition tasks. 

The starter kit can be downloaded from the competition website. Please go to the "Participate" tab, then click on the "Starting Kit" to download the starter kit. 

Detailed instructions of using the starter kit, including data loading, local evaluation and submission preparation, are available in the official competition guide.


  • July 15, 2020: Submission to all tasks of the competition opens; dataset and starter kit released.
  • October 23 2020: Final submission deadline for all tasks.
  • October 26 2020: Final competition results announced, private evaluation leaderboards revealed, prize-winners notified.
  • November 7 2020: Extended abstract submission deadline for top 25% participants for each task who wish to publish their solutions (see Writen Submissions subsection for more information)

Competition Prizes

Microsoft and Eedi will provide more than $5,000 cash prize in total for the competition. There will be $1,000 prizes awarded to the winning team for each task. In addition, a $1,000 prize for the overall winner across all tasks will be awarded, as determined by the team's average rank across each competition task (smallest average rank wins). If a team hasn't submitted a working solution to a particular task, their rank for that task will be considered to be equal to the number of entrants across all tasks in total.

In the event of a tie, this prize will be split evenly between the tied teams.

Azure Cloud Computing Credits

Microsoft made available 50 grants of $250 Azure cloud computing credits to students participating in the contest. Students are not required to use Azure to compete. Please fill in the application form here to apply. The credits will be allocated to the first 50 valid applications.

In order to be eligible for the cash prize, the top-scoring team for each task is required to make the code for their solution publicly available under an open source license. These teams' solutions will be summarised in the competition summary paper published as part of the NeurIPS 2020 proceedings. These teams will be invited to give a short talk summarising their solution as part of the competition's talk at NeurIPS 2020.

Furthermore, for each competition task, teams scoring in the top 25% of the private leaderboard (where multiple members of a single team are counted as a single entry) will be invited to submit an extended abstract describing their solution to the task, which will be published on the competition homepage. In addition, these submissions will be referenced in the competition summary paper and the competition’s summary talk. Participants/teams who are eligible to submit written submissions for multiple tasks may make a separate submission for each task, or combine multiple tasks into a single written submission if the approach was largely the same. The deadline for both of these written submission will be midnight UTC on the night of the 7th of November 2020. Note that publication on the competition homepage does not constitute archival publication, and no formal proceedings will be published for the competition, so that competitors are free to additionally publish their solution in archival venues.

Submissions should use the provided style files, which are largely identical to the NeurIPS 2020 style files, and submissions are limited to a maximum length of 4 pages (excluding references and supplementary material). Teams eligible for submitting abstracts will be notified shortly after the competition closes. Submissions are to be submitted to the competition email address (edu_competition@outlook.com), ensuring that the Codalab username associated with your submission is included in the email's subject line.

The organizers of the competition are:

Jack Wang, Rice University (lead organizer)

Angus Lamb, Microsoft Research (lead organizer)

Evgeny Saveliev, Microsoft Research

Pashmina Cameron, Microsoft Research

Yordan Zaykov, Microsoft Research

José Miguel Hernández-Lobato, University of Cambridge

Richard Turner, University of Cambridge

Richard G. Baraniuk, Rice University

Craig Barton, Eedi

Simon Peyton Jones, Microsoft Research

Simon Woodhead, Eedi

Cheng Zhang, Microsoft Research





If you have questions about the competition which may be of use to others, please post them in the forums and we will try to get back to you as soon as possible!

The organizers can also be contacted via email at edu_competition@outlook.com

Task 1 Public

Start: July 15, 2020, midnight

Description: Predict Student Responses – Right or Wrong

Task 1 Private

Start: July 15, 2020, midnight

Description: Predict Student Responses – Right or Wrong

Task 2 Public

Start: July 15, 2020, midnight

Description: Predict Student Responses – Answer Prediction, private evaluation

Task 2 Private

Start: July 15, 2020, midnight

Description: Predict Student Responses – Answer Prediction, private evaluation

Task 3 Public

Start: July 15, 2020, midnight

Description: Global Question Quality Assessment

Task 3 Private

Start: July 15, 2020, midnight

Description: Global Question Quality Assessment, private evaluation

Task 4 Public

Start: July 15, 2020, midnight

Description: Personalized Questions

Task 4 Private

Start: July 15, 2020, midnight

Description: Personalized Questions

Competition Ends

Oct. 23, 2020, midnight

You must be logged in to participate in competitions.

Sign In