Diagnostic Questions - The NeurIPS 2020 Education Challenge

Organized by moonlightlane - Current server time: Aug. 11, 2020, 10:09 p.m. UTC

Previous

Task 4 Public
July 15, 2020, midnight UTC

Current

Task 4 Private
July 15, 2020, midnight UTC

End

Competition Ends
Oct. 23, 2020, midnight UTC

Introducing the NeurIPS 2020 Education Challenge

Welcome to the NeurIPS 2020 Education Challenge! 

Digital technologies are becoming increasingly prevalent in education, enabling personalized, high quality education resources to be accessible by students across the world. Importantly, among these resources are diagnostic questions: the answers that the students give to these questions reveal key information about the specific nature of misconceptions that the students may hold. Analyzing the massive quantities of data stemming from students’ interactions with these diagnostic questions can help us more accurately understand the students’ learning status and thus allow us to automate learning curriculum recommendations.

In this competition, you will focus on the students’ answer records to these multiple-choice diagnostic questions, with the aim of 1) accurately predicting which answers the students provide; 2) accurately predicting which questions have high quality; and 3) determining a personalized sequence of questions for each student that best predicts the student’s answers. These tasks closely mimic the goals of a real-world educational platform and are highly representative of the educational challenges faced today.

We provide data from the last two school years (2018-2020) of students’ answers to mathematics questions from Eedi, a leading educational platform which millions of students interact with daily around the globe. By participating, you have a chance to make a lasting, real-world impact on the quality of personalized education for millions of students across the world.

More information, including instructions to get started and make submissions, can be found in the official competition guide here

 

 

Official Competition Guide

We have provided a document that contains more detailed information about the competition including dataset description, detailed task descriptions and evaluation metrics, submission instructions and how to get started. The document can be downloaded here.

Get Data

We provide training data for all tasks along with metadata on students, questions and answers. All data can be downloaded from this website. Please go to "Participate" tab, then click the "Public Data" button to download the data.

A detailed description of each data file can be found in the official competition guide

Starter Kit

We prepare useful files and scripts in a starter kit to help you get started on the competition. The starter kit contains 1) useful scripts to help you get started on the competition and 2) submission templates to help you prepare submission files for the competition tasks. 

The starter kit can be downloaded from the competition website. Please go to the "Participate" tab, then click on the "Starting Kit" to download the starter kit. 

Detailed instructions of using the starter kit, including data loading, local evaluation and submission preparation, are available in the official competition guide.

Rules

Eligibility

  • Users may work in teams of up to 10 people.
  • For participants to be eligible for prizes, their code must be publicly available.
  • Any parties or individuals affiliated with Eedi, Microsoft Research, the Computational, Biological Learning Group@University of Cambridge and Baraniuk Group@Rice University are not eligible for the prizes, but may participate in the competition. In particular, participants who are affiliated with these organisations may still enter the competition, be ranked and present their solutions at the competition workshop, but are not eligible for receipt of the cash prize.

Data

  • The use of supplementary existing open-source datasets for e.g. pre-training is permitted provided they are credited. The use of private, proprietary datasets is not permitted.
  • The questions images have been shared solely for the purpose of this competition and must not be used for any other purpose. The question images must not printed or shared with anyone outside of the competition.

Submission

  • The submitted code must generate predictions using a machine learning model. Submissions making hard-coded predictions are forbidden.
  • For the 4th task that requires a code submission to be executed, your code must run on the provided compute environment within 30 minutes. This is to encourage computationally efficient solutions appropriate for real-time personalized education.
  • During the evaluation of the submitted code, the computation environment has no access to any external network. This is to avoid any information leak.
  • Users may make use of open source libraries given proper attribution. At the end of the competition, we encourage all code to be open-sourced so the results can be reproduced.
  • In order to prevent cheating, all the evaluation data will be kept completely inaccessible to the participants during the competition. The aforementioned rules on manual review of the submissions and lack of network access during code evaluation also aim to prevent cheating.

In addition, any instances of cheating, defined as violation of the rules above or any other attempt to circumvent the spirit and intent of the competition, as determined by the organizers, will result in the invalidation of all of the offending team’s submissionsand a disqualification from the competition.

 

 

 

Competition Tasks

There are four tasks of varying styles of this competition. Below is a brief description of each task.

Each task contains a public evaluation phase and a private evaluation phase. In the public evaluation phase, you can see the evaluation results of your submission in a public leaderboard compared to other participants. In the private evaluation phase, the evaluation results of the competition is not publicly viewable.

More Information on the tasks, including the evaluation metrics, can be found in the official competition guide here

Task 1

Predict whether or not students will answer questions correctly.

Real world impact: Enable recommending questions of an appropriate difficulty to a given student that best fit their background and learning status.

Task 2

The second task extends this to the prediction of which multiple-choice answer students choose for each question.

Real world impact: Enable discovering potential common misconceptions that students have by clustering of question-answer pairs which may indicate the same or related misconceptions.

Task 3

The third task is to devise a metric to measure the quality of the questions. This metric will be evaluated against the opinions of a number of domain experts.

Real world impact: Enable  feedback to be provided to authors of diagnostic questions so they can revise poor quality questions and to guide teachers to choose questions for their students.

Task 4

The fourth task is to acquire a limited set of answers from students in order to accurately predict student performance prediction on unseen questions. This requires personalized machine learning methods with the ability to estimate the value of information.

Real world impact: Enable personalized assessments for each student to improve learning outcomes.

 

 

 

 

 

Schedule

  • July 15, 2020: Submission to all tasks of the competition opens; dataset and starter kit released.
  • October 23 2020: Final submission deadline for all tasks.
  • November 15 2020: Final competition results announced, private evaluation leaderboards revealed, prize-winners notified.

Competition Prizes

Microsoft and Eedi will provide more than $5,000 cash prize in total for the competition. There will be $1,000 prizes awarded to the winning team for each task. In addition, a $1,000 prize for the overall winner across all tasks will be awarded, as determined by the team's average rank across each competition task (smallest average rank wins). If a team hasn't submitted a working solution to a particular task, their rank for that task will be considered to be equal to the number of entrants across all tasks in total.

In the event of a tie, this prize will be split evenly between the tied teams.

Azure Cloud Computing Credits

Microsoft made available 50 grants of $250 Azure cloud computing credits to students participating in the contest. Students are not required to use Azure to compete. Please fill in the application form here to apply. The credits will be allocated to the first 50 valid applications.

The organizers of the competition are:

Jack Wang, Rice University (lead organizer)

Angus Lamb, Microsoft Research

Evgeny Saveliev, Microsoft Research

Pashmina Cameron, Microsoft Research

Yordan Zaykov, Microsoft Research

José Miguel Hernández-Lobato, University of Cambridge

Richard Turner, University of Cambridge

Richard G. Baraniuk, Rice University

Craig Barton, Eedi

Simon Peyton Jones, Microsoft Research

Simon Woodhead, Eedi

Cheng Zhang, Microsoft Research

 

Contact

edu_competition@outlook.com

Contact

If you have questions about the competition which may be of use to others, please post them in the forums and we will try to get back to you as soon as possible!

The organizers can also be contacted via email at edu_competition@outlook.com

Task 1 Public

Start: July 15, 2020, midnight

Description: Predict Student Responses – Right or Wrong

Task 1 Private

Start: July 15, 2020, midnight

Description: Predict Student Responses – Right or Wrong

Task 2 Public

Start: July 15, 2020, midnight

Description: Predict Student Responses – Answer Prediction, private evaluation

Task 2 Private

Start: July 15, 2020, midnight

Description: Predict Student Responses – Answer Prediction, private evaluation

Task 3 Public

Start: July 15, 2020, midnight

Description: Global Question Quality Assessment

Task 3 Private

Start: July 15, 2020, midnight

Description: Global Question Quality Assessment, private evaluation

Task 4 Public

Start: July 15, 2020, midnight

Description: Personalized Questions

Task 4 Private

Start: July 15, 2020, midnight

Description: Personalized Questions

Competition Ends

Oct. 23, 2020, midnight

You must be logged in to participate in competitions.

Sign In
# Username Score
1 anlam 0.6535
2 simonwoodhead 0.6535
3 moonlightlane 0.6535