Competition

2018 Duolingo Shared Task on Second Language Acquisition Modeling (SLAM)

Organized by cbrust - Current server time: April 2, 2025, 5:07 p.m. UTC

First phase

Dev phase
March 6, 2018, midnight UTC

End

Competition Ends
March 20, 2018, 11:59 a.m. UTC

Welcome!

This is the CodaLab site for the 2018 Duolingo Shared Task on Second Language Acquisition Modeling (SLAM).

Important Dates

Jan 10, 2018 Data release (phase 1): TRAIN and DEV sets
Mar 9, 2018 Data release (phase 2): blind TEST set
Mar 19, 2018 Final predictions deadline (technically 3/20 @ noon UTC)
Mar 21, 2018 Final results announcement
Mar 28, 2018 Draft system papers due
Apr 09, 2018 System paper reviews returned
Apr 16, 2018 Camera-ready system papers due
Jun 05, 2018 Workshop at NAACL-HLT in New Orleans!

Evaluation Criteria

The primary evaluation metric will be area under the ROC curve (AUROC). F1 score (with a threshold of 0.5) will also be reported and analyzed. As such, some teams may wish to attempt combined classification and ranking methods. Note that the label 1 (denoting a mistake) is considered the "positive" class for both metrics.

The official AUROC and F1 metrics (plus a few others) are implemented in eval.py alongside the baseline code.

Terms and Conditions

All shared task participants are expected to submit a system paper describing their approach and results, to be presented at the BEA Workshop at the NAACL-HLT conference in June 2018. This paper will be published in the workshop proceedings and available through the ACL Anthology website.

The shared task data set is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).

Dev phase

Start: March 6, 2018, midnight

Description: Submission format: upload a zip containing prediction files for all tracks you are participating in (up to three). The grading script will automatically look for and grade one file containing for each track (i.e., by matching 'en_es', 'es_en', or 'fr_en' in the filename).

Test phase

Start: March 9, 2018, noon

Description: Submission format: upload a zip containing prediction files for all tracks you are participating in (up to three). The grading script will automatically look for and grade one file containing for each track (i.e., by matching 'en_es', 'es_en', or 'fr_en' in the filename).

Competition Ends

March 20, 2018, 11:59 a.m.

You must be logged in to participate in competitions.

Sign In