2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE)

Organized by cbrust - Current server time: Sept. 23, 2020, 10:56 p.m. UTC

Previous

Test phase
March 30, 2020, noon UTC

Current

Dev phase
March 2, 2020, midnight UTC

End

Competition Ends
April 14, 2020, 11:59 a.m. UTC

Welcome!

This is the CodaLab site for the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE).

Shared Task Details: http://sharedtask.duolingo.com

Google Group: duolingo-sharedtask-2020

Data Sets: Available on Dataverse

Starter Code: Available on GitHub

Important Dates

Jan 13, 2020   Data release (phase 1): TRAIN sets
Mar 2, 2020   Data release (phase 2): blind DEV set
Mar 30, 2020   Data release (phase 3): blind TEST set
Apr 6, 2020   Final predictions deadline
Apr 8, 2020   Final results announcement
Apr 13, 2020   Draft system papers due
May 4, 2020   System paper reviews returned
May 18, 2020   Camera-ready system papers due
Jul 9/10, 2020   Workshop at ACL in Seattle!

Evaluation Criteria

The primary evaluation metric will be the macro-weighted F1 score (with a threshold of 0.5). By macro, we mean the F1 for each prompt will be computed individually (from precision and weighted recall) and then averaged. By weighted, we mean that instead of summing the number of true and false positives and negatives, we sum the weight of the translation, as defined on the shared task website. Note that we only weight recall in the weighted F1 score.

Terms and Conditions

All shared task participants are strongly encouraged to submit a system paper describing their approach and results, to be presented at the WNGT Workshop at the ACL conference in July 2020. This paper will be published in the workshop proceedings and available through the ACL Anthology website.

The shared task data set is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).

Dev phase

Start: March 2, 2020, midnight

Description: Submission format: Zip and upload a folder called 'results' which contains prediction files for whichever tracks you're competing in (note that you don't need to compete in all tracks). NOTE: The grading script will look for and grade one file per track by searching for the track names (e.g. 'en_ja') in the filename. If your filenames do not contain the track names, the script will NOT find your solutions. The format for submission follows the training example provided; the first line of the block contains the prompt ID and the sentence, pipe-separated; subsequent lines contain your translations. A blank line indicates that a new prompt will be beginning. If you're competing in more than one track but trying to update your predictions for just one track, please re-upload your predictions for all tracks.

Test phase

Start: March 30, 2020, noon

Description: Submission format: Zip and upload a folder called 'results' which contains prediction files for whichever tracks you're competing in (note that you don't need to compete in all tracks). NOTE: The grading script will look for and grade one file per track by searching for the track names (e.g. 'en_ja') in the filename. If your filenames do not contain the track names, the script will NOT find your solutions. The format for submission follows the training example provided; the first line of the block contains the prompt ID and the sentence, pipe-separated; subsequent lines contain your translations. A blank line indicates that a new prompt will be beginning. If you're competing in more than one track but trying to update your predictions for just one track, please re-upload your predictions for all tracks. If you experience bugs during uploading and need to increase the max number of uploads available, please contact the organizers for assistance.

Competition Ends

April 14, 2020, 11:59 a.m.

You must be logged in to participate in competitions.

Sign In