2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE)
Previous
Test phaseCurrent
Dev phaseEnd
Competition EndsWelcome!
This is the CodaLab site for the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE).
Shared Task Details: http://sharedtask.duolingo.com
Google Group: duolingo-sharedtask-2020
Data Sets: Available on Dataverse
Starter Code: Available on GitHub
Important Dates
| Jan 13, 2020 | Data release (phase 1): TRAIN sets | |
| Mar 2, 2020 | Data release (phase 2): blind DEV set | |
| Mar 30, 2020 | Data release (phase 3): blind TEST set | |
| Apr 6, 2020 | Final predictions deadline | |
| Apr 8, 2020 | Final results announcement | |
| Apr 13, 2020 | Draft system papers due | |
| May 4, 2020 | System paper reviews returned | |
| May 18, 2020 | Camera-ready system papers due | |
| Jul 9/10, 2020 | Workshop at ACL in Seattle! |
Evaluation Criteria
The primary evaluation metric will be the macro-weighted F1 score (with a threshold of 0.5). By macro, we mean the F1 for each prompt will be computed individually (from precision and weighted recall) and then averaged. By weighted, we mean that instead of summing the number of true and false positives and negatives, we sum the weight of the translation, as defined on the shared task website. Note that we only weight recall in the weighted F1 score.
Terms and Conditions
All shared task participants are strongly encouraged to submit a system paper describing their approach and results, to be presented at the WNGT Workshop at the ACL conference in July 2020. This paper will be published in the workshop proceedings and available through the ACL Anthology website.
The shared task data set is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
Dev phase
Start: March 2, 2020, midnight
Description: Submission format: Zip and upload a folder called 'results' which contains prediction files for whichever tracks you're competing in (note that you don't need to compete in all tracks). NOTE: The grading script will look for and grade one file per track by searching for the track names (e.g. 'en_ja') in the filename. If your filenames do not contain the track names, the script will NOT find your solutions. The format for submission follows the training example provided; the first line of the block contains the prompt ID and the sentence, pipe-separated; subsequent lines contain your translations. A blank line indicates that a new prompt will be beginning. If you're competing in more than one track but trying to update your predictions for just one track, please re-upload your predictions for all tracks.
Test phase
Start: March 30, 2020, noon
Description: Submission format: Zip and upload a folder called 'results' which contains prediction files for whichever tracks you're competing in (note that you don't need to compete in all tracks). NOTE: The grading script will look for and grade one file per track by searching for the track names (e.g. 'en_ja') in the filename. If your filenames do not contain the track names, the script will NOT find your solutions. The format for submission follows the training example provided; the first line of the block contains the prompt ID and the sentence, pipe-separated; subsequent lines contain your translations. A blank line indicates that a new prompt will be beginning. If you're competing in more than one track but trying to update your predictions for just one track, please re-upload your predictions for all tracks. If you experience bugs during uploading and need to increase the max number of uploads available, please contact the organizers for assistance.
Competition Ends
April 14, 2020, 11:59 a.m.
You must be logged in to participate in competitions.
Sign In