The task on Quality Estimation aims to examine automatic methods for estimating the quality of machine translation output at run-time, without relying on reference translations.This variant looks at sentence-level prediction, where systems are required to score each translated sentence according to direct assessments (DA) on their quality. The DA score is a number in 0-100 which has been given by humans, where 0 is the lowest possible quality and 100 is a perfect translation. This task focuses on predictions for English-German and English-Chinese with few labelled datapoints. It pushes for methods for few-shot learning as well as transfer learning from data for other three languages.
Scoring
Submissions will be evaluated according to Pearson correlation as the main metric.
Submission Format
The output of your system for a given language pair should produce scores for the translations at the segment-level formatted in the following way:
<SEGMENT SCORE>
Where:
SEGMENT SCORE
is the predicted score for the particular segment.Each participating team can submit at most 100 systems for each of the two language pairs.
To allow the automatic evaluation of your predictions, please submit them in a file named as follows: predictions.txt
The file has then to be zipped to be submitted as a codalab submission.
Start: April 3, 2020, midnight
Start: April 3, 2020, midnight
April 30, 2020, 11:59 p.m.
You must be logged in to participate in competitions.
Sign In