Competition

Sentence-level QE task

Organized by lspecia - Current server time: May 31, 2020, 6:45 a.m. UTC

Previous

English-Chinese
April 3, 2020, midnight UTC

Current

English-Chinese
April 3, 2020, midnight UTC

End

Competition Ends
April 30, 2020, 11:59 p.m. UTC

Sentence-level Quality Estimation Task

The task on Quality Estimation aims to examine automatic methods for estimating the quality of machine translation output at run-time, without relying on reference translations.This variant looks at sentence-level prediction, where systems are required to score each translated sentence according to direct assessments (DA) on their quality. The DA score is a number in 0-100 which has been given by humans, where 0 is the lowest possible quality and 100 is a perfect translation. This task focuses on predictions for English-German and English-Chinese with few labelled datapoints. It pushes for methods for few-shot learning as well as transfer learning from data for other three languages.

 

Scoring

Submissions will be evaluated according to Pearson correlation as the main metric.

Submission Format

The output of your system for a given language pair should produce scores for the translations at the segment-level formatted in the following way:

<SEGMENT SCORE>

Where:

  • SEGMENT SCORE is the predicted score for the particular segment.

Each participating team can submit at most 100 systems for each of the two language pairs.

To allow the automatic evaluation of your predictions, please submit them in a file named as follows: predictions.txt

The file has then to be zipped to be submitted as a codalab submission. 

 

English-German

Start: April 3, 2020, midnight

English-Chinese

Start: April 3, 2020, midnight

Competition Ends

April 30, 2020, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In