Sentence-level QE task 2020

Organized by dcw19 - Current server time: April 9, 2020, 11:58 a.m. UTC

Previous

English-Chinese
Jan. 27, 2020, midnight UTC

Current

English-Chinese
Jan. 27, 2020, midnight UTC

End

Competition Ends
Feb. 28, 2020, 11:59 p.m. UTC

Sentence-level Quality Estimation Shared Task 2020

The shared task on Quality Estimation aims to examine automatic methods for estimating the quality of machine translation output at run-time, without relying on reference translations.This variant looks at sentence-level prediction, where participating systems are required to score each translated sentence according to direct assessments (DA) on their quality. The DA score is a number in 0-100 which has been given by humans, where 0 is the lowest possible quality and 100 is a perfect translation. For the task, we collected three annotations per instance and use the average of a z-standardised (per annotator) version of the raw scores.

Submission Format

The output of your system for a a given subtask should produce scores for the translations at the segment-level formatted in the following way:

<SEGMENT SCORE> 

Where:

  • SEGMENT SCORE is the predicted score for the particular segment.

Each participating team can submit at most 10 systems for a given language pair.

To allow the automatic evaluation of your predictions, please submit them in a file named as follows: predictions.txt

The file has then to be zipped to be submitted as a codalab submission. 

 

Evaluation Criteria

The scores to be submitted are an absolute quality score for each sentence translation where higher scores mean better translations.

Evaluation is performed against the true label using as metrics:

  • Pearson's correlation (primary)
  • Mean Average Error (MAE)
  • Root Mean Squared Error (RMSE).

Terms and Conditions

This page enumerated the terms and conditions of the competition.

English-German

Start: Jan. 27, 2020, midnight

English-Chinese

Start: Jan. 27, 2020, midnight

Competition Ends

Feb. 28, 2020, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In
# Username Score
1 yangst 0.5104
2 rep15 0.4858
3 Rain9876 0.4650