metadata.txt
file to be included in the submission zip file of the test phases. Please see more information in the Submission Instructions tab.
This shared task (organized by the 2nd Eval4NLP workshop) consists of building a quality estimation system that (i) predicts the quality score for an input pair of source text and MT hypothesis and (ii) provides word-level evidence for its predictions as explanations. In other words, the explanations should highlight specific errors in the MT output which lead to the quality score predicted. We will evaluate how similar the generated explanations are to human explanations, using a test set with manually annotated rationales.
The competition consists of two main phases.
The aim of evaluation is to assess the quality of explanations, not sentence-level predictions. Therefore, the metrics for evaluation will be (1) AUC (2) AP (Average Precision) and (3) Recall at top-K for word-level explanations. The leaderboard will be sorted by the average rank of the three metrics on the target sentence explanations (i.e., Rank Target
).
Each submission is a zip file consisting of three or four files.
metadata.txt
must have exactly three non-empty lines.
constrained
or unconstrained
, indicating the submission track. constrained
means that you did not train your system on word-level labels, whereas unconstrained
means that you trained your system on word-level labels.sentence.submission
with sentence-level scores, one score per line.target.submission
with target token-level scores. Each line must contain a sequence of scores separated by white space. The number of scores must correspond to the number of target tokens.source.submission
with source token-level scores. Each line must contain a sequence of scores separated by white space. The number of scores must correspond to the number of source tokens.Examples of the submission files for the two development phases can be found here.
Start: June 19, 2021, midnight
Description: Submit your results on the dev set of Et-En
Start: June 19, 2021, midnight
Description: Submit your results on the dev set of Ro-En
Start: Aug. 20, 2021, midnight
Description: Submit your results on the test set of Et-En
Start: Aug. 20, 2021, midnight
Description: Submit your results on the test set of Ro-En
Start: Aug. 20, 2021, midnight
Description: Submit your results on the test set of De-Zh
Start: Aug. 20, 2021, midnight
Description: Submit your results on the test set of Ru-De
Sept. 4, 2021, noon
You must be logged in to participate in competitions.
Sign In# | Username | Score |
---|---|---|
1 | Raphael_NICT | 1.000 |
2 | mtreviso2 | 2.000 |
3 | Gringham | 4.333 |