Competition

Shared Task on Transliteration of Named Entities

Evaluation

As in previous years' competitions, the quality of the transliteration task is measured using the following 4 metrics:

• Word Accuracy in Top-1 (ACC) Also known as Word Error Rate, it measures correctness of the first transliteration candidate in the candidate list produced by a transliteration system.
• Fuzziness in Top-1 (Mean F-score) The mean F-score measures how different, on average, the top transliteration candidate is from its closest reference.
• Mean Reciprocal Rank (MRR) Measures traditional MRR for any right answer produced by the system, from among the candidates.
• Mean Average Precision (MAP_ref) Measures the precision of the n-best candidate list with respect to the reference transliterations that are available. If all of the references are produced, then the MAP is 1.

Participants can submit several runs for each individual language pair phase. Notice this is the NON-STANDARD SUBMISSION competition. In non-standard runs, in addition to the training and development data provided by the Shared Task, participants are welcomed to use external data, either parallel or monolingual. Each submission can contain up to 10 output candidates in a ranked list for each input entry.

Each system output submission must be saved in a file named "results.xml" and submitted into the system in a .zip compressed file format. For more details on how to prepare your submissions, please refer to the Shared Task Whitepaper. For additional details on the Workshop and Shared Task evaluation please refer to the Workshop's website.

Shared Task on Transliteration of Named Entities

Term and Conditions

• Research groups or individuals interested in participating in the Shrared Task must register first. To register in the Shared Task and requesting the datasets, please follow the instructions in this link.
• Five different datasets comprising a total of 14 different transliteration tasks are available for the competition. A license agreement has to be completed online before having access to the data.
• By registering and obtaining the datasets, you express your intention for (1) participating in the Shared Task, (2) submitting at least one of your system results to the competition before the evaluation deadline, and (3) submitting a short paper describing your system to the workshop.