NEWS 2016 (Non-standard Submissions)

Organized by rembanchs - Current server time: Nov. 14, 2018, 12:04 p.m. UTC

First phase

English to Chinese
April 25, 2016, midnight UTC

End

Competition Ends
May 4, 2016, midnight UTC

Shared Task on Transliteration of Named Entities

Overview

Named Entities (NE) play a crucial role in many monolingual and multilingual Natural Language Processing (NLP) and Information Retrieval (IR) tasks, such as document search, clustering, information extraction, etc. The phenomenal growth of the Internet data and the dramatic changes in the user demographics especially among the non-English speaking parts of the world, has made identification, association and transformation of Named Entities across languages a critical path problem for most NLP and IR Tasks.

The Sixth Workshop on Named Entities (NEWS 2016) together with a Shared Task on Transliteration is a continuous effort on top of the success of NEWS 2009 (ACL-IJCNLP 2009), NEWS 2010 (ACL 2010), NEWS 2011(IJCNLP 2011), NEWS 2012 (ACL 2012) and NEWS 2015 (ACL 2015). NEWS 2016 continues the tradition to run a Shared Task on Transliteration, which aims to provide a common platform for benchmarking the transliteration approaches and systems across many languages.

For additional details about the Workshop and the Shared Task, please refer to the Workshop's website.

Shared Task on Transliteration of Named Entities

Evaluation

As in previous years' competitions, the quality of the transliteration task is measured using the following 4 metrics:

  • Word Accuracy in Top-1 (ACC) Also known as Word Error Rate, it measures correctness of the first transliteration candidate in the candidate list produced by a transliteration system.
  • Fuzziness in Top-1 (Mean F-score) The mean F-score measures how different, on average, the top transliteration candidate is from its closest reference.
  • Mean Reciprocal Rank (MRR) Measures traditional MRR for any right answer produced by the system, from among the candidates.
  • Mean Average Precision (MAPref) Measures the precision of the n-best candidate list with respect to the reference transliterations that are available. If all of the references are produced, then the MAP is 1.

Participants can submit several runs for each individual language pair phase. Notice this is the NON-STANDARD SUBMISSION competition. In non-standard runs, in addition to the training and development data provided by the Shared Task, participants are welcomed to use external data, either parallel or monolingual. Each submission can contain up to 10 output candidates in a ranked list for each input entry.

Each system output submission must be saved in a file named "results.xml" and submitted into the system in a .zip compressed file format. For more details on how to prepare your submissions, please refer to the Shared Task Whitepaper. For additional details on the Workshop and Shared Task evaluation please refer to the Workshop's website.

Shared Task on Transliteration of Named Entities

Term and Conditions

  • Research groups or individuals interested in participating in the Shrared Task must register first. To register in the Shared Task and requesting the datasets, please follow the instructions in this link.
  • Five different datasets comprising a total of 14 different transliteration tasks are available for the competition. A license agreement has to be completed online before having access to the data.
  • By registering and obtaining the datasets, you express your intention for (1) participating in the Shared Task, (2) submitting at least one of your system results to the competition before the evaluation deadline, and (3) submitting a short paper describing your system to the workshop.

English to Chinese

Start: April 25, 2016, midnight

Description: EnCh: English to Chinese Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnCh_1008.xml"

Chinese to English *

Start: April 25, 2016, midnight

Description: ChEn: Chinese to English Back-transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_ChEn_1019.xml"

English to Thai

Start: April 25, 2016, midnight

Description: EnTh: English to Thai Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnTh_1236.xml"

Thai to English *

Start: April 25, 2016, midnight

Description: ThEn: Thai to English Back-transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_ThEn_1236.xml"

English to Persian

Start: April 25, 2016, midnight

Description: EnPe: English to Persian Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnPe_1042.xml"

English to Hindi

Start: April 25, 2016, midnight

Description: EnHi: English to Hindi Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnHi_1000.xml"

English to Tamil

Start: April 25, 2016, midnight

Description: EnTa: English to Tamil Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnTa_1000.xml"

English to Kannada

Start: April 25, 2016, midnight

Description: EnKa: English to Kannada Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnKa_1000.xml"

English to Bangla

Start: April 25, 2016, midnight

Description: EnBa: English to Bangla Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnBa_1000.xml"

English to Hebrew

Start: April 25, 2016, midnight

Description: EnHe: English to Hebrew Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnHe_1100.xml"

English to Korean

Start: April 25, 2016, midnight

Description: EnKo: English to Korean (Hangul) Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnKo_1050.xml"

English to Japanese (Katakana)

Start: April 25, 2016, midnight

Description: EnJa: English to Japanese (Katakana) Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_EnJa_1033.xml"

English to Japanese (Kanji) *

Start: April 25, 2016, midnight

Description: JnJk: English to Japanese (Kanji) Back-transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_JnJk_1095.xml"

Arabic to English

Start: April 25, 2016, midnight

Description: ArEn: Arabic to English Transliteration Task. NON-STANDARD "results.xml" for test file "NEWS12_test_ArEn_1156.xml"

Competition Ends

May 4, 2016, midnight

You must be logged in to participate in competitions.

Sign In