Competition

We expect from participant a ranked list of 10 possible candidates for each new word in the test set. We will evaluate the systems using the Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) scores. MAP score pays attention to the whole range of possible hypernyms, whereas MRR looks at how close to the top of the list a first correct prediction is. In addition to that, the F1 score will be computed to evaluate the performance of the top 1 prediction of the methods. MAP will be the official metric to rank the submissions.

In order to be less restrictive during the evaluation, we consider as correct answers not only immediate hypernyms of new words, but also hypernyms of these hypernyms. Therefore, if a system predicted a hypernym of a correct hypernym, this will also be considered a match. 

However, the specificity of the ruWordNet taxonomy and our assumption about second-order hypernyms may result in confusion in the evaluation process. Let us consider the following examples

One hypernym may be a “parent” of another hypernym (synset “Moksha” has two parents “tributary” and “river”, whereas “river” is the hypernym for “tributary”). While computing MAP score, it may not be clear, which hypernym group gains the score: the one with “river” synset as immediate hypernym or “river” as second-order hypernym.

Hypernyms may share common parents: “string instrument” and “folk instrument” have hypernym “musical instrument” in common. In this case if “musical instrument” appears in the candidate list, MAP score will also be confused. In order to avoid this hypernym ambiguity, we split immediate and second-order hypernyms into separate groups.

Each group corresponds to the connectivity component in the subgraph reconstructed from these hypernyms. We see that the first and the second subgraphs possess only one connectivity component in comparison to the last subfigure, where immediate hypernyms
form different hypernym groups. 

Therefore, the list of possible candidates of a given word should contain hypernyms (at least one) from each hypernym group.

The results are published in the following paper:

Irina Nikishina, Varvara Logacheva, Alexander Panchenko, and Natalia Loukachevitch (2020): RUSSE'2020: Findings of the First Taxonomy Enrichment Task for the Russian Language. In Proceedings of the 26-th International Conference on Computational Linguistics and Intellectual Technologies (Dialogue-2020). Moscow, Russia.

BibTeX:

@inproceedings{nikishina2020taxonomy,
title={{RUSSE'2020: Findings of the First Taxonomy Enrichment Task for the Russian Language}},
author={Nikishina, Irina and Logacheva, Varvara and Panchenko, Alexander and Loukachevitch, Natalia},
booktitle={Computational Linguistics and Intellectual Technologies: papers from the Annual conference ``Dialogue''},
year={2020}
}

Predicted hypernyms for nouns from Yuriy's answer (top-1 for nouns)

Predicted hypernyms for verbs from cointegrated's answer (top-1 for verbs)

• Irina Nikishina, Skoltech (Irina.Nikishina@skoltech.ru)
• Varvara Logacheva, Skoltech (v.logacheva@skoltech.ru)
• Alexander Panchenko, Skoltech
• Natalia Loukachevitch, Lomonosov Moscow State University

Info sponsors:

• AINL: Artificial Intelligence and Natural Language Conference
• AIST: Analysis of Images, Social networks and Texts International Conference
• RusVectōrēs: word embeddings for Russian online
• Natural Language Processing Seminar (Higher School of Economics)

Join our discussion group in Telegram: https://t.me/joinchat/Ckja7Vh00qPOU887pLonqQ

Baselines

We will provide simple baselines based on distributional and neural language models. Besides, we believe that popular neural context-aware models (like ELMo and BERT) will be of particular use for this task as they can represent out-of-vocabulary words on the basis of their context. Therefore, everyone interested in testing these and other distributional semantic models are welcome to participate.

Tracks

The task will feature two tracks: detection of hypernyms for nouns and for verbs. The participants are allowed to use any additional datasets and corpora in addition to the train set based on the ruWordNet taxonomy. Moreover, we also provide the additional data: news text corpus, parsed Wikipedia corpus and hypernym database from Russian Distributional Thesaurus. However, we ask participants to mention all additional resources used for training of models.

The participants can test their models on the public test set by submitting the results to the leaderboards for each track (Nouns and Verbs). Once the private test sets is released, the participants will have two weeks to predict hypernyms for it and submit their final results.

Important dates:

• First Call for Participation: December 15, 2019.
• Release of the Training Data: December 15, 2019.
• Release of the Test Data: January 31, 2020.
• Submission of the Results: February 14, 2020 March, 1 (Anywhere on Earth).
• Results of the Shared Task: February 28, 2020  March 3, 2020.
• Article submission deadlines: March 10, 2020

Contacts

Irina.Nikishina@skoltech.ru

v.logacheva@skoltech.ru