IMPORTANT UPDATES (16.01.2020):
- Development sets released for 5 languages and 10 cross-lingual pairs, available here. Please carefully read the README file included in the archive.
- The official evaluation period starts on February 19, 2020. This is when we will release the test sets (which will include one more surprise language). The evaluation period ends on March 11, and this is the end date by which the participants need to submit their runs. We will release detailed instructions on how and where to submit your runs (latest with the release of the test data, on Feb 19).
This shared task is about predicting binary and graded Lexical Entailment (i.e., is-a or hyponym-hypernym relation) for several different languages (multilingual component) and across languages for several language pairs (cross-lingual component).
For Graded LE, the participants need to predict the degree (on a 0-6 scale) to which the LE relation holds between two given concepts (the two concepts in each pair come from the same language in multilingual subtasks and from different languages in the cross-lingual subtasks). For Binary LE the participants merely need to predict whether the LE relation holds between two concepts or not.
The two main branches of subtasks are as follows:
- Subtask 1: Monolingual in multiple languages (i.e., multilingual) - Subtask 2: Cross-lingual
We cover the following languages along with their language ISO codes:
- English: EN - German: DE - Italian: IT - Croatian: HR - Turkish: TR (currently not covered in the trial data, data preparation and annotation is in progress; it will be available beginning of September) - One surprise evaluation language
We will evaluate systems in two different tracks:
The trial data has been released (for online and offline evaluation), see the "Data" tab. The development data will be released by Sep 5, 2019. The official evaluation (with the release of the test data) starts on January 10, 2020 and will be open until January 31, 2020.
There are two evaluation tracks:
Groups can submit more than one system only if the systems differ in a meaningful way from one another, if unsure, contact the organizers. In any case, maximal number of runs per track (for the official SemEval evaluation) is 3 (i.e., max 3 runs for the DIST track; max 3 runs for the ANY track; in both cases the runs must have meaningful distinctions between them).
All data released for this task is done so under the Creative Commons License (Non-Commercial-Attribution-ShareAlike).
Organizers of the competition might choose to publicize, analyze and change in any way any content sent as a part of this task. Whenever appropriate academic citation for the sending team will be added (e.g. in a task overview paper).
Competitions should comply with any general rules of SEMEVAL.
The organizers are free to penalize or disqualify participants for any violation of the above rules or for misuse, unethical behaviour or other behaviours they agree are not accepted in a scientific competition in general and in the specific one at hand.
Start: Aug. 13, 2019, midnight
Start: Feb. 19, 2020, midnight
Start: Feb. 20, 2020, midnight
You must be logged in to participate in competitions.Sign In