GramEval-2020
Previous
Private TestCurrent
Private TestEnd
Competition EndsGramEval-2020
We believe that multi-level language structures need to be labelled together, otherwise errors in one tag level will lead to errors in the following. Existing pipelines “tokenization - morphology - lemmatization - syntax” accumulate errors at each stage.
During the competition, participants aim to build systems that define:
- Morphological characteristics of the word (part-of-speech and full tags).
- Lemma of the word.
- Syntactic relations (dependencies).
GramEval Results 🏆.
- qbic - 0.91609 https://github.com/DanAnastasyev/GramEval2020
- ADVance - 0.90762 https://github.com/AlexeySorokin/GramEval2020
- lima - 0.87870 https://github.com/aymara/lima/tree/grameval-2020
- vocative - 0.85198 https://github.com/Koziev/rupostagger2 https://github.com/Koziev/rulemma
- baseline - 0.80377 https://github.com/dialogue-evaluation/GramEval2020/tree/master/baseline
We offer the participants to try to build systems that implement complete morphological and syntactic markup with lemmatization within the framework of Universal Dependencies.
GramEval-2020: Evaluation
The cumulitive evaluation consists of:
- POS tagging accuracy.
- Morphological features accuracy.
- LAS accuracy (syntax).
- Lemmatization accuracy.
The testing procedure will include tests on “golden” texts in many genres and from different sources in Russian. We welcome systems that steadily process all the variety of texts in the Russian language (including texts that differ in style, scope and genre, region, time of creation).
GramEval-2020: Rules
Important Dates
- February 1, 2020 - the release of gold and additional "dirty" training data obtained using automatic marking
- February 24, 2020 - final submission
- March 5, 2020 - announcement of the results
We are open to questions about data, metrics, and testing procedures.
grameval2020@gmail.com
telegram
Public Test
Start: Feb. 1, 2020, midnight
Private Test
Start: Feb. 23, 2020, 9 p.m.
Competition Ends
Feb. 25, 2020, 11:59 p.m.
You must be logged in to participate in competitions.
Sign In| # | Username | Score |
|---|---|---|
| 1 | qbic | 0.91609 |
| 2 | AlexeySorokin | 0.90762 |
| 3 | Shnurre | 0.90411 |