GramEval-2020

Organized by Rybolos - Current server time: Feb. 24, 2020, 9:40 p.m. UTC

Previous

Public Test
Feb. 1, 2020, midnight UTC

Current

Private Test
Feb. 23, 2020, 9 p.m. UTC

End

Competition Ends
Feb. 25, 2020, 11:59 p.m. UTC

GramEval-2020

We invite you to participate in the GramEval-2020 competition.

We believe that multi-level language structures need to be labelled together, otherwise errors in one tag level will lead to errors in the following. Existing pipelines “tokenization - morphology - lemmatization - syntax” accumulate errors at each stage.
During the competition, participants aim to build systems that define:

  • Morphological characteristics of the word (part-of-speech and full tags).
  • Lemma of the word.
  • Syntactic relations (dependencies).

We offer the participants to try to build systems that implement complete morphological and syntactic markup with lemmatization within the framework of Universal Dependencies.

 

 

GramEval-2020: Evaluation

The cumulitive evaluation consists of:

  • POS tagging accuracy.
  • Morphological features accuracy.
  • LAS accuracy (syntax).
  • Lemmatization accuracy.

The testing procedure will include tests on “golden” texts in many genres and from different sources in Russian. We welcome systems that steadily process all the variety of texts in the Russian language (including texts that differ in style, scope and genre, region, time of creation).

GramEval-2020: Rules

Important Dates

  • February 1, 2020 - the release of gold and additional "dirty" training data obtained using automatic marking
  • February 24, 2020 - final submission
  • March 5, 2020 - announcement of the results

We are open to questions about data, metrics, and testing procedures.
grameval2020@gmail.com
telegram

Public Test

Start: Feb. 1, 2020, midnight

Private Test

Start: Feb. 23, 2020, 9 p.m.

Competition Ends

Feb. 25, 2020, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In
# Username Score
1 qbic 0.91296
2 AlexeySorokin 0.89081
3 vocative 0.85198