Russian sQuAD (sberQuAD)

Organized by alenusch - Current server time: Feb. 19, 2025, 2:32 p.m. UTC

Current

Evaluation phase
Feb. 9, 2021, midnight UTC

End

Competition Ends
Never

Russian QuAD (sberQuAD) challenge

SberQuAD is one of the basic reading comprehension datasets for Russian language. However after Sberbank Data Science Journey 2017 materials were not available officially in open source. The quality of the dataset have also been criticized a lot for example in the paper of the same name. Now, we publish an original sberQuAD dataset with a number of improvements:

  • delete multi-hop sentences from train/test examples
  • fix most of the typos and errors in the data
  • add in training set extra data for you
We release the data in endless competition mode. We are glad to make this dataset useful and always available for the scientific community. In this competition you can test your models and systems to solve updated QuAD. We provide some baselines:
  1. results of the RuBert model by Deeppavlov team's system for SQuAD.
  2. results of the simple max match baseline.
Do your best, compete with models of other participants on updated dataset. Good luck!

Evaluation Criteria

SberQuAD (like in origin competition) evaluation employs two metrics to assess model performance:

  1. the percentage of system’s answers that exactly match (EM) any of the gold standard answers
  2. the maximum overlap between the system response and ground truth answer at the token level expressed via F1 (averaged over all questions)
* Metric scripts apply preprocessing and ignore punctuation and capitalization.

Terms and Conditions

The competition is realized in the endless mode. We provide data for training and offer a private testset. You can score your solution and make public submit on the leaderboard. We don't require obligatory the systems code, but you are free to share your results and systems. This dataset is distributed under the CC BY-SA 4.0 license.

Evaluation phase

Start: Feb. 9, 2021, midnight

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In
# Username Score
1 antivist 0.87
2 tim_atnashev 0.87
3 pokld 0.87