CMRC 2019: The Third Evaluation Workshop on Chinese Machine Reading Comprehension

Organized by CMRC_2019 - Current server time: June 19, 2019, 2:43 p.m. UTC
Reward $6,500

Previous

Trial Phase
May 15, 2019, midnight UTC

Current

Development Phase
June 10, 2019, midnight UTC

Next

Qualifying Phase
Aug. 1, 2019, midnight UTC

[LATEST NEWS] June 10, 2019: Development data has been released, check GitHub

Welcome to CMRC 2019!

For Chinese Websites, please visit: https://hfl-rc.github.io/cmrc2019/

China National Conference on Computational Linguistics (CCL) has been holding evaluation workshops since 2017. As one of the evaluation workshops of CCL, The Third Evaluation Workshop on Chinese Machine Reading Comprehension (CMRC 2019) will be held together with The Eighteenth China National Conference on Computational Linguistics (CCL 2019,October 18, 2019~ October 20, 2019, Kunming, Yunnan) in October, 2019. CMRC 2019 is sponsored by Technical Committee on Computational Linguistics, Chinese Information Processing Society of China (CIPS-CL),organized by Joint Laboratory of HIT and iFLYTEK Research (HFL), crowned by iFLYTEK、Hebei iFLYTEK A.I Research. CMRC 2019 aims to further promote the research and development of Chinese machine reading comprehension and provide a good communication platform for scholars in related fields. CMRC 2019 evaluation committee invites everyone to participate in this evaluation.

Joint Laboratory of HIT and iFLYTEK Research (HFL) has long been devoted to the research and development of machine reading comprehension. In the field of Chinese reading comprehension, HFL released the first Chinese cloze-style reading comprehension dataset PD&CFT In July 2016. In 2017 and 2018, HFL held the evaluation workshops CMRC 2017 and CMRC 2018, which further promote the research on Chinese machine reading comprehension. We hope to promote the Chinese machine reading comprehension together with scholars in related fields by continually holding Chinese machine reading comprehension evaluation workshops.

Prizes

The winners will be awarded both certificates (provided by CIPS-CL) and bouns (provided by iFLYTEK and Hebei iFLYTEK A.I Research)

  • First Prize (1): 20,000 CNY + Certificate
  • Second Prize (1): 10,000 CNY + Certificate
  • Thrid Prize (3): 5,000 CNY + Certificate

Important Dates

All deadlines are Beijing Time (UTC+8) 23:59.

Phase Procedure Status Time
Registration Registration for participation Running Now ~ June 30, 2019 (Registration Due)
Qualifying Evaluation Train/Trial set release Released May 23, 2019
Development set release Released June 10, 2019
System tuning Running Now ~ July 31, 2019
Qualifying set release N/A August 1, 2019 ~ August 7, 2019
Final Evaluation Submit final evaluation system N/A August 14, 2019 ~ August 21, 2019
Post-Evaluation Notifications for final results N/A September, 2019 (TBD)
System description N/A September, 2019 (TBD)
CMRC 2019 evaluation workshop N/A October 19, 2019 (TBD)

Registration for Participation

中文注册地址: https://wj.qq.com/s2/3744098/bf48

For non-Chinese speakers, please send the following information to evaluation official for participation: cmrc2019 [at] 126 [dot] com

  • Your Name
  • Your Team Name
  • Your CodaLab Competition Account
  • Your Affiliation

Organization Committee

Host: Chinese Information Processing Society of China (CIPS)

Organizer: Joint Laboratory of HIT and iFLYTEK (HFL)

Sponsor: iFLYTEK Co., Ltd.

Evaluation Co-Chairs

Ting Liu, Harbin Institute of Technology

Yiming Cui, iFLYTEK Research

Official HFL WeChat Account

Follow Joint Laboratory of HIT and iFLYTEK Research (HFL) on WeChat.

Contact

Please send mails to cmrc2019 [at] 126 [dot] com

Task Description

The task of The Third Evaluation Workshop on Chinese Machine Reading Comprehension (CMRC 2019) is the sentence cloze-Style machine reading comprehension (SC-MRC). Given a narrative passage and some sentences extract from the passage, the participators should build models to complete the passage by putting the sentences back into the passages on the right places. This task requires the machine to infer the missing segment by reasoning from the context, increasing the difficulty of machine reading comprehension. The differences between CMRC 2019 and the cloze-style reading comprehension of CMRC 2017 are:

  • The missing segment is a sentence rather than a word.
  • Every passage contains multiple missing segments, not only one. The information available to the machine is greatly reduced.
  • There are additional fake options in the candidate choices. The fake options do not fit any missing segment in the passage, which greatly increases the difficulty of the task.

The competition is divided into two phases: the qualifying (May 27 ~ August 7) and the finals (August 14 ~ August 21)

  • Qualifying: no need to submit the source code. Submit the results on qualification set at the end of the phase. The top ten teams will be qualified for the finals.
  • Finals: all the finalists are required to submit their system source code and evaluate their system on the hidden test set to get the final results (80% credits), and also be evaluated by Evaluation Committee (20% credits).

Data Format

  • context: the passage that contains blanks (denoted as [blank])
  • context_id: the passage id (unique)
  • choices: the candidate sentences that need to fill in
  • answers: correct order of the sentence (count from 0)

Example (JSON format):

{ "data": [ { "context": "森林里有一棵大树,树上有一个鸟窝。[BLANK1],还从来没有看到过鸟宝宝长什么样。 小松鼠说:“我爬到树上去看过,鸟宝宝光溜溜的,身上一根羽毛也没有。” “我不相信,”小白兔说,“所有的鸟都是有羽毛的。” “鸟宝宝没有羽毛。”小松鼠说,“你不信自己去看。” 小白兔不会爬树,它没有办法去看。小白兔说:“我请蓝狐狸去看一看,我相信蓝狐狸的话。” 小松鼠说:“蓝狐狸跟你一样,也不会爬树。” 蓝狐狸说:“我有魔法树叶,我能变成一只狐狸鸟。” [BLANK2],一下子飞到了树顶上。 “蓝狐狸,你看到了吗?”小白兔在树下大声喊。 “我看到了,鸟窝里有四只小鸟,他们真是光溜溜的,一根羽毛也没有。”蓝狐狸说。 就在这时候,鸟妈妈和鸟爸爸回来了,[BLANK3],立刻大喊大叫: “抓强盗啊!抓强盗啊!强盗闯进了我们家里,想偷我们的孩子!” [BLANK4],全都飞了过来。他们扇着翅膀,朝蓝狐狸冲过来,用尖尖的嘴啄他,用爪子抓他。 蓝狐狸扑扇翅膀,赶紧飞。 鸟儿们排着队伍,紧紧追上来。[BLANK5],它飞得不高,也飞得不快。 “救命啊,救命!”蓝狐狸说,“我不是强盗,我是蓝狐狸!” 小白兔在草丛说:“你不是鸟,你飞不过他们,你赶快变回狐狸吧!” 蓝狐狸急忙落到地上,变回了狐狸,躲进深深的草丛里。 鸟儿们找不到蓝狐狸,只得飞走了。 蓝狐狸对小白兔说:“谢谢你。", "choices": [ "蓝狐狸是第一次变成狐狸鸟", "森林里所有的鸟听到喊声", "他们看到鸟窝里蹲着一只蓝色的大鸟", "蓝狐狸真的变成了一只蓝色的大鸟", "小动物们只看到过鸟妈妈和鸟爸爸在鸟窝里飞进飞出" ], "context_id": "SAMPLE_00001", "answers": [4,3,2,1,0] } ] }

Download Datasets

All data should be accessed through: GitHub

Evaluation Metrics

  • Question Accuracy (QAC): calculate question-level accuracy
  • Passage Accuracy (PAC): calculate passage-level accuracy

Final ranking will be judged on the test set, according to QAC (PAC as secondary metric when there is a tie).

Please infer the official evaluation script through GitHub

Baseline System

We provide a simple baseline system which is based on BERT (Chinese, base).

Please infer the official GitHub

How To Submit

中文请参考:提交流程

Please follow the instructions below.

Your MUST register before submitting your results.

中文注册地址: https://wj.qq.com/s2/3744098/bf48

For non-Chinese speakers, please send the following information to evaluation official for participation: cmrc2019 [at] 126 [dot] com

  • Your Name
  • Your Team Name
  • Your CodaLab Competition Account
  • Your Affiliation

Prepare Your Submission File

Prepare your prediction json file in the following format.

  • Key: context_id
  • Value: ordered sentence_id (count from 0)
  • Format: { 'TRIAL_00001': [4,2,1,3,0], 'TRIAL_00002': [2,1,0], ... 'TRIAL_00010': [0,2,1,5,4,3] }

Offline Test

  • Evaluate your prediction file (e.g. pred.json) offline before submit to CodaLab Competition Platform.
  • $ python cmrc2019_evaluate.py cmrc2019_trial.json pred.json

Online Submission

  • Please make a .zip file that ONLY contains your prediction file.
  • Click Participate-> Submit / View Results (select the right phase: trial/dev/qualifying)
  • Fill in your team name, model name
  • Click submit button to upload your zip file

View Results

  • If the status turn into Finished, then your results will be automatically updated in leaderboard
  • If it shows Failed, please debug your submission according to the stderr output
  • Failed submissions will not count towards the submission limit

Sample Submission

We provide a sample submission for trial phase for your reference.

Trial Prediction: GitHub

Troubleshooting

If there is anything wrong with your submission, please send an email to cmrc2019 [at] 126 [dot] com

Alternatively, you can also use CodaLab Competition Forum to seek help from the community.

Organization Committee

Host: Chinese Information Processing Society of China (CIPS)
Organizer: Joint Laboratory of HIT and iFLYTEK (HFL)
Sponsor: iFLYTEK Co., Ltd. and iFLYTEK Research (Hebei)

Evaluation Co-Chairs

Ting Liu, Harbin Institute of Technology
Yiming Cui, iFLYTEK Research

Frequently Asked Questions (FAQ)

When I clicked 'Participate', an email is sent to my mailbox that telling me "permission revoked"?

    • Please register first before submitting results. Check: How To Submit on the left sidebar or 中文注册. When you've done your registration, we will accept your participation.

When I clicked 'Participate', it shows "Your request to participate in this competition was denied or your privileges have been suspended"

    • Please register first before submitting results.

Trial Phase

Start: May 15, 2019, midnight

Description: 试验阶段是为了让参赛者更好的熟悉提交过程,不会计入总成绩。<br/>This phase is to let participants be familiar with the submission procedure. Will not count towards qualification.

Development Phase

Start: June 10, 2019, midnight

Description: 参赛者可以提交开发集上的结果,不会计入总成绩。每个队伍至少要提交一次结果。<br/>Participants are free to submit their development predictions here. Will not count towards qualification. All teams should submit at least one submission.

Qualifying Phase

Start: Aug. 1, 2019, midnight

Description: 资格集排名前十的队伍可以进入到最终的决赛。(决赛队伍需要在CodaLab Worksheet提交代码)<br/>Only Top-10 teams will be admitted to the final test phase. (Qualified teams should submit their codes to CodaLab Worksheets.)

Post Qualifying Phase

Start: Aug. 8, 2019, midnight

Description: 我们暂时关闭这个阶段,将在CMRC 2019结束后开启。<br/>We temporarily close this phase. Will open after CMRC 2019 test phase over.

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In
# Username Score
1 Decalogue 75.762
2 ewrfcas 73.993
3 luckyboys 71.634