The Third Evaluation Workshop on Chinese Machine Reading Comprehension (CMRC 2019)

Organized by CMRC_2019 - Current server time: Dec. 5, 2019, 5:18 p.m. UTC
Reward $6,500

Previous

资格赛 (Qualification Phase)
Aug. 1, 2019, midnight UTC

Current

赛后 (Post Evaluation Phase)
Aug. 7, 2019, 4 p.m. UTC

End

Competition Ends
Never

Welcome to CMRC 2019!

China National Conference on Computational Linguistics (CCL) has been holding evaluation workshops since 2017. As one of the evaluation workshops of CCL, The Third Evaluation Workshop on Chinese Machine Reading Comprehension (CMRC 2019) will be held together with The Eighteenth China National Conference on Computational Linguistics (CCL 2019,October 18, 2019~ October 20, 2019, Kunming, Yunnan) in October, 2019. CMRC 2019 is sponsored by Technical Committee on Computational Linguistics, Chinese Information Processing Society of China (CIPS-CL),organized by Joint Laboratory of HIT and iFLYTEK Research (HFL), crowned by iFLYTEK、Hebei iFLYTEK A.I Research. CMRC 2019 aims to further promote the research and development of Chinese machine reading comprehension and provide a good communication platform for scholars in related fields. CMRC 2019 evaluation committee invites everyone to participate in this evaluation.

Joint Laboratory of HIT and iFLYTEK Research (HFL) has long been devoted to the research and development of machine reading comprehension. In the field of Chinese reading comprehension, HFL released the first Chinese cloze-style reading comprehension dataset PD&CFT In July 2016. In 2017 and 2018, HFL held the evaluation workshops CMRC 2017 and MRC 2018, which further promote the research on Chinese machine reading comprehension. We hope to promote the Chinese machine reading comprehension together with scholars in related fields by continually holding Chinese machine reading comprehension evaluation workshops.

Prizes

The winners will be awarded both certificates (provided by CIPS-CL) and bouns (provided by iFLYTEK and Hebei iFLYTEK A.I Research)

  • First Prize (1): 20,000 CNY + Certificate
  • Second Prize (1): 10,000 CNY + Certificate
  • Thrid Prize (3): 5,000 CNY + Certificate

Important Dates

All deadlines are Beijing Time (UTC+8) 23:59.

Phase Procedure Status Time
Registration Registration for participation Started Now ~ June 30, 2019 (Registration Due)
Qualifying Evaluation Train/Trial set release Started May 23, 2019
Development set release N/A June 10, 2019
System tuning N/A Now ~ July 31, 2019
Qualifying set release N/A August 1, 2019 ~ August 7, 2019
Final Evaluation Submit final evaluation system N/A August 14, 2019 ~ August 21, 2019
Post-Evaluation Notifications for final results N/A September, 2019 (TBD)
System description N/A September, 2019 (TBD)
CMRC 2019 evaluation workshop N/A October 19, 2019 (TBD)

Registration for Participation

Please send the following information to evaluation official.

  • Your Name
  • Your Team Name
  • Your CodaLab Competition Account
  • Your Affiliation

Organization Committee

Host: Chinese Information Processing Society of China (CIPS)

Organizer: Joint Laboratory of HIT and iFLYTEK (HFL)

Sponsor: iFLYTEK Co., Ltd.

Evaluation Co-Chairs

Ting Liu, Harbin Institute of Technology

Yiming Cui, iFLYTEK Research

Official HFL WeChat Account

Follow Joint Laboratory of HIT and iFLYTEK Research (HFL) on WeChat.

Contact

Please send mails to cmrc2019 [at] 126 [dot] com

Task Description

The task of The Third Evaluation Workshop on Chinese Machine Reading Comprehension (CMRC 2019) is the sentence cloze-Style machine reading comprehension (SC-MRC). Given a narrative passage and some sentences extract from the passage, the participators should build models to complete the passage by putting the sentences back into the passages on the right places. This task requires the machine to infer the missing segment by reasoning from the context, increasing the difficulty of machine reading comprehension. The differences between CMRC 2019 and the cloze-style reading comprehension of CMRC 2017 are:

  • The missing segment is a sentence rather than a word.
  • Every passage contains multiple missing segments, not only one. The information available to the machine is greatly reduced.
  • There are additional fake options In the final test set. The fake options do not fit any missing segment in the passage, which greatly increases the difficulty of the task.

Attention: All public sets (trail set, training set, development set, qualification set) contain **NO** fake options.

The competition is divided into two phases: the qualifying (May 27 ~ August 7) and the finals (August 14 ~ August 21)

  • Qualifying: no need to submit the source code. Submit the results on qualification set at the end of the phase. The top ten teams will be qualified for the finals.
  • Finals: all the finalists are required to submit their system source code and evaluate their system on the hidden test set to get the final results (80% credits), and also be evaluated by Evaluation Committee (20% credits).

Data Format

  • context: the passage that contains blanks (denoted as [blank])
  • context_id: the passage id (unique)
  • choices: the candidate sentences that need to fill in
  • answers: correct order of the sentence (count from 0)

Example (JSON format):

{ "data": [ { "context": "森林里有一棵大树,树上有一个鸟窝。[BLANK1],还从来没有看到过鸟宝宝长什么样。 小松鼠说:“我爬到树上去看过,鸟宝宝光溜溜的,身上一根羽毛也没有。” “我不相信,”小白兔说,“所有的鸟都是有羽毛的。” “鸟宝宝没有羽毛。”小松鼠说,“你不信自己去看。” 小白兔不会爬树,它没有办法去看。小白兔说:“我请蓝狐狸去看一看,我相信蓝狐狸的话。” 小松鼠说:“蓝狐狸跟你一样,也不会爬树。” 蓝狐狸说:“我有魔法树叶,我能变成一只狐狸鸟。” [BLANK2],一下子飞到了树顶上。 “蓝狐狸,你看到了吗?”小白兔在树下大声喊。 “我看到了,鸟窝里有四只小鸟,他们真是光溜溜的,一根羽毛也没有。”蓝狐狸说。 就在这时候,鸟妈妈和鸟爸爸回来了,[BLANK3],立刻大喊大叫: “抓强盗啊!抓强盗啊!强盗闯进了我们家里,想偷我们的孩子!” [BLANK4],全都飞了过来。他们扇着翅膀,朝蓝狐狸冲过来,用尖尖的嘴啄他,用爪子抓他。 蓝狐狸扑扇翅膀,赶紧飞。 鸟儿们排着队伍,紧紧追上来。[BLANK5],它飞得不高,也飞得不快。 “救命啊,救命!”蓝狐狸说,“我不是强盗,我是蓝狐狸!” 小白兔在草丛说:“你不是鸟,你飞不过他们,你赶快变回狐狸吧!” 蓝狐狸急忙落到地上,变回了狐狸,躲进深深的草丛里。 鸟儿们找不到蓝狐狸,只得飞走了。 蓝狐狸对小白兔说:“谢谢你。", "choices": [ "蓝狐狸是第一次变成狐狸鸟", "森林里所有的鸟听到喊声", "他们看到鸟窝里蹲着一只蓝色的大鸟", "蓝狐狸真的变成了一只蓝色的大鸟", "小动物们只看到过鸟妈妈和鸟爸爸在鸟窝里飞进飞出" ], "context_id": "SAMPLE_00001", "answers": [4,3,2,1,0] } ] }

Download Datasets

All data should be accessed through: GitHub

Evaluation Metrics

  • Question Accuracy (QAC): calculate question-level accuracy
  • Passage Accuracy (PAC): calculate passage-level accuracy

Final ranking will be judged on the test set, according to QAC (PAC as secondary metric when there is a tie).

Please infer the official evaluation script through GitHub

Baseline System

We provide a simple baseline system which is based on BERT (Chinese, base).

Please infer the official GitHub

How To Submit

中文请参考:提交流程

Please follow the instructions below.

Prepare Your Submission File

Prepare your prediction json file in the following format.

  • Key: context_id
  • Value: ordered sentence_id (count from 0)
  • Format: { 'TRIAL_00001': [4,2,1,3,0], 'TRIAL_00002': [2,1,0], ... 'TRIAL_00010': [0,2,1,5,4,3] }

Offline Test

  • Evaluate your prediction file (e.g. pred.json) offline before submit to CodaLab Competition Platform.
  • $ python cmrc2019_evaluate.py cmrc2019_trial.json pred.json

Online Submission

  • Please make a .zip file that ONLY contains your prediction file.
  • Click Participate-> Submit / View Results (select the right phase: trial/dev/qualifying)
  • Fill in your team name, model name
  • Click submit button to upload your zip file

View Results

  • If the status turn into Finished, then your results will be automatically updated in leaderboard
  • If it shows Failed, please debug your submission according to the stderr output
  • Failed submissions will not count towards the submission limit

Sample Submission

We provide a sample submission for trial phase for your reference.

Trial Prediction: GitHub

Troubleshooting

If there is anything wrong with your submission, please send an email to cmrc2019 [at] 126 [dot] com

Alternatively, you can also use CodaLab Competition Forum to seek help from the community.

试验阶段 (Trial Phase)

Start: June 30, 2013, midnight

开发阶段 (Development Phase)

Start: June 10, 2019, midnight

Description: Please submit DEVELOPMENT set result. 请提交开发集结果。

资格赛 (Qualification Phase)

Start: Aug. 1, 2019, midnight

Description: Please submit QUALIFYING set result. 请提交资格集结果,前10名(不含官方)将进入决赛。

赛后 (Post Evaluation Phase)

Start: Aug. 7, 2019, 4 p.m.

Description: Please submit QUALIFYING set result. 请提交资格集结果。(该阶段成绩不计入本次评测)

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In
# Username Score
1 ewrfcas 89.70675
2 RichardRui 87.62055
3 sixEstates 83.84176