DSTC 8: End-to-End Multi-Domain Dialog Challenge Track - Task 1

Organized by convlab - Current server time: Aug. 25, 2019, 4:10 p.m. UTC

Previous

Test Submissions
June 17, 2019, midnight UTC

Current

Final Submissions
June 17, 2019, midnight UTC

End

Competition Ends
Oct. 6, 2019, 11:59 p.m. UTC

Welcome to DSTC 8: End-to-End Multi-Domain Dialog Challenge!

Quick links

News

  • [2019-06-17] Task description and data are released.
  • [2019-07-15] Codalab competition back online. Due to a major outage in the Codalab platform, participants who registered before July 12, 2019 must re-register.

Task Description

In this task, participants will build an end-to-end multi-domain dialog system for tourist information desk settings based on the recently released MultiWOZ dataset which we enrich with further annotation to support a wider range of learning approaches. The submitted dialog system will be evaluated with both system evaluation and human evaluation. 

Schedule

Multi-domain Task Completion

  • June 3, 2019: Registration opens.
  • June 17, 2019: Training data is released, and development begins.
  • September 23, 2019 - September 29, 2019: users can make submissions to verify whether the system is errorless.  
  • October 6, 2019: Entry submission deadline.
  • October 14, 2019: Objective evaluation results are released. 
  • October 21, 2019: Human evaluation results are released. 
  • TBA: DSTC8 paper submission deadline.
  • TBA: DSTC8 workshop. 

Organizers

You can contact all the contest organizers at dstc8-task1@service.microsoft.com

The organizers are listed below:

  • Sungjin Lee (junion.sjlee@gmail.com)
  • Qi Zhu (zhu-q18@mails.tsinghua.edu.cn)
  • RyuichiTakanobu (gxly15@mails.tsinghua.edu.cn)
  • Yaoqin Zhang  (zhangyq17@mails.tsinghua.edu.cn)
  • Zheng Zhang (z-zhang15@mails.tsinghua.edu.cn)
  • Jinchao Li    (jincli@microsoft.com)
  • Baolin Peng (Baolin.Peng@microsoft.com)
  • Minlie Huang  (aihuang@tsinghua.edu.cn)
  • Jianfeng Gao (jfgao@microsoft.com)

 

  • Participants are required to use ConvLab as the development toolkit. ConvLab is an open-source multidomain end-to-end dialog system platform, that enables researchers to quickly set up experiments with reusable components and compare a large set of different approaches, ranging from conventional pipeline systems to end-to-end neural models, in common environments. ConvLab offers a set of fully annotated datasets and associated pre-trained reference models.

    ConvLab Code Repository: https://github.com/ConvLab/ConvLab

    ConvLab Paper: https://arxiv.org/pdf/1904.08637.pdf

Submission Evaluation

  1. Extend ConvLab with your code, and include submission.json under the convlab/spec directory.
  2. In submission.json, specify up to 5 specs with the name submission[1-5].
  3. Make sure the code with the config is runnable in the docker environment.
  4. If your code uses external packages beyond the existing docker environment, please choose one of the following two approaches to specify your environment requirements:
    • Add install.sh under the convlab directory. install.sh should include all required extra packages.
    • Create your own Dockerfile with the name dev.dockerfile
  5. Zip the system and submit.

Evaluation Methods

  1. Automatic end2end Evaluation: The submitted system will be evaluated using the user-simulator setting in spec milu_rule_rule_template in convlab/spec/baseline.json. We will use the evaluator MultiWozEvaluator in convlab/evaluator/multiwoz to report metrics including success rate, average reward, number of turms, precision, recall, and F1 score.
  2. Human Evaluation: The submitted system will be evaluated in Amazon Mechanic Turk. Crowd-workers will communicate with your summited system, and provide a rating based on the whole experience (language understanding, appropriateness, etc.)

Test Submissions

Start: June 17, 2019, midnight

Description: Test the format of your submissions and troubleshoot errors here. Note the leaderboard does nothing

Final Submissions

Start: June 17, 2019, midnight

Description: Final model predictions submitted to the competition.

Competition Ends

Oct. 6, 2019, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In