Competition

The submissions are evaluated using classification accuracy, which is #(correct predicted QAs) / #(all QAs)

Submission format

A valid submission file is a .zip file containing the following 3 json files (no additional enclosing folder):

• prediction_val.json: model predictions for each question in validation set
• prediction_test_public.json: model predictions for each question in test_public set
• meta.json: description of the submission

prediction_val.json and prediction_test_public.json are organized as {QID: ANSWER_IDX, ...}, ANSWER_IDX is an integer in the range [0, 4]. For example:

  {
    "1108": 2,
    "1006": 0,
    ...
  }

meta.json file contains the following entries：

Key	Type	Description
model_name	str	Name of you model, which will be shown in the leaderboard
is_ensemble	bool	false for single model, true for ensemble
with_ts	bool	Is timestamp annotation used?
show_on_leaderboard	bool	Do you want to show your results on TVQA leaderboard?
author	str	Name of the author(s), separated by comma
institution	str	Name of your institution(s)
description	str	Brief description of your model
paper_link	str	link to your paper
code_link	str	link to your code

Example:

  {
    "model_name": "multi-stream model",
    "is_ensemble": false,
    "with_ts": false,
    "show_on_leaderboard": true,
    "author": "Jie Lei, Licheng Yu, Mohit Bansal, Tamara L. Berg",
    "institution": "UNC Chapel Hill",
    "description": "We introduce a multi-stream end-to-end trainable neural network ...",
    "paper_link": "https://arxiv.org/abs/1809.01696",
    "code_link": "https://github.com/jayleicn/TVQA"
  }

We suggest using online json editors or validators, such as JSONLint to validate your json files before submitting.

Note

• Registration is needed to participate in this challenge, please allow at least 3 days for us to approve your registration. We advise you to register early in case missing possible deadlines. There is no need to send us emails, the system will notify us after your registration.
• Number of submission to the server is limited to 5 per user to prevent overfitting. It is not acceptable to create multiple accounts for a single project to circumvent this limit. However, if your group has multiple papers describing unrelated methods, you are allowed to submit results from all of them.

This page enumerated the terms and conditions of the competition.