Note, this portal is only used for models that did not use ground-truth 'ts' at inference.
TVQA is a large-scale video QA dataset based on 6 popular TV shows (Friends, The Big Bang Theory, How I Met Your Mother, House M.D., Grey's Anatomy, Castle). It consists of 152.5K QA pairs from 21.8K video clips, spanning over 460 hours of video. The questions are designed to be compositional, requiring systems to jointly localize relevant moments within a clip, comprehend subtitles-based dialogue, and recognize relevant visual concepts.
Jie Lei, Licheng Yu, Mohit Bansal, Tamara L. Berg
UNC Chapel Hill
Following a major crash of Codalab (in July 2019), some user data could not restored.
Send emails to faq-tvqa-unc@googlegroups.com
The submissions are evaluated using classification accuracy, which is #(correct predicted QAs) / #(all QAs)
A valid submission file is a .zip
file containing the following 3 json files (no additional enclosing folder):
prediction_val.json
: model predictions for each question in validation setprediction_test_public.json
: model predictions for each question in test_public setmeta.json
: description of the submissionprediction_val.json
and prediction_test_public.json
are organized as {QID: ANSWER_IDX, ...}, ANSWER_IDX is an integer in the range [0, 4]
. For example:
{ "1108": 2, "1006": 0, ... }
meta.json
file contains the following entries:
Key | Type | Description |
---|---|---|
model_name | str | Name of you model, which will be shown in the leaderboard |
is_ensemble | bool | false for single model, true for ensemble |
with_ts | bool | Is timestamp annotation used? |
show_on_leaderboard | bool | Do you want to show your results on TVQA leaderboard? |
author | str | Name of the author(s), separated by comma |
institution | str | Name of your institution(s) |
description | str | Brief description of your model |
paper_link | str | link to your paper |
code_link | str | link to your code |
Example:
{ "model_name": "multi-stream model", "is_ensemble": false, "with_ts": false, "show_on_leaderboard": true, "author": "Jie Lei, Licheng Yu, Mohit Bansal, Tamara L. Berg", "institution": "UNC Chapel Hill", "description": "We introduce a multi-stream end-to-end trainable neural network ...", "paper_link": "https://arxiv.org/abs/1809.01696", "code_link": "https://github.com/jayleicn/TVQA" }
We suggest using online json editors or validators, such as JSONLint to validate your json files before submitting.
This page enumerated the terms and conditions of the competition.
Start: Nov. 16, 2018, midnight
Description: val and test_public evaluation for models that did not use ground-truth 'ts' at inference
Never
You must be logged in to participate in competitions.
Sign In