VQA Real Image Challenge (Open-Ended)

Organized by vqateam - Current server time: Jan. 20, 2017, 5:38 a.m. UTC

Current

Real test2015 (oe)
Oct. 21, 2015, midnight UTC

Next

Real test2015 (oe)
Oct. 21, 2015, midnight UTC

Visual Question Answering (VQA)

Recent progress in computer vision and natural language processing has demonstrated that lower-level tasks are much closer to being solved. We believe that the time is ripe to pursue higher-level tasks, one of which is Visual Question Answering (VQA), where the goal is to be able to understand the semantics of scenes well enough to be able to answer open-ended, free-form natural language questions (asked by humans) about images.

To promote and measure progress in this area, we have carefully created the VQA dataset of questions and answers about real images and abstract scenes. We also offer two versions of the task: open-ended and multiple-choice. Open-ended requires a system to produce a natural language answer, while multiple-choice only requires a system to pick an option out of the provided answers. This dataset partially builds on top of the recent Microsoft Common Objects in COntext (MSCOCO) dataset by using its images as part of the VQA dataset. In addition, it also builds a VQA dataset on top of a new collection of 50,000 abstract scenes (see the paper for more details).

To participate in one of the challenges, you can find instructions on the VQA website. In particular, please see the overview, download, evaluation, and challenge pages for more details. We also provide dataset visualization and browser pages to give everyone a sense of the dataset contents.

Evaluation Criteria

The challenge evaluation procedure and code is described on the VQA Evaluation page.

Please refer to the VQA Terms of Use.

Real test-dev2015 (oe)

Start: Oct. 6, 2015, midnight

Description: This phase evaluates algorithms on the VQA Open-Ended task for the dataset built on top of MSCOCO test-dev2015 real images for algorithm experimentation (see website for guidelines). Please do not try to make your results public; doing so will just fill the table with 0s. Evaluation (i.e., 'Running' status) takes about 2 minutes. ***REMINDER*** Your full test set submissions will be entered in our challenge if you either submit to the phase marked with 'Challenge' (Please do not try to make these results public; doing so will just fill the table with 0s.) OR you make your submission to the non-challenge phase public (i.e., on the leaderboard) before the challenge ends.

Real test2015 (oe)

Start: Oct. 21, 2015, midnight

Description: This phase evaluates algorithms on the VQA Open-Ended task for the dataset built on top of MSCOCO test2015 real images. Evaluation (i.e., 'Running' status) takes about 10 minutes. ***REMINDER*** Your full test set submissions will be entered in our challenge if you either submit to the phase marked with 'Challenge' (Please do not try to make these results public; doing so will just fill the table with 0s.) OR you make your submission to the non-challenge phase public (i.e., on the leaderboard) before the challenge ends.

Real Challenge test2015 (oe)

Start: Oct. 21, 2015, midnight

Description: This phase evaluates algorithms on the VQA Open-Ended task for the dataset built on top of MSCOCO test2015 real images for the explicit challenge. Evaluation (i.e., 'Running' status) takes about 10 minutes. ***REMINDER*** Your full test set submissions will be entered in our challenge if you either submit to the phase marked with 'Challenge' (Please do not try to make these results public; doing so will just fill the table with 0s.) OR you make your submission to the non-challenge phase public (i.e., on the leaderboard) before the challenge ends.

Competition Ends

June 5, 2016, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In