The Large Scale Movie Description Challenge (LSMDC) v2: Multi-Sentence Description

Organized by arohrbach - Current server time: Oct. 22, 2019, 6 a.m. UTC

First phase

Public Test
Aug. 1, 2019, midnight UTC

End

Competition Ends
Oct. 1, 2019, 11:59 p.m. UTC

Overview

Automatically describing open-domain videos using rich natural sentences is among the most challenging tasks of computer vision, natural language processing and machine learning. This year we introduce Large Scale Movie Description Challenge v2 (LSMDCv2), aiming at a more realistic and practical setting of multi-sentence movie description generation. Specifically, movie descriptions are evaluated on sets of 5 clips.

The challenge consists of two phases: public test set evaluation and blind (where we will not provide the sentence descriptions) test set evaluation. The evaluation is performed on sets of 5 clips, i.e. reference and predicted descriptions are grouped for each 5 consecutive clips. We also report results evaluated per individual clip, for completeness.

Participation

To participate, you should first create an account on CodaLab. In order to submit your results, please, perform these steps:

      • To officially take part in a challenge you have to submit your results on both, public and blind test sets.
      • Convert your generated descriptions into the following JSON format:
        [
        {
        "video_id": int,
        "caption": str,
        },
        ...
        ]
        where "video_id" are integer numbers starting with 1.
      • Name your JSON file publictest_[your_algorithm_name]_results.json or blindtest_[your_algorithm_name]_results.json, depending on a challenge phase, zip it in an archive.
      • Go to "Participate" tab, click "Submit / View Results" and select the respective challenge phase.
      • Fill in the form (specify any external training data used by your algorithm in the "Method description" field) and upload your ZIP archive.
      • Click "Refresh Status" to see how your submission is being processed. In case of errors, please, check and correct your submission.
      • Once the submission is successfully processed, you can view your scores via "View scoring output log" and click "Post to leaderboard" to make your results publicly available. You can  access the detailed evaluation output via "Download evaluation output from scoring step".

Note, that we allow up to 5 submissions per day / 100 in total for the public test phase and 1 submision per day / 5 in total for the blind test phase.

Baseline

We provide a baseline code for generating movie descriptions with "SOMEONE"-s: https://github.com/jamespark3922/lsmdc-baseline

Acknowledgement

We thank the "Microsoft COCO Image Captioning Challenge" organizers for sharing the evaluation code.

The MS COCO Caption Evaluation API is used to evaluate results. The software uses both candidate and reference captions, applies sentence tokenization, and output several performance metrics including BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, METEOR and CIDEr-D. More details can be found in the paper Microsoft COCO Captions: Data Collection and Evaluation Server.

Public Test

Start: Aug. 1, 2019, midnight

Description: Public Test set

Blind Test

Start: Aug. 1, 2019, midnight

Description: Blind Test set

Competition Ends

Oct. 1, 2019, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In