EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge

Organized by m_wray - Current server time: Nov. 30, 2020, 5:56 p.m. UTC


CVPR 2021 Challenge
Aug. 5, 2020, midnight UTC


Competition Ends
May 28, 2021, midnight UTC

EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge

Welcome to the EPIC-KITCHENS-100 Multi-Instance Retrieval challenge.

EPIC-KITCHENS-100 is an unscripted egocentric action dataset collected from 45 kitchens from 4 cities across the world.

Dataset details

  • 100 hours of video
  • 20M frames
  • Full HD
  • 90k action segments
  • 20k unique narrations
  • 97 verb classes, 300 noun classes


Perform cross-modal retrieval by searching between vision and text modalities.

Evaluation Criteria

Submissions are evaluated on the test set for action retrieval. We report two evaluation metrics: mean Average Precision (mAP) and normalised Discounted Cumulative Gain (nDCG). For each metric there are three different tests performed:

  • vis2txt: The score when searching with the visual modality to retrieve in the textual modality.
  • txt2vis: The score when searching with the textual modality to retrieve from the visual modality.
  • AVG.: The average of the above two scores

Terms and Conditions

  • You agree to us storing your submission results for evaluation purposes.
  • You agree that if you place in the top-10 at the end of the challenge you will submit your code so that we can verify that you have not cheated.
  • You agree not to distribute the EPIC-KITCHENS-100 dataset without prior written permission.


To submit your results to the leaderboard you must construct a submission zip file containing a single file test.pkl containing the model’s results on the test set. This file should follow format detailed in the subsequent section.

Pickle Submission Format

Pickle files are serialised python objects. They are used within this challenged due to the size of the submission files. More information about pickle can be found here.

The pickle should be a python dictionary with the following keys:

  • 'version': The version of the challenge, currently 0.1
  • 'challenge': The type of challenge, multi_instance_retrieval
  • 'sim_mat': A 9668x3842 matrix (M) with scores between 0 and 1 with M[i][j] representing the similarity between the ith video and the jth caption.
  • 'vis_ids': A list of video IDs (of size 9668) which represent the order of videos within the 'sim_mat'.
  • 'txt_ids': A list of caption IDS (of size 3842) which represent the order of captions within the 'sim_mat'.
  • sls_pt: SLS Pretraining level (see Supervision Levels Scale (SLS) page for more details).
  • sls_tl: SLS Training Labels level (see Supervision Levels Scale (SLS) page for more details).
  • sls_td: SLS Training Data level (see Supervision Levels Scale (SLS) page for more details).
  "version": "0.1",
  "challenge": "multi_instance_retrieval",
  "sls_pt": -1,
  "sls_tl": -1,
  "sls_td": -1,
  "sim_mat": array([[0.473, 0.669, 0.232, ..., 0.497, 0.116, 0.027],
                    [0.357, 0.222, 0.301, ..., 0.014, 0.143, 0.019]], dtype=float32),
  "txt_ids": array(['P01_11_0', 'P01_11_1', ..., 'P08_15_47', 'P01_14_59']),
  "vis_ids": array(['P01_11_0', 'P01_11_1', ..., 'P32_10_42', 'P32_10_42'])

sim_mat must be a 2D iterable python object, it is recommended that this is a numpy array (details about numpy can be found here). It is also recommended that txt_ids and vis_ids are also numpy arrays.

Submission archive

To upload your results to CodaLab you have to zip the test file into a flat zip archive (it can’t be inside a folder within the archive).

You can create a flat archive using the command providing the JSON file is in your current directory.

$ zip -j my-submission.zip test.pkl

CVPR 2021 Challenge

Start: Aug. 5, 2020, midnight

Description: CVPR 2021 Multi-Instance Retrieval Challenge

Competition Ends

May 28, 2021, midnight

You must be logged in to participate in competitions.

Sign In