Evaluation Criteria

Submissions are evaluated on the test set for action retrieval. We report two evaluation metrics: mean Average Precision (mAP) and normalised Discounted Cumulative Gain (nDCG). For each metric there are three different tests performed:

• vis2txt: The score when searching with the visual modality to retrieve in the textual modality.
• txt2vis: The score when searching with the textual modality to retrieve from the visual modality.
• AVG.: The average of the above two scores

Terms and Conditions

• You agree to us storing your submission results for evaluation purposes.
• You agree that if you place in the top-10 at the end of the challenge you will submit your code so that we can verify that you have not cheated.
• You agree not to distribute the EPIC-KITCHENS-100 dataset without prior written permission.

Submissions

To submit your results to the leaderboard you must construct a submission zip file containing a single file test.pkl containing the model’s results on the test set. This file should follow format detailed in the subsequent section.

Pickle Submission Format

Pickle files are serialised python objects. They are used within this challenged due to the size of the submission files. More information about pickle can be found here.

The pickle should be a python dictionary with the following keys:

• 'version': The version of the challenge, currently 0.1
• 'challenge': The type of challenge, multi_instance_retrieval
• 'sim_mat': A 9668x3842 matrix (M) with scores between 0 and 1 with M[i][j] representing the similarity between the ith video and the jth caption.
• 'vis_ids': A list of video IDs (of size 9668) which represent the order of videos within the 'sim_mat'.
• 'txt_ids': A list of caption IDS (of size 3842) which represent the order of captions within the 'sim_mat'.
• sls_pt: SLS Pretraining level (see Supervision Levels Scale (SLS) page for more details).
• sls_tl: SLS Training Labels level (see Supervision Levels Scale (SLS) page for more details).
• sls_td: SLS Training Data level (see Supervision Levels Scale (SLS) page for more details).

{
  "version": "0.1",
  "challenge": "multi_instance_retrieval",
  "sls_pt": -1,
  "sls_tl": -1,
  "sls_td": -1,
  "sim_mat": array([[0.473, 0.669, 0.232, ..., 0.497, 0.116, 0.027],
                    ...,
                    [0.357, 0.222, 0.301, ..., 0.014, 0.143, 0.019]], dtype=float32),
  "txt_ids": array(['P01_11_0', 'P01_11_1', ..., 'P08_15_47', 'P01_14_59']),
  "vis_ids": array(['P01_11_0', 'P01_11_1', ..., 'P32_10_42', 'P32_10_42'])
}

sim_mat must be a 2D iterable python object, it is recommended that this is a numpy array (details about numpy can be found here). It is also recommended that txt_ids and vis_ids are also numpy arrays.

Submission archive

To upload your results to CodaLab you have to zip the test file into a flat zip archive (it can’t be inside a folder within the archive).

You can create a flat archive using the command providing the JSON file is in your current directory.

$ zip -j my-submission.zip test.pkl