EPIC-Kitchens Action Anticipation

Organized by antoninofurnari - Current server time: Sept. 24, 2018, 11:08 a.m. UTC

Current

ECCV 2018 Action Anticipation Challenge
Aug. 1, 2018, midnight UTC

End

Competition Ends
Never

EPIC-Kitchens 2018 Action Anticipation Challenge

Welcome to the EPIC-Kitchens 2018 Action Anticipation challenge.

Description

The challenge requires the anticipation of a future action from the observation of a preceding video segment. The challenge will be carried out on the EPIC-KITCHENS 2018 dataset of egocentric videos. More information on the dataset & downloads can be found at http://epic-kitchens.github.io/.

Goal

Let Ta be the "anticipation time", i.e. how far in advance to anticipate the action, and To be the "observation time", i.e. the length of the observed video segment preceding the action. Given an action video segment Ai = [tsi, tei], the goal is to predict the action of Ai by observing the video segment preceding the action start time tsi by Ta, that is [tsi-(Ta+To),tsi-Ta]. The anticipation time Ta is set to Ta = 1 second for this challenge. Participants are allowed to set the observation time To to whatever they find convenient. Please mind that the developed algorithms are not allowed to observe any visual content temporally located after time tsi-Ta.

For further details about the challenge, please see Sec. 4.4 of [1].

This challenge is part of the ECCV 2018 workshop.

References

[1] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro and T. Perrett, W. Price, M. Wray, Scaling Egocentric Vision: The EPIC-KITCHENS Dataset, In European Conference on Computer Vision 2018.

Evaluation Criteria

Submissions are evaluated across 2 test sets (please see Sec. 4 of [1] for more details):

  • Seen kitchens (kitchens that have action segments in the training set)
  • Unseen kitchens (kitchens that have no action segments in the training set)

We evaluate model performance across two sets of metrics:

  • Aggregate

    These metrics are micro-averaged giving weight to each class proportionately to their frequency in the test set under evaluation. The considered aggregate metrics are:

    • Top-1 accuracy
    • Top-5 accuracy
  • Per-class

    These metrics are macro-averaged giving equal weight to all classes regardless of their prevalence. We compute these for many-shot classes only: these are verb/noun classes that appear more than 100 times in training, and in the case of actions the cross product between the many shot verb and many shot noun classes given that the action appears at least once in training.

    Many shot classes can be found on github for verbs, nouns, and actions . The considered per-class metrics are:

    • Precision
    • Recall

Please refer to Sec. 4.2 of [1] for more details on the definition of the metrics.

[1] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro and T. Perrett, W. Price, M. Wray, Scaling Egocentric Vision: The EPIC-KITCHENS Dataset, In European Conference on Computer Vision 2018.

Terms and Conditions

  • You agree to us storing your submission results for evaluation purposes.
  • You agree that if you place in the top-10 at the end of the challenge you will submit your code so that we can check for cheating.
  • You agree not to distribute the EPIC-Kitchens dataset without prior written permission.

Submissions

To submit your results to the leaderboard you must construct a submission zip file containing two files:

  • seen.json - Model inference on the seen kitchens test set (S1)
  • unseen.json - Model inference on the unseen kitchens test set (S2)

Both of these files follow the same format detailed below:

JSON Submission Format

The JSON submission format is composed of a single JSON object containing entries for every action in the test set. Specifically, the JSON file should contain:

  • a 'version' property, set to '0.1' (the only supported version so far);
  • a 'challenge' property, which can assume the following values, depending on the challenge: ['action_recognition', 'action_anticipation'];
  • a 'results' object containing entries for every action in the test set (e.g. '1924' is the first action ID in the seen test set).

Each action segment entry is a nested object composed of two entries: 'verb', specifying the class score for every verb class and the other, 'noun' specifying the score for every noun class. Action scores are automatically computed by applying softmax to the verb and noun scores and computing the probability of each possible action.

{
  "version": "0.1",
  "challenge": "action_recognition",
  "results": {
    "1924": {
      "verb": {
        "0": 1.223,
        "1": 4.278,
        ...
        "124": 0.023
      },
      "noun": {
        "0": 0.804,
        "1": 1.870,
        ...
        "351": 0.023
      }
    },
    "1925": { ... },
    ...
  }
}

If you wish to compute your own action scores, you can augment each segment submission with exactly 100 action scores with the key 'action'

{
  "version": "0.1",
  "challenge": "action_recognition",
  "results": {
    "1924": {
      "verb": {
        "0": 1.223,
        "1": 4.278,
        ...
        "124": 0.023
      },
      "noun": {
        "0": 0.804,
        "1": 1.870,
        ...
        "351": 0.023
      },
      "action": {
        "0,1": 1.083,
        ...
        "124,351": 0.002
      }
    },
    "1925": { ... },
    ...
  }
}

The keys of the action object are of the form <verb_class>,<noun_class>.

You can provide scores in any float format that numpy is capable of reading (i.e. you do not need to stick to 3 decimal places).

If you fail to provide your own action scores we will compute them by

  1. Obtaining softmax probabilites from your verb and noun scores
  2. Find the top 100 action probabilities where p(a = (v, n)) = p(v) * p(n)

Submission archive

To upload your results to CodaLab you have to zip both files into a flat zip archive (they can't be inside a folder within the archive).

You can create a flat archive using the command providing the JSON files are in your current directory.

$ zip -j my-submission.zip seen.json unseen.json

ECCV 2018 Action Anticipation Challenge

Start: Aug. 1, 2018, midnight

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In