Submissions are evaluated on the test set. We report mean Average Precision (mAP) for verbs, nouns and actions at different IoU thresholds (0.1, 0.2, 0.3, 0.4, 0.5) and average mAP across IoU thresholds on the overall test set:
We consider mAP as implemented in [1]. Methods are ranked by average action mAP.
To submit your results to the leaderboard you must construct a submission zip file containing a single file test.json
containing the model’s results on the test set. This file should follow format detailed in the subsequent section.
The JSON submission format is composed of a single JSON object containing entries for every detected action in all the videos of the test set. Specifically, the JSON file should contain:
'version'
property, set to '0.2'
'challenge'
property, set to 'action_detection'
sls
properties (see the Supervision Levels Scale (SLS) page for more details):
sls_pt
: SLS Pretraining level.sls_tl
: SLS Training Labels level.sls_td
: SLS Training Data level.'results'
object containing entries for every video in the test set (e.g . 'P01_101'
).Each video entry is a list of objects describing each detected action. Each of these objects should contain:
'verb'
property, reporting the detected verb class (e.g., 1).'noun'
property, reporting the detected noun class (e.g., 34).'action'
property, reporting the detected action class in the format '<verb_class>,<noun_class>'
(e.g., ‘1,34’). The predicted action may differ from the pair of predicted verb and noun classes.'score'
property, reporting the confidence score of the prediction (e.g., 0.78).'segment'
property, which is a list containing the starting and ending timestamps of the detected action in seconds (e.g., [6.13, 9.20]).{
"version": "0.2",
"challenge": "action_detection",
"sls_pt": -1,
"sls_tl": -1,
"sls_td": -1,
"results": {
"P26_122": [
{
"verb": 0,
"noun": 16,
"action": "0,16",
"score": 0.7398802638053894,
"segment": [
25.223000000000004,
32.86633333333334
]
},
...
{
"verb": 0,
"noun": 58,
"action": "0,58",
"score": 0.0001102862200564619,
"segment": [
292.73966666666669,
300.38300000000006
]
}
],
...
"P36_102": [
{
"verb": 9,
"noun": 27,
"action": "9,27",
"score": 0.8049795031547546,
"segment": [
599.3945,
613.28775
]
},
...
{
"verb": 17,
"noun": 65,
"action": "17,65",
"score": 0.0006565209107522163,
"segment": [
567.6384999999999,
573.59275
]
}
]
}
}
You can provide scores and timestamps in any float format that numpy is capable of reading (i.e. you do not need to stick to a given number of decimal places).
To upload your results to CodaLab you have to zip the test file into a flat zip archive (it can’t be inside a folder within the archive).
You can create a flat archive using the command providing the JSON file is in your current directory.
$ zip -j my-submission.zip test.json
Start: July 14, 2021, midnight
Description: 2021 Open Testing Phase - Action Detection
Nov. 25, 2021, 11:59 p.m.
You must be logged in to participate in competitions.
Sign In