HEART-MET Gesture Recognition Challenge

Organized by heartmet - Current server time: Jan. 7, 2025, 2:35 a.m. UTC

First phase

Validation
April 8, 2021, midnight UTC

End

Competition Ends
June 30, 2021, 11:59 p.m. UTC

HEART-MET Gesture Recognition Challenge

Results

The challenge has been completed, and the results have been announced on our website.

Congratulations to team AcRec@UniKoblenz and RAAR3D!

 

Overview

In the HEART-MET Gesture Recognition Challenge, the task is to recognize gestures from videos. The gestures are meant to communicate intentions or commands to the robot, and include the stop sign, nodding, pointing etc. Some examples can be seen in this video.

HEART-MET is one of the competitions in the METRICS project, which has received funding from European Union’s Horizon 2020 research and innovation program under grant agreement No 871252. The competition aims to benchmark assistive robots performing healthcare-related tasks in unstructured domestic environments.

Gesture recognition is one of the functional benchmarks by which assistive robots are evaluated. Gesture recognition is necessary for non-verbal communication with a user. The datasets for this challenge are collected from real robots performing gesture recognition in domestic environments with several different volunteers performing the gestures.

Other challenges presented by HEART-MET include Activity Recognition, Object Detection and Handover Failure Detection. More details can be found on the HEART-MET website.

Participation

In order to participate:

  1. Create an account on Codalab
  2. Fill out this form with your details (if you have not already done so for a another HEART-MET challenge)
  3. Register under the Participate tab

Your request will be approved within 24 hours.

If you are participating as a team, the team leader can create a new team by clicking the Team tab. Team members can then request to join the team. A team member must first be registered to participate in the challenge before they can request to join a team. See Competition Teams for more details.

Challenge

The challenge consists of two phases:

  1. the validation phase (ends on 15.05.2021)
  2. the competition phase (ends on 31.05.2021)

During the validation phase, you are provided with a subset of the dataset with which you can validate your models, and test the submission procedure. No training data is provided since it is expected that you use existing public datasets for training.

The competition phase will start at the end of the validation phase, and you will be provided with a larger competition set. The labels for the validation set will be released as well; hence you can use the validation set for fine-tuning models if required. The competition phase will have a limit on the total number of submissions and the maximum submissions per day, and the duration is shorter compared to the validation phase.

Evaluation

Submission Format

You must submit a submission bundle (a .zip archive) with a single text file named submission.txt. This file has the following format:

video0000.mp4 3
video0001.mp4 7
video0002.mp4 2
...
...

Each line consists of the name of the video file and the recognized gesture ID.

A sample submission file has been provided in the starting kit.

The gesture IDs are from 0-8 (inclusive), and can be found under Dataset.

Evaluation Criteria

The evaluation is based on the true positive rate = number of true positives / total number of videos

Dataset format

The dataset consists of video clips up to 7 seconds long, of a person performing a gesture. The gesture may be performed once (for e.g. stop sign) or multiple times (for e.g. waving). The videos are recorded from multiple cameras; hence the dataset may contain videos of the same volunteer performing a gesture from different viewpoints.

Gesture Classes

There are a total of 9 gestures in this challenge. There is some (intentional) overlap with the classes in the 20BN-jester dataset and the NVIDIA Dynamic Hand Gesture Dataset, with some additional classes not present in either dataset.

The gesture IDs and the corresponding classes contained in this dataset are listed below:

0Nodding
1Stop sign
2Thumb down
3Waving
4Pointing
5Pulling hand in
6Thumb up
7Pushing hand out
8Shaking head

Label Format

The labels for the validation set will be released at the end of the validation phase. It will consist of a single text file with each line consisting of the name of the video and the gesture ID, as seen in the example below:

video0000.mp4 3
video0001.mp4 7
video0002.mp4 2
...
...

Validation and Competition Datasets

The validation set consists of 100 videos and the competition set consists of 314 videos.

Terms and Conditions

  1. This challenge is organized by HEART-MET and has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 871252 (METRICS).
  2. It is open to the public, but requires participants to fill out a registration form, and register using a Codalab account.
  3. Participation is possible by individuals and teams.
  4. If participating in a team, each team member must be registered as a participant (for details see Overview).
  5. Each participant is only allowed to create one account.
  6. The competition consists of two phases: validation and competition.
  7. Final rankings are based on the results in the competition phase.
  8. Participants are allowed (and encouraged) to use external data for training models.
  9. If external data is used for training, participants are encouraged to specify the details of data and models used in the forum.
  10. Manually labeled submissions are not permitted. Any submission which includes results which have been manually entered will be disqualified.
  11. It is not permitted to use the provided data in a particular phase for training models for that phase. It is, however, permitted to use the validation dataset for training a model for the competition phase
  12. At the end of the competition phase, the top 3 participants must submit their software and trained models to the organizers. The package should contain instructions for setting up the development environment and running the code with the competition set. The submitted code and models will not be made public or shared outside of the organizers, and will only be used for verifying the results.
  13. The datasets are released under the Creative Commons Attribution 4.0 International license.

Validation

Start: April 8, 2021, midnight

Competition

Start: May 15, 2021, 11:59 p.m.

Competition Ends

June 30, 2021, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In