WIDER Face & Person Challenge 2019 - Track 1: Face Detection

Organized by wider - Current server time: Nov. 17, 2019, 10:29 a.m. UTC

Previous

Final Runtime Test
June 15, 2019, 6:59 a.m. UTC

Current

Development
May 10, 2019, 6:59 a.m. UTC

End

Competition Ends
Aug. 8, 2019, 6:59 a.m. UTC

Overview

The WIDER Face Challenge aims at soliciting new approaches to advance the state-of-the-art in face detection. The challenge uses the WIDER Face dataset, which is a face detection benchmark dataset proposed in CVPR 2016. WIDER Face dataset contains 32,203 images and 393,703 faces bounding box annotations. Faces in the WIDER Face dataset has a high degree of variability in scale, pose and occlusion as depicted in the sample images. WIDER Face dataset is organized based on 61 event classes. For each event class, we randomly select 40%/10%/50% data as training, validation and testing sets. Users are required to submit final prediction files, which we shall proceed to evaluate.

Data Description

We follow the WIDER Face convention to provide image-level annotations. Each image contains a set of face bounding boxes with the format of "[left, top, width, height]". The annotations share a data structure below:

  • image name
    • face bounding boxes
      • left
      • top
      • width
      • height

 

Submission Format

Different from WIDER Face convention, all the detection results across test images should be writtern in a single text file. For example, if the directory of a testing image is "./0--Parade/0_Parade_marchingband_1_5.jpg", The detection output is expected in the follwing format:

	...
	< # 0--Parade/0_Parade_marchingband_1_5.jpg >
	< face i1 >
	< face i2 >
	...
	< face im >
	...

The text file should contain 1 row per detected bounding box, in the format "[left, top, width, height, score]". A zip file should be generated to pack the text file. Each zip file should contain only one evaluation result. Do not pack multiple submissions into a single zip file. The evaluation server only accepts a zip as valid input. If the above descriptions are unclear, please see the example of submission file, which can be downloaded with the dataset.

 

Runtime evaluation

In order to encourage pratical use of current state-of-the-art algorithms, we add an experimental track by considering the runtime of the face detection algorithm. The submission to this track will be in the form of docker images. They will be evaluated against a new inhouse collected testing dataset with similar distribution as the original WIDER Face test set. This set contains 10,000 images.

The participates need to wrap the algorithm into the docker and implement the predefined interface to generating valid detection output.

Please follow the steps below to participate in the runtime tracks.

1. Request evaluation resources

Please start with registering for an AWS account at any AWS global regions. Then, each participant (or a team) please send one email to wider-challenge@ie.cuhk.edu.hk with the following information: 1) your AWS ID; 2) your email address. AWS ID is a 12-digit string you can view in the AWS account.
You will receive one notification email when we have finished allocating computation resources for your evaluation.

 

2. Build and submit the evaluation docker image

We have provided a basic docker image template at HERE. The notification email in the step 1 will contain on ECR docker image repository id where you can push your docker image to. Please follow the instruction we provided to push your docker image.
After each new image is pushed, the evaluation for it will automatically start. Thanks to generous donation from AWS, we run participants' the submitted docker images on AWS. After this step the you will receive an encrypted result file by email for each submission.

3. Submit result files to codalab

The participant are expected to submit the result files directly to the "Final Runtime Test" phase to get their evaluation results onto the leaderboard. The evaluation details together with "composite score" for the final ranking will be available in the Detailed Results column of the result page.

 

General Rules

Please check the terms and conditions for further details.

Detection Evaluation

This section describes the detection evaluation metrics used by WIDER Face Challenge. The evaluation code provided can be used to obtain results on the publicly available WIDER Face validation set. It computes average AP metrics described below. To obtain results on the WIDER Face test set, for which ground-truth annotations are hidden, generated results must be uploaded to the evaluation server. The exact same evaluation code, described below, is used to evaluate results on the test set.

Performance Metrics

The average precision is used for characterizing the performance of an object detector on WIDER Face:

  • AP are averaged over multiple Intersection over Union (IoU) values. Specifically we use 10 IoU thresholds of .50:.05:.95. Averaging over IoUs rewards detectors with better localization.
  • AP (averaged across all 10 IoU thresholds) will determine the challenge winner. This should be considered the single most important metric when considering performance on WIDER Face.
  • In WIDER Face, there are a large number of small faces. Faces with a height not shorter than 10 pixels will be regarded as valid ground truth.
  • Different from the WIDER Face dataset that contains three subsets for evaluation. We only evaluate WIDER FACE hard set for this challenge.

 

Runtime Metrics

To evualte the algorithms' accuracies as well runtime speed, this track uses the following metric, called composite score:
runtime_score.gif
The aap is the average average precision defined above. The fps is computed by 1 / (average runtime per-image) on the inhouse test set for this track.

A detailed explaination of the composite score can be found HERE.

The evaluation details together with above "composite score" for the final ranking will be available in the Detailed Results column of the result page. 

Terms and Conditions

General Rules

Participants are recommended but not restricted to train their algorithms on the provided train and val sets. The CodaLab page of each track has links to the respective data. The test set is divided into two splits: test-dev and test-challenge. Test-dev is as the default test set for testing under general circumstances and is used to maintain a public leaderboard. Test-challenge is used for the workshop competition; results will be revealed at the workshop. When participating in the task, please be reminded that:

  • Any and all external data used for training must be specified in the "method description" when uploading results to the evaluation server.
  • Results in the correct format must be uploaded to the evaluation server. The evaluation page on the individual site of each challenge track lists detailed information regarding how results will be evaluated.
  • Each entry much be associated to a team and provide its affiliation.
  • The results must be submitted through the CodaLab competition site of each challenge track. The participants can make up to 5 submissions per day in the development phases. A total of 5 submissions are allowed during the final test phase. Using multiple accounts to increase the number of submissions is strictly prohibited.
  • The organizer reserves the absolute right to disqualify entries which is incomplete or illegible, late entries or entries that violate the rules.
  • The best entry of each team will be public in the leaderboard at all time.
  • To compete for awards, the participants must fill out a fact sheet briefly describing their methods. There is no other publication requirement.

Datasets and Annotations

The datasets are released for academic research only and it is free to researchers from educational or research institutions for non-commercial purposes. When downloading the dataset you agree not to reproduce, duplicate, copy, sell, trade, resell or exploit for any commercial purposes, any portion of the images and any portion of derived data.

Software

Copyright © 2019, WIDER Consortium. All rights reserved. Redistribution and use software in source and binary form, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of the WIDER Consortium nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE AND ANNOTATIONS ARE PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Contact Us

For more information, please refer to the challenge webpage or contact us at wider-challenge@ie.cuhk.edu.hk.

Development

Start: May 10, 2019, 6:59 a.m.

Description: In this phase, you can submit the result of validation set and see your rank in leaderboard.

Final Accuracy Test

Start: June 15, 2019, 6:59 a.m.

Description: In this phase, we will release testing set and the leaderboard will show the result of testing set.

Final Runtime Test

Start: June 15, 2019, 6:59 a.m.

Description: In this phase, we will show the runtime in detailed result page.

Competition Ends

Aug. 8, 2019, 6:59 a.m.

You must be logged in to participate in competitions.

Sign In
# Username Score
1 JuneZhang 0.5756
2 pfluo 0.5635
3 ReRe 0.5512