WebVision Challenge on Visual Understanding by Learning from Web Data - Pascal VOC Transfer Learning

Organized by etagust - Current server time: July 22, 2018, 2:48 a.m. UTC

Previous

Development
May 15, 2017, midnight UTC

Current

Testing
June 15, 2017, midnight UTC

End

Competition Ends
Never

Challenge

The goal of this challenge is to advance the area of learning knowledge and representation from web data. The web data not only contains huge numbers of visual images, but also rich meta information concerning these visual data, which could be exploited to learn good representations and models. We organize two tasks to evaluate the learned knowledge and representation: (1) WebVision Image Classification Task, and (2) Pascal VOC Transfer Learning Task. The second task is built upon the first task. Researchers can participate into only the first task, or both tasks.

News: A 10,000$ cash award will be given to the winners of the challenge!

WebVision Image Classification Task

The WebVision dataset is composed of training, validation, and test set. The training set is downloaded from Web without any human annotation. The validation and test set are human annotated, where the labels of validation data are provided but the labels of test data are withheld. To imitate the setting of learning from web data, the participants are required to learn their models solely on the training set and submit classification results on the test set. The validation set could only be used to evaluate the algorithms during development (see details in Honor Code). Each submission will produce a list of 5 labels in the descending order of confidence for each image. The recognition accuracy is evaluated based on the label which best matches the ground truth label for the image. Specifically, an algorithm will produce a label list: \(c_i\), \(i=1,...,5\) for each image and the ground truth labels of the image are: \(y_j\), \( j = 1,..., n \) with n class labels. The error of this prediction is defined as: $$E = \frac{1}{n} \sum_{j=1}^n \min_{i} d(c_i, y_j).$$ The \(d(c_i,y_j)\) is calculated as 0 if \(c_i=y_j\) and 1 otherwise. The final errors of the algorithm is the average corresponding error across all test images. For this version of the challenge, there is only one ground truth label for each image (i.e., \(n=1\)).

Pascal VOC Transfer Learning Task

This task is designed for verify the knowledge and representation learned from the WebVision training set on the new task. Hence, participants are required to submit results to the first task and transfer only models learned in the first task. We choose the image classification task of Pascal VOC 2012 [link to VOC 2012 page] to test the transfer learning performance. Participants could exploit different ways to transfer the knowledge learned in the first task perform image classification Pascal VOC 2012. For example, treating the learned models as feature extractors and learning the SVM classifier based on the features (Note that the model used in this transfer learning task has to be submitted to the WebVision Image Classification task for evaluation.). The evaluation protocol strictly follows the previous Pascal VOC. The participants are required to submit results in the Pascal VOC format to our server and we perform the evaluation by submitting these results to the Pascal VOC evaluation server.

 

Awards

Award will be given to top 3 performers of each track.

Submission Policy

To encourage more teams to participate in this challenge, we will maintain a leaderboard to show the recognition results of all teams. In our schedule, we have three submission deadlines. Each team can submit 3 results to our evaluation server for the first two deadlines, and 5 results for the final deadline. The leaderboard will be updated after each deadline. The final rank is based on the best of 5 results in the final submission for each team.

Honor Code

This challenge aims to learn knowledge and visual representation from web data without human annotations. Therefore, we request all participants:

  1. For the first task, during training phase, algorithms can be designed to learn from web images and meta information on the training set. Only the provided training data can be used. External data is strictly prohibited in this challenge.
  2. For the first task, the validation set is human annotated. Therefore, it can not be merged into training data for learning models. It could only be used to evaluate the algorithms during development.
  3. For the second task, the standard protocol of PASCAL VOC should be strictly followed. Training data and the models in the first task can be used for improving the PASCAL VOC classification performance. Other external data or models are not allowed.

 

Important Dates

March 15, 2017 Develop kit, data, and evaluation code are public
April 15, 2017 First submission deadline
May 15, 2017 Second submission deadline
June 15, 2017 Final submission deadline
July 15, 2017 Challenge results are released
July 26, 2017 Workshop date (co-located with CVPR'17)

All deadlines are at 23:59 Pacific Standard Time.

Development

Start: May 15, 2017, midnight

Description: To submit to the challenge, please update a .zip file containing link.txt file, containing the link to the anonymous results of the Pascal VOC2012 submission (e.g. http://host.robots.ox.ac.uk:8080/anonymous/XXXXXX.html ). ) An example submission file can be found at https://data.vision.ee.ethz.ch/cvl/webvision/example_submission_pascal.zip

Testing

Start: June 15, 2017, midnight

Description: To submit to the challenge, please update a .zip file containing link.txt file, containing the link to the anonymous results of the Pascal VOC2012 submission (e.g. http://host.robots.ox.ac.uk:8080/anonymous/XXXXXX.html ). ) An example submission file can be found at https://data.vision.ee.ethz.ch/cvl/webvision/example_submission_pascal.zip

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In