2019 FG Challenge Recognizing Families In the Wild (RFIW19)

Organized by jvision - Current server time: Oct. 16, 2018, 8:46 p.m. UTC

Current

Training
Sept. 15, 2018, midnight UTC

Next

Validation
Nov. 10, 2018, midnight UTC

End

Competition Ends
Jan. 10, 2019, midnight UTC

Recognizing Families In the Wild (RFIW2019)

A 2019 FG Challenge

Family Classification (Track II)

LOGO

Overview

Large-scale kinship recognition data challenge, Recognizing Families In the Wild (RFIW), in conjunction with the 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019). We have the largest & most comprehensive database for visual kinship recognition, Families in the Wild(FIW), which is used to organize this, RFIW 2019 (3rd Edition).  

RFIW 2019 is supported by 3 laboratory style evaluation protocols: (1) Kinship Verification, (2), Family Classification (i.e., this), and (3) Tri-Subject, where (1) and (2) are repeated and (3) is being held for the 1st time. Register and get more information about Kinship Verification (Track 1) via https://competitions.codalab.org/competitions/20180 and stay-tuned for Track 3 to go live (i.e., opening portal soon)!

Note that each Track is  hosted in independent portals. The reason it is done this way is that the data splits, metrics used, results, and such are task-specific. Therefore, the different tasks are handled entirely seperate in terms of registration, downloads, and scoreboard rankings. Participants and teams can sign-up for one or all tracks. 

Look back at previous competitions, RFIW2017 and RFIW2018.
For more information about FIW visit https://web.northeastern.edu/smilelab/fiw/index.html.

Important dates

  • 2018.09.15 Team registration opens.
  • 2018.09.15 Training (labels) and validation (no labels) data made available (Phase Ⅰ).
  • 2018.09.15 Validation server online.
  • 2018.11.01 Validation labels released (Phase Ⅱ).
  • 2019.01.01 Test "blind" set and labels for validation set are released; validation server closed (Phase Ⅲ).
  • 2019.01.10 Test results and README's (i.e., brief descriptions of each submission) are due.
  • 2019.01.10 Challenge submissions due (i.e., scoring servers closed).
  • 2019.02.01 Challenge papers due.
  • 2019.02.10 Author notifications (i.e., oral or poster).
  • 2019.02.15 Results made public.
  • 2019.03.01 Camera-ready.
  • TBD RFIW2019 Challenge in conjunction with FG 2019; Winners announced during workshop.

Provided Resources

  • Scripts: Scripts will be provided to facilitate the reproducibility of the images and performance evaluation results once the validation server is online. More information is provided on the data page.
  • Forum: Any questions, comments, etc., use the public forum on the data page. (Highly recommended!)
  • Contact: Feel free to contact organizers directly (Joseph Robinson robinson.jo [at] husky.neu.edu (webpage) of SMILE Lab at NEU and Ming Shao mshao [at] umassd.edu (webpage) of UMass Dartmouth .

Recognizing Families In the Wild (RFIW2019)

A 2019 FG Challenge

Kinship Verification (Track I)

LOGO

A description of track 1 of RFIW2019, Kinship Verification, that includes a brief task overview, specifications for data splits and metrics used (i.e., evaluation protocol). We encourage participants to bring up any and all task-specific questions in the public forum provided as part of this portal-- If something seems unclear, lacking needed details, or even missing entirely, then please point this out for the sake of current and future participants (i.e., modifications will be made accordingly).

Overview

Provided multiple members from a known set of families (i.e., classes), the goal is to model each family (i.e., build a classifier) to then recognize the family from which a set of unseen individuals belongs to. Thus, Family Classification is the one-to-many view of the kinship recognition problem.

Family classification focuses on a slightly different problem than the popular task kinship verification (kinship verification is offered as Track 1 of RFIW, https://competitions.codalab.org/competitions/20180): Given a facial image, find the family to which the face in the image belongs to, i.e., families are modeled using facial images of other family members. As one might guess, challenge-level increases with increasing in number of families, as families contain large intra-class variations that typically fools trained models. Similar to conventional facial recognition, when the target data are unconstrained images captured in the wild (e.g., variations in pose, illumination, expression, etc.), the task gets increasingly more difficult, as it more heavily imitates that of real-world scenarios. Although such data are challenging these challenges must be overcome to advance automatic family recognition technologies, i.e., robust systems capably of handling unconstrained family data is needed to employ in practical use cases.

In this task (i.e., family recognition), a pre-specified gallery of face images is given with all family labels provided. The goal is to then identify the family label from the face of an unseen subject (i.e., all but a single member are included in training set). For instance, say we had 10 samples for 5 members of 25 different families (i.e., 50 samples per family). The gallery would contain all 10 samples for just 4 of the members, which is used to train on. Then, the 5th member of each family is given as test data. Thus, the task is then to predict which of the of the 25 families each sample for each (held out) member belongs to. This is how the evaluation is organized, except each family has at least 5 members (up to over 30 members for a single family), and variable number of samples per family member (i.e., how ever many family photos they appear in). Thus, an added challenge is age variation with respect to each member. Another added challenge is the number of families used in the evaluation is much greater (i.e., 517 of the 1,000 families of FIW are used).

As mentioned throughout, but worth repeating, the individuals making up the set of unseen faces (i.e., test set) are not included in the gallery and, thus, entirely unknown at test time.

Data Splits

 


FamiliesPhotos of families sampled randomly from FIW (i.e., 27 of 1,000).

Evaluation Settings and Metrics

The results for this multi-class problem will be reported as top 1% error ratings and visualized as confusion matrices.


Confusion Confusion metrics used for Family Classification.

Data Splits

FIW includes a total of 1,000 families with multiple samples for each of the members. Thus, the range of modern-day data-driven approaches (i.e., deep learning models) that is now possible opens doors to possibilities in terms of proposed solutions to the problem of family classification.

The data for family classification are split in 3 disjoint sets (i.e., train, validation, and test). Ground truth for the train set is released at start of Phase Ⅰ. At which time the scoring server is open to evaluate the validation set. Then, validation labels are released in Phase Ⅱ. Finally, the "blind" test set is released come Phase Ⅲ, for which no labels will be provided (i.e., test set labels are are not meant to be known until the competition is adjourned. It is expected that participants will only process the test set to generate submissions and, hence, any and all attempts to analyze and/or understand test set data is prohibited. All sets will be made up of an equal number of positive and negative pairs. Lastly, no family or subject (i.e., identity) overlap between any of the sets.

 

Submissions

Submissions should include

For submitting the results, you need to follow these steps:

  1. process each face image and record results in the same order they are listed. Predicted labels are integers representing the family ID provided in the training set.
  2. write a CSV file named results.csv (note delimiter ',').
  3. zip archive containing results.csv and a readme.txt Note that the archive should not include folders, both files should be in the root of the archive.
  4. the readme.txt file should contain a brief description of the method used to generate results.

Additional Resources

Source code and trained models will be made available on github, https://github.com/visionjo/FIW_KRT.

Recognizing Families In the Wild (RFIW2019)

A 2019 FG Challenge

Family Classification (Track II)

LOGO

 

 

 

 

Submissions

 
TBD

Recognizing Families In the Wild (RFIW2019)

A 2019 FG Challenge

Family Classification (Track II)

LOGO

Organizers

Picture
Joseph Robinson, Northeastern University
http://www.jrobsvision.com
 
Picture
Ming Shao, University of Massachusetts (Dartmouth)
http://www.cis.umassd.edu/~mshao/
 
Picture
Yun Fu, Northeastern University
http://www1.ece.neu.edu/~yunfu/
 
 
 

Training

Start: Sept. 15, 2018, midnight

Description: Training and validation data made available. Labels available for Training. Scoring server open.

Validation

Start: Nov. 10, 2018, midnight

Description: Labels for Validation made available. Evaluation scripts provided to participants. Validation will remain open for automatic scoring.

Challenge

Start: Dec. 20, 2018, midnight

Description: Test data release. Validation server closed. Open for final submissions.

Competition Ends

Jan. 10, 2019, midnight

You must be logged in to participate in competitions.

Sign In