The VoxCeleb Speaker Recognition Challenge - Audio speaker verification - OPEN training data

Organized by vgg - Current server time: Oct. 22, 2019, 5:46 a.m. UTC

Previous

Challenge workshop
July 15, 2019, midnight UTC

Current

Permanent
Aug. 30, 2019, 8 a.m. UTC

End

Competition Ends
Never

The VoxCeleb Speaker Recognition Challenge (VoxSRC)

Audio only speaker verification - Open training data

Welcome to the VoxCeleb speaker verification challenge! The goal of this challenge is to probe how well current methods can recognize speakers from speech obtained 'in the wild'. The dataset is obtained from YouTube videos of celebrity interviews, consisting of multi-speaker audio from both professionally edited and red carpet interviews as well as more casual conversational multi-speaker audio in which background noise, laughter, and other artefacts are observed in a range of recording environments.

The task of speaker verification is to determine whether two samples of speech are from the same person.

This the competition site for the Open training data. Participants can train using the VoxCeleb datasets and any other data (including that which is not publicly released) except the challenge's test data, see more details in the "Evaluation" section.

Organisers

Arsha Nagrani, VGG, University of Oxford
Joon Son Chung, Naver, South Korea
Andrew Zisserman, VGG, University of Oxford
Ernesto Coto, VGG, University of Oxford
Weidi Xie, VGG, University of Oxford
Mitchell McLaren, Speech Technology and Research Laboratory, SRI International, CA
Douglas A Reynolds, Lincoln Laboratory, MIT

More information

For more information, visit the VoxSRC Challenge page and read the rest of the sections under "Learn the Details".

Acknowledgements

This work is supported by the EPSRC programme grant Seebibyte EP/M013774/1: Visual Search for the Era of Big Data.

The VoxCeleb Speaker Recognition Challenge (VoxSRC)

Audio only speaker verification - Open training data

Evaluation Plan

The Task

The test data for the challenge consists of pairs of audio segments, and the goal is to simply say whether they are from the same speaker or from different speakers. Teams are invited to create a system that takes the test data and produces a list of floating-point scores, with a single score for each pair of segments.

*****
All scores must be lying between the closed interval [0, 1], where 1 means the pair of segments correspond to the same speaker and 0 means the pair of segments correspond to different speakers.
*****

Via this CodaLab competition, teams will be able to compare the results of their system with the competition's blind set of results, during (and even after) the evaluation period. To participate, open the competition page and click on the "Participate" tab. Then accept the Terms and Conditions and click on the "Register" button. After this you will be able to upload files via the "Participate" tab. Note that a team must only submit to CodaLab the outputs of its system, not the system itself. Make sure you have read the "Submission Instructions" under the "Participate" tab before uploading any files.

In order to submit a file you need to click on the "Submit/View Results" link under the "Participate" tab. After this, you will be able to see two buttons, corresponding to the Competition Phases. Click on one of the buttons to choose the phase you want to submit to. The available phases are:

  • Challenge workshop: This is the phase during which participants will be submitting their results for the challenge workshop to be held in conjunction with Interspeech 2019. The leaderboard will be visible during this phase.
  • Permanent: During this phase new submissions will be accepted but they will not be taken into account for the Challenge workshop. During this phase the leaderboard will be INVISIBLE until the challenge workshop takes place. It will be made visible after that.

In order to prevent overfitting to the test data participants can only submit one result per day. There is also a limit over the total number of submissions for each phase, see more details about this under "Submit/View Results".

*****
Update 25 July 2019: Please note that any deliberate attempts to bypass the submission limit (for instance, by creating multiple accounts and using them to submit) will lead to automatic disqualification.
*****

Scoring

System output will be evaluated using the Equal Error Rate (EER), a rate used to determine the threshold value for a system when its false acceptance rate (FAR) and false rejection rate (FRR) are equal, see below

FAR = FP/(FP+TN) FRR = FN/(TP+FN)

where

  • FN is the number of false negatives
  • FP is the number of false positives
  • TN is the number of true negatives
  • TP is the number of true positives

The code used for calculating the EER can be found in the development kit. Some examples for the ground truth and the predictions are also provided, but bear in mind that the format of these example files is a bit different to the format of the file that you need to submit in this competition. Make sure you have read the "Submission Instructions" under the "Participate" tab.

Training data

This competition site corresponds to the Open training data task, so participants can train using the VoxCeleb datasets and any other data (including that which is not publicly released) except the challenge's test data.

Validation data

We encourage participants to validate their models using the VoxCeleb2 publicly released hard and easy test lists:

List of trial pairs - VoxCeleb1
List of trial pairs - VoxCeleb1 (cleaned)
List of trial pairs - VoxCeleb1-H
List of trial pairs - VoxCeleb1-H (cleaned)
List of trial pairs - VoxCeleb1-E
List of trial pairs - VoxCeleb1-E (cleaned)

These can also be found on the VoxCeleb2 website.

Test data

The test data is blind, i.e., it does not include annotations. It can only be used strictly for reporting of results alone. It cannot be used in any way to train or tune systems. You will find a link to download the data under "Learn the Details".

Competition Results

The results of this CodaLab competition will be announced at the challenge workshop, where we will invite presentations from the most exciting and novel submissions, as well as from the challenge winners. The challenge workshop will be held on the 14th of September 2019, in conjunction with Interspeech 2019 in Austria.

Terms and Conditions

Participation in this competition is open to all who are interested and willing to comply with the rules laid out under the "Learn the Details" and "Participate" tabs. There is no cost to participate, although teams are encouraged to submit a paper to the corresponding Interspeech 2019 challenge workshop, to be held in Austria on the 14th of September 2019.

We also kindly ask you to associate the CodaLab account you will use for the competition to your institutional e-mail. We reserve the right to revoke your access to the competition otherwise.

*****
Update 25 July 2019: Please note that any deliberate attempts to bypass the submission limit (for instance, by creating multiple accounts and using them to submit) will lead to automatic disqualification.
*****

In case of any issues, all decisions made by the Organizing Committee will be final.

You can download a ZIP file with the test data from here.

The file contains 19154 .wav files and one text file called "list_pairs_test_data.txt". The text file contains the pairs that you are to evaluate for the competition. There are 208008 pairs to be evaluated. It is important to note the order of the pairs, as you need the keep the same order in the results file that you will submit to CodaLab.

For reference, we have added a baseline result to the leaderboard, submitted by the vgg user.

Good Luck!.

Challenge workshop

Start: July 15, 2019, midnight

Description: Submissions for the challenge workshop that will be held in conjunction with Interspeech 2019

Permanent

Start: Aug. 30, 2019, 8 a.m.

Description: Submissions for comparison with previous ones. Not to be taken into account for the challenge workshop

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In
# Username Score
1 matejkap 0.0126
2 dgromero 0.0146
3 TZ 0.0163