Competition

Evaluation Criteria

This page describes how the competition submissions will be evaluated and scored.

We use the Binary Cross-Entropy (BCE) loss to evaluate the performance of face forgery detection models:

where:

N is the number of videos in the hidden test set.

y_i is the ground truth label of video i (fake: 1, real: 0).

p(y_i) is the predicted probability that video i is fake.

A smaller BCELoss score is better, which directly contributes to a higher ranking. If the BCELoss score is the same, the one with less execution time (in seconds) will achieve a higher ranking. To avoid an infinite BCELoss that is both too confident and wrong, the score is bounded by a threshold value.

Terms and Conditions

General Rules

The DeeperForensics Challenge 2020 will be around two months long (nine weeks), where eight weeks for the development phase and one week for the final test phase. Please refer to Phases for the detailed time points. The challenge has officially started at the ECCV 2020, The 2nd Workshop on Sensing, Understanding and Synthesizing Humans.

Participants are recommended but not restricted to train their algorithms on the publicly available DeeperForensics-1.0 dataset. The overview page provides the link to download the data.

The hidden test set used for online evaluation is divided into two parts: test-dev and test-final. Test-dev contains around 1,000 videos that represents general circumstances of the hidden test set and is used to maintain a public leaderboard. Test-final contains around 3,000 videos with a similar distribution as test-dev (also including test-dev videos), which is used to evaluate final submissions for competition results. The final results will be revealed around Dec. 2020. Participants are expected to develop more robust and generalized methods for face forgery detection in real-world scenarios.

When participating in the competition, please be reminded that:

• Any and all external data used for the competition must be available to all the participants without additional cost, and the used external data must be specified in the "method description" when uploading results to the submission website.
• Results in the correct format must be uploaded to the evaluation server. The evaluation page lists detailed information regarding how results will be evaluated.
• Each entry must be associated with a team and provide its affiliation. Please provide this information when requesting the evaluation resources.
• The online evaluation results must be submitted through this CodaLab competition site of the DeeperForensics Challenge. The online evaluation chances will be updated on Friday, 11:59 p.m. UTC every week. The participants can conduct 4 online evaluations (each with 2.5 hours of runtime limit) per week in the development phase. A total of 2 online evaluations (each with 7.5 hours of runtime limit) are allowed during the final test phase. Each participant will be assigned 1 Tesla V100 GPU with 16 GB memory capacity. The organizer will offer some hints/codes to ensure a successful submission.
• Using multiple accounts to increase the number of submissions and private sharing outside teams are strictly prohibited.
• The organizer reserves the absolute right to disqualify entries which is incomplete or illegible, late entries or entries that violate the rules.
• The organizer reserves the absolute right to adjust the competition schedule and rules based on situations.
• The best entry of each team should be public in the leaderboard at all time.
• To be qualified to compete for awards, the participants should submit a fact sheet to introduce their team information, method and the used data. After the final results are decided, the main winners will be invited to submit a detailed technical report.

Terms of Use: DeeperForensics-1.0 Dataset

Before downloading and using the DeeperForensics-1.0 dataset, please agree to the following terms of use. You, your employer and your affiliations are referred to as "User". The authors and their affiliations, Nanyang Technological University and SenseTime, are referred to as "Producer".

• The DeeperForensics-1.0 dataset is used for non-commercial/non-profit research purposes only.
• All the source videos in DeeperForensics-1.0 dataset collected by the Producer are bounded by a formal agreement with all the actors. All the videos in DeeperForensics-1.0 dataset can be used for academic purposes. However, the Producer is NOT responsible for any further use in a defamatory, pornographic or any other unlawful manner, or violation of any applicable regulations or laws.
• The User takes full responsibility for any consequence caused by his/her use of DeeperForensics-1.0 dataset in any form and shall defend and indemnify the Producer against all claims arising from such uses.
• The User should NOT distribute, copy, reproduce, disclose, assign, sublicense, embed, host, transfer, sell, trade, or resell any portion of the DeeperForensics-1.0 dataset to any third party for any purpose.
• The User can provide his/her research associates and colleagues with access to DeeperForensics-1.0 dataset (the download link or the dataset itself) provided that he/she agrees to be bound by these terms of use and guarantees that his/her research associates and colleagues agree to be bound by these terms of use.
• The User should NOT remove or alter any copyright, trademark, or other proprietary notices appearing on or in copies of the DeeperForensics-1.0 dataset.
• This agreement is effective for any potential User of the DeeperForensics-1.0 dataset upon the date that the User first accesses the DeeperForensics-1.0 dataset in any form.
• The Producer reserves the right to terminate the User's access to the DeeperForensics-1.0 dataset at any time.
• For using DeeperForensics-1.0 dataset, please cite the following paper:

  @inproceedings{jiang2020deeperforensics1,
    title={{DeeperForensics-1.0}: A Large-Scale Dataset for Real-World Face Forgery Detection},
    author={Jiang, Liming and Li, Ren and Wu, Wayne and Qian, Chen and Loy, Chen Change},
    booktitle={CVPR},
    year={2020}
  }
  

The download link will be sent to you once your request is approved.

Software

Copyright © 2020, DeeperForensics Consortium. All rights reserved. Redistribution and use software in source and binary form, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of the DeeperForensics Consortium nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE AND ANNOTATIONS ARE PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.