Competition

Evaluation Criteria

In this challenge, you will need to submit per-frame predictions of Heart Rate (and Respiration Rate) for the videos in the validation/test set. Your submission will be evaluated against the private ground truth. The V4V dataset we use in this challenge has ground-truth readings for HR per frame, RR per frame and blood-pressure signal at 1KHz frequency synchronized with video recording.

We use standard MAE, RMSE and Pearson Correlation Coefficient for scoring your submission. Note that we measure error per frame of the validation video and average the error across all frames of the validation set. We also release the evaluation code for your reference. Please find the evaluation script here. We will rank the submission according the MAE score across validation set. However, we also provide RMSE and Pearson Correlation Coefficient for additional feedback on your submission. If N is the total number of frames across validation set, we measure MAE as

More details on the submission file format and sub-challenge participation is available on the "submission" tab on this portal.

Winner will be announced for each sub-challenge at the end of the V4V challenge. Good luck! :-)

For any issues/questions on this section, please reach out to arevanur(at)andrew(dot)cmu(dot)edu.

Official Rules

Common terms used in these rules:

These are the official rules that govern how the Vision For Vitals (V4V) Challenge will operate. This challenge will be simply referred to as the “contest” or the “challenge” throughout the rest of these rules and may be abbreviated on our website, in our documentation, and other publications as V4V.

In these rules, “organizers”, “we,” “our,” and “us” refer to the organizers of the V4V Challenge; “Database” refer to all the distributed image and annotation data; "participant”, “you,” and “yourself” refer to an eligible contest participant.

Contest Description

This is a skill-based contest and chance plays no part in the determination of the winner(s).

1. Focus of the Contest: Predicting Heart-Rate and Respiration-Rate from facial videos.
2. All eligible entries received will be judged using the criteria described below to determine winners

Data Description and Usage Terms

1. “V4V” Database was developed in the group of Dr. Lijun Yin of the State University of New York at Binghamton and Dr. Laszlo Jeni of Carnegie Mellon University. The data includes 2D videos and physiological measurements.
The project was funded by the United States National Science Foundation for human affective behavior research.
2. The organizers make no warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose. The copyright of the images remain the property of their respective owners. By downloading and making use of the data, you accept full responsibility for using the data. You shall defend and indemnify the organizers, including their employees, Trustees, officers and agents, against any and all claims arising from your use of the data.
3. Test data: The organizers will use test data to perform the final evaluation, hence the participants’ final entry will be based on test data.
4. Training and validation data: The contest organizers will make available to the participants a training dataset with truth values, and a validation set with no truth values. The validation data will be used by the participants for practice purposes to validate their systems. It will be similar in composition to the test set (validation labels will be provided in the final test stage of the challenge).

The datasets may be used for the V4V Challenge of ICCV 2021 only. The recipient of the datasets must be a full-time faculty, researcher or employee of an organization (not a student) and must agree to the following terms:

1. The participants receive a non-exclusive, non-transferable license to use the Database for internal research purposes. You may not sell, rent, lease, sublicense, lend, time-share or transfer, in whole or in part, or provide third parties access to the Database.
2. The data will be used for non-for-profit research only. Any use of the Database in the development of a commercial product is prohibited.
3. If this Database is used, in whole or in part, for any publishable work, the following papers must be referenced:

Eligibility criteria

1. You are an individual or a team of people desiring to contribute to the tasks of the challenge and accepting to follow its rules; and
2. You are NOT a resident of any country constrained by US export regulations included in the OFAC sanction page http://www.treasury.gov/resource-center/sanctions/Programs/Pages/Programs.aspx. Therefore residents of these countries / regions are not eligible to participate; and
3. You are not involved in any part of the administration and execution of this contest; and
4. You are not an immediate family (parent, sibling, spouse, or child) or household member of a person involved in any part of the administration and execution of this contest.

This contest is void within the geographic area identified above and wherever else prohibited by law.

Entry

1. All members of the team meets eligibility criteria.
2. To be considered in the competition, provide a description of your approach. To be considered for publication in the proceedings, submit a workshop paper. All participants are invited to submit a maximum 8-page paper for the proceedings of the ICCV 2021 Vision For Vitals Challenge workshop.
3. Workshop report: The organizers may write and publish a summary of the results.
4. Submission: The entries of the participants will be submitted on-line via the Codalab web platform. During the development period, the participants will receive immediate feed-back on validation data released for practice purpose. For the final evaluation, the results will be computed automatically on test data submissions. The performances on test data will not be released until the challenge is over.
5. Original work, permissions: In addition, by submitting your entry into this contest you confirm that, to the best of your knowledge:
• Your entry is your own original work; and
• Your entry only includes material that you own, or that you have permission from the copyright / trademark owner to use.

On-line notification

We will post changes in the rules or changes in the data as well as the names of confirmed winners (after contest decisions are made by the judges) online on the https://vision4vitals.github.io/

Conditions.

By entering this contest you agree all terms of use. You understand that the violation of the use will be pursued.

This contest is void within the geographic area identified above and wherever else prohibited by law.

Entry

1. All members of the team meets eligibility criteria.
2. To be considered in the competition, provide a description of your approach.
3. Workshop report: The organizers may write and publish a summary of the results.
4. Submission: The entries of the participants will be submitted on-line via the Codalab web platform. During the development period, the participants will receive immediate feed-back on validation data released for practice purpose. For the final evaluation, the results will be computed automatically on test data submissions. The performances on test data will not be released until the challenge is over.
5. Original work, permissions: In addition, by submitting your entry into this contest you confirm that, to the best of your knowledge:
• Your entry is your own original work; and
• Your entry only includes material that you own, or that you have permission from the copyright / trademark owner to use.

On-line notification

We will post changes in the rules or changes in the data as well as the names of confirmed winners (after contest decisions are made by the judges) online on https://vision4vitals.github.io/.

Conditions.

By entering this contest you agree all terms of use. You understand that the violation of the use will be pursued.

Submission

Participants should submit their results as a single zip archive, containing a text file that has per-frame predictions for the physiological estimates of Heart Rate (HR) and Respiration Rate (RR). Note that the HR and RR are different subchallenges and have different leaderboards. All participants are required to submit predictions to HR sub-challenge. Although, participation in RR sub-challenge is optional, we encourage all participants to involve in both sub-challenges.

Submission File Format

A sample submission file in the right format can be downloaded here. For participants who do not wish to participate in RR sub-challenge, please predict "-1.0" as the per-frame RR prediction. We have provided another sample here for your reference Please name your file as "results.txt" with each line containing per-frame prediction AND zip the "results.txt" file into a zip archive named "submission.zip".

video_name_1, HR, XX, XX, XX, XX, XX ...., XX

video_name_1, RR, XX, XX, XX, XX, XX ...., XX

video_name_2, HR, XX, XX, XX, XX, XX ...., XX

video_name_2, RR, XX, XX, XX, XX, XX ...., XX

...

...

Please upload the "submission.zip" file to the codalab submissions page. If your submission cannot be processed, the evaluation script will inform you why it could not process the submission. It might also provide some additional information to help you correct your submission. If your submission is processed, then the script will place your scores on the leaderboard directly.

• The order of the file names and the physiology measurements must match the order indicated in the sample submission file
• The number of predictions per video (same as number of frames) should match with the sample submission
• If you are not participating in RR, please place a -1.0 prediction (for each frame) corresponding to RR rows.
• For any issues/questions, please reach out to arevanur(AT)andrew(DOT)cmu(DOT)edu.