Charades Challenge - CVPR 2017 Workshop

Organized by dustins - Current server time: Dec. 18, 2017, 3:24 a.m. UTC

Current

Testing
June 9, 2017, midnight UTC

Next

Testing
June 9, 2017, midnight UTC

Charades Challenge 2017

The Charades Challenge aims towards automatic understanding of daily activities, by providing realistic videos of people doing everyday activities. The Charades dataset is collected for an unique insight into daily tasks such as drinking coffee, putting on shoes while sitting in a chair, or snuggling with a blanket on the couch while watching something on a laptop. This enables computer vision algorithms to learn from real and diverse examples of our daily dynamic scenarios. The challenge consists of two separate tracks: classification and localization track. The classification track is to recognize all activity categories for given videos ('Activity Classification'), where multiple overlapping activities can occur in each video. The localization track is to find the temporal locations of all activities in a video ('Activity Localization')

Track Information

 

This challenge is split between these two tracks, with separate leader boards and winners for each. Performance will be measured on:

Multi-label Video Classification Task: The task accepts submissions that use any available training data to train a model that predicts a score for 157 classes in each video. The performance of the algorithms is evaluated with mean average precision (mAP) across the videos.

Multi-label Action Localization Task: The task accepts submission that use any available training data to train a model that predicts a score for 157 classes for 25 equally spaced time-points in each video. The performance of the algorithms is evaluated with mean average precision (mAP) across all frames in all the videos.

Evaluation Criteria

For the action recognition competition, MAP (mean average precision) is used as the evaluation metric. Separate leader boards are kept for performance on the competition's two subtasks.

Action Recognition

For this task, participants should submit a zipped a csv file of the form:

        id   vector
    

where 'id' is a unique video id, and 'vector' is a whitespace delimited list of 157 floating point numbers representing the scores for each action class in a video.

Once you have this file, make sure that the string "class" is in the filename (so the sever knows which track the submission is for), and then zip it:

 zip -r submission.zip  my_class_sub_1.csv

 

An example submission file is provided in the starter kit. The evaluation server calculates the mean average precision (mAP) for the videos. That is, the average of the average precision (AP) for a single activity in all the videos.

Temporal Segmentation

For this task, participants should submit a zipped a csv file of the form:

        id   framenumber   vector
    

where 'id' is a unique video id, 'framenumber' is the number of the individual frame described below, and 'vector' is a whitespace delimited list of 157 floating point numbers representing the scores of each action in a frame.

Once you have this file, make sure that the string "loc" is in the filename (so the sever knows which track the submission is for), and then zip this file:

 zip -r submission.zip  my_loc_sub_1.csv

 

If you're submitting to both tracks, results can be uploaded together:

 zip -r submission.zip  my_loc_sub_1.csv my_class_sub_1.csv

 

An example submission file is provided in the starter kit (download this file with get_test_submission_localize.sh).

To avoid extremely large submission files, the evaluation script evaluates mAP on 25 equally spaced frames throughout each video. The frames are chosen as follows

 

        for j = 1, frames_per_video:
            timepoint(j) = (j-1)*video_duration/frames_per_video
    

We use frames_per_video = 25, and so a sequence of timepoints for a video is:

    [0, video_duration/25, 2*video_duration/25, ..., 24*video_duration/25]
    

 

VU Challenge

OFFICIAL RULES

Important: Please read these Official Rules before entering this Challenge (the "Contest"). By participating in this Contest, you agree to be bound by these Official Rules and represent that you satisfy all of the eligibility requirements below.

NO PURCHASE IS NECESSARY TO ENTER THIS CONTEST. Contest void where prohibited by law. Subject to applicable federal, state, and local laws.

1. Eligibility: Contest open only to individuals (and individuals representing teams) that meet the following requirement upon entry: Are at least 18 years old (or the age of majority in their jurisdiction of residence, whichever is older) at the time of entry.

Directors, officers, members, managers, and employees of Vulcan, Inc. ("Sponsor") and any of its respective parent companies, affiliate companies, subsidiaries, agents, professional advisors, advertising, promotional, public relations, and fulfillment agencies, consultants, legal counsel, website providers, web masters, and immediate family members (spouses, parent, child, sibling, grandparent, and "step" child, wherever they may live) of each are not eligible to win any prize. IF YOU DO NOT MEET ANY OF THESE REQUIREMENTS, OR ANY OTHER ELIGIBILITY REQUIREMENTS IN THESE OFFICIAL RULES, YOU ARE NOT ELIGIBLE TO WIN A PRIZE. To be eligible to win a prize, entries must be completed and received by Sponsor in the manner and format designated below.

2. Contest Period: The Contest starts at 11:59 p.m. Pacific Time (PT) on June 1st and ends at 11:59 p.m. PT on June 30th, 2017 (the "Contest Period"). All entries must be received during the Contest Period and meet other requirements in these Official Rules to be eligible to win a prize. Sponsor's or its designee's computer is the official time-keeping device for the Contest.

3. How to Enter: There will be two rounds of Submissions during the Contest Period and Submissions for each round will be scored separately. The first round will be the Validation Phase and the second round will be the Final Test Phase. To enter the Validation Phase, complete the following steps between 11:59 p.m. PT on June 1st and 11:59 p.m. PT on June 26, 2017: Round 1 - Validation Phase:

To enter the Charades Challenge:

  • Step 1: First, visit http://vuchallenge.org/Charades and follow the link to the competition page. At the Sign In screen, enter the username and password for your individual or team account. To create an account, click on the "Sign Up" link and follow the instructions to create a username and password. (There is no charge to create an account.)
  • Step 2: Next, click on the "Learn the Details" tab to review the prompts and Evaluation Criteria for the Multi-Label Video Classification Task and the Multi-Label Action Localization Task.
  • Step 3: Then, prepare a solution to the chosen subtask, either the Multi-Label Video Classification Task or the Multi-Label Action Localization Task.
  • Step 4: Finally, upload your solution to the evaluation server in the manner and format described in the Evaluation Criteria for the chosen subtask.


To enter the Textbook Answering Challenge:

  • Step 1: Step 1: First, visit http://vuchallenge.org/TQA and follow the link to the competition page. At the "Sign In" screen, enter the username and password for your individual or team account. To create an account, click on the "Sign Up" link and follow the instructions to create a username and password. (There is no charge to create an account.)
  • Step 2: Next, click on the "Learn the Details" tab to review the prompt and Evaluation Criteria for the Multi-Modal Machine Comprehension Task.
  • Step 3: Then, prepare a solution to the Multi-Modal Machine Comprehension Task.
  • Step 4: Finally, upload your solution to the evaluation server in the manner and format described in the Evaluation Criteria.

By submitting an entry during the Validation Phase as instructed above, you will be eligible to participate in the Final Test Phase. The Final Test Phase task will not be open to entrants until 11:59 p.m. PT on June 26, 2017. Round 2 - Final Test Phase: To enter the Final Test Phase, complete the following steps between 12:01 a.m. PT on June 26, 2017 and 11:59 p.m. on July 7, 2017:

To enter the THOR Challenge:

  • Step1: First, visit http://vuchallenge.org/THOR and click on the "Participate" tab. At the "Sign In" screen, enter the username and password for your individual or team account.
  • Step 2: Next, click on the "Learn the Details" tab top review the prompt and Evaluation Criteria for the Final Test Phase Task.
  • Step 3: Then, prepare a solution to the Final Test Phase Task.
  • Step 4: Finally, upload your solution to the evaluation server in the manner and format described in the


Evaluation Criteria before the expiration of the Contest Period. By submitting an entry or entries as instructed above ("Submission"), you agree that your Submission conforms to the Guidelines detailed below. Sponsor may choose to remove any Submission or disqualify you from the Contest if it believes that your Submission fails to conform with the Guidelines. Upon completion of all required entry steps and compliance with the Guidelines, you will automatically receive one entry into the Contest. Eligible participants may enter as one of the following categories of participants:

 

Individual Entries: If you enter as an individual, you must agree to these Official Rules and complete the entry process as instructed above.

Team Entries: If you enter to represent a team ("Team Leader"), you must agree on behalf of yourself and your team members that everyone on your team has read and agreed to the Official Rules, and that they agree to comply with the terms of this Contest. A Team Leader is solely responsible for obtaining all necessary consents and confirmations from each team member. A team's Submission must be made solely by the Team Leader. If a team is selected as a potential winner, each individual team member may be required to sign the Declaration of Eligibility and Liability/Publicity Release required by Sponsor (or Sponsor's designee) in order to be eligible to potentially win a prize. For any winning team, the prize will be given to the Team Leader who submitted the Submission for the team. Sponsor will have no responsibility thereafter for distribution of the prize to the individual team members.

Limit: LIMIT OF 20 ENTRIES PER INDIVIDUAL OR TEAM PER DAY DURING THE VALIDATION PHASE. LIMIT OF THREE ENTRIES PER INDIVIDUAL OR TEAM DURING THE FINAL TEST PHASE. However, no individual person may obtain more than 100 total entries into the Contest. For example, an individual is not permitted to enter 100 times on behalf of him or herself and then submit additional entries on behalf of his or her team. Likewise, no team may obtain more than 100 total Contest entries.

Submission Guidelines: You represent and warrant that: (i) the Submission does not violate any applicable law, rule, regulation, contract, or third-party rights; (ii) for team entries, that each of your team members has authorized the Submission and expressly agrees to these Official Rules; (iii) the Submission complies fully with the website terms of use, policies, and guidelines at http://www.vulcan.com/ and any other policies or guidelines specified by Sponsor in connection with the entry process; and (iv) the Submission complies fully with all requirements of these Official Rules.

4. Identity of Entrant: As described above in Section 3, all Submissions must be submitted by the individual entrant for an individual entry or the Team Leader for a team entry. Bulk or automated entries will be disqualified (including entries made using any script, macro, bot, or promotional service). Multiple participants are not permitted to share the same email account. Any attempt to obtain additional entries through fraud, bots, or other illegitimate means will result in disqualification, at Sponsor's sole discretion. If there is a dispute about the identity of the entrant, the associated entries will be declared made by the authorized account holder of the email address associated with the account on which the Submission is made. An "authorized account holder" means the natural person who is assigned to an e-mail address by an Internet access provider, online service provider or other organization (e.g., business, educational institution, etc.) that is responsible for assigning e-mail addresses for the domain associated with the submitted e-mail address. Each potential winner may be required to provide Sponsor with proof that the potential winner is the authorized account holder of the email account associated with the winning entry.

5. Selection and Notification of Winner: On or about July 7th, 2017, Sponsor will evaluate all eligible Submissions for the Multi-Label Video Classification Task, the Multi-Label Action Localization Task, and the Multi-Modal Machine Comprehension Task for the Validation Phase. Submissions for each task will be evaluated separately, with a separate leaderboard and winner for each task. On or about July 7, 2017, Sponsor will evaluate all eligible Submissions for the Final Test Phase Task. Sponsor will score each Submission based on the following criteria ("Scoring Criteria"):

  • TQA: Accuracy
  • Charades: Mean average precision
  • THOR: Average of the optimum number of actions


The top scoring Submission for each task will be deemed the potential winner, for a total of 3 potential winners. In the unlikely event of a tie, the winner will be determined by the Submission's score in the following criteria: performance on the validation set. Odds of winning a prize depend on number and quality of eligible entries received. Each potential winner will be notified via email on or about July 7, 2017. To the fullest extent permitted by applicable law, Sponsor may require each potential winner (including all team members if a team is selected as a potential winner) to sign and return, within seven days of being notified, an Affidavit of Eligibility and Liability/Publicity Release in order to claim the prize. If (i) the attempted notification is returned as undeliverable without a forwarding address; (ii) any required documents are not returned within seven days, such as the Affidavit of Eligibility and Liability/Publicity Release; or (iii) if potential winner is otherwise unable to accept the prize as stated, the prize will be forfeited and may be awarded to an alternate winner accordance with the Judging Criteria. Only one alternate winner will be selected for each prize. If, by reason of a print or other error, more prizes are claimed than the number set forth in these Official Rules, an alternate winner will be selected in accordance with the above selection process from among all eligible claimants making purportedly valid claims to award the prizes available. Verification of Potential Winner: EACH POTENTIAL WINNER IS SUBJECT TO VERIFICATION BY SPONSOR, WHOSE DECISIONS ARE FINAL AND BINDING IN ALL MATTERS RELATED TO THE CONTEST. AN ENTRANT IS NOT A PRIZE WINNER UNTIL ENTRANT'S ELIGIBILITY HAS BEEN VERIFIED AND ENTRANT HAS BEEN NOTIFIED THAT VERIFICATION IS COMPLETE AND NO FURTHER ACTION IS NEEDED.

 

6. Prize: Six winners will each receive $3,000. Total ARV of all prizes: $18,000. Prize Restrictions: ALL FEDERAL, STATE AND LOCAL TAXES ASSOCIATED WITH THE RECEIPT OR USE OF ANY PRIZE ARE THE SOLE RESPONSIBILITY OF THE WINNER. Prizes are not transferable. No substitutions or exchanges of any prize (including for cash) will be permitted, except that Sponsor reserves the right to substitute a prize of comparable or greater value for any prize. Sponsor will not replace any lost or stolen prizes. Only the number of prizes stated in these Official Rules are available to be won in the Contest.

7. Disclaimer: Sponsor, and any of its respective parent companies, subsidiaries, affiliates, directors, officers, professional advisors, consultants, contractors, legal counsel, public relations firms, employees and advertising, fulfillment and marketing agencies (collectively, the "Released Parties") will not be responsible for: (a) any late, lost, misrouted, garbled or distorted or damaged transmissions or entries; (b) phone, electronic, hardware, software, network, Internet, or other computer or communications-related malfunctions or failures beyond Sponsor or the Released Parties' reasonable control; (c) any Contest disruptions, injuries, losses or damages caused by events beyond the reasonable control of Sponsor or the Released Parties by non-authorized human intervention; or (d) any printing or typographical errors in any materials associated with the Contest.

8. Limitations of Liability; Releases: By entering the Contest, you release Sponsor and all Released Parties from any liability whatsoever, and waive any and all causes of action, related to any claims, costs, injuries, losses, or damages of any kind arising out of or in connection with the Contest or delivery, misdelivery, acceptance, possession, use of or inability to use any prize (including claims, costs, injuries, losses and damages related to rights of publicity or privacy, defamation or portrayal in a false light, whether intentional or unintentional), whether under a theory of contract, tort (including negligence), warranty or other theory. TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL THE SPONSOR OR THE RELEASED PARTIES BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOSS OF USE, LOSS OF PROFITS OR LOSS OF DATA, WHETHER IN AN ACTION IN CONTRACT, TORT (INCLUDING NEGLIGENCE) OR OTHERWISE, ARISING OUT OF OR IN ANY WAY CONNECTED TO YOUR PARTICIPATION IN THE CONTEST OR USE OR INABILITY TO USE ANY EQUIPMENT PROVIDED FOR USE IN THE CONTEST OR ANY PRIZE, EVEN IF A RELEASED PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL THE AGGREGATE LIABILITY OF THE RELEASED PARTIES (JOINTLY) ARISING OUT OF OR RELATING TO YOUR PARTICIPATION IN THE CONTEST OR USE OF OR INABILITY TO USE ANY EQUIPMENT PROVIDED FOR USE IN THE CONTEST OR ANY PRIZE EXCEED $10. THE LIMITATIONS SET FORTH IN THIS SECTION WILL NOT EXCLUDE OR LIMIT LIABILITY FOR PERSONAL INJURY OR PROPERTY DAMAGE CAUSED BY PRODUCTS RENTED FROM THE SPONSOR, OR FOR THE RELEASED PARTIES' GROSS NEGLIGENCE, INTENTIONAL MISCONDUCT, OR FOR FRAUD.

9. Use of Use of Winner's Name, Likeness, etc.: To the fullest extent permitted by applicable law, entry into the Contest constitutes permission to use your name, hometown, aural and visual likeness, persona, and prize information for advertising, marketing, and promotional purposes without further permission or compensation (including in a public-facing winner list). As a condition of being awarded any prize, except where prohibited by law, winner may be required to execute a consent to the use of their name, hometown, aural and visual likeness, persona, and prize information for advertising, marketing, and promotional purposes without further permission or compensation. By entering this Contest, you consent to being contacted by Sponsor for any purpose in connection with this Contest.

10. License to Submission: By entering the Contest and submitting your Submission to Sponsor, you grant Sponsor a temporary, internal use only, royalty-free, paid-up, personal, non-transferable, non-sublicensable right and license to use, copy, modify, execute and make derivative works of your Submission solely to judge the Contest and otherwise evaluate your Submission in connection with the Contest. Sponsor will discard your Submission in a reasonable time after the completion of the Contest. Notwithstanding the above, you agree and acknowledge that Sponsor may retain copy(s) of your Submission as an incidental result of its normal information technology backup processes. Sponsor will have no obligation to publish or use or retain any Submission you submit or to return any such Submission to you. You agree that it is your sole responsibility to obtain all permissions and releases necessary for the grant of the rights contained in this paragraph. You agree to take, at Sponsor's expense, any further action (including execution of affidavits, tax forms, and other documents) reasonably requested by Sponsor to effect, perfect or confirm Sponsor's rights as set forth above in this paragraph. You will not be entitled to compensation for any use by Sponsor, or its agents, licensees or assignees, of your Submission

11. Privacy: See Sponsor's Privacy Policy at http://www.vulcan.com/Privacy-Policy/.

12. Winner List; Rules Request: For a copy of the winner list, send an email to vuchallenge@allenai.org after July 14, 2017 and before July 31, 2018 to the address listed in Section 16 below, Attn: Vulcan Challenge. To obtain a copy of these Official Rules, visit [insert URL] or send a stamped, self-addressed business-size envelope to the address listed in Section 16 below, Attn: Vulcan Challenge. VT residents may omit return postage.

13. Intellectual Property Notice: VULCAN and the Vulcan Logo are trademarks of Vulcan, Inc. The Contest and all accompanying materials are copyright (c) 2017 by Vulcan, Inc. All rights reserved.

14. Disputes: EACH ENTRANT AGREES THAT ANY DISPUTES, CLAIMS, AND CAUSES OF ACTION ARISING OUT OF OR CONNECTED WITH THIS CONTEST OR ANY PRIZE AWARDED WILL BE RESOLVED INDIVIDUALLY, WITHOUT RESORT TO ANY FORM OF CLASS ACTION, AND EXCLUSIVELY BY THE UNITED STATES DISTRICT COURT FOR THE WESTERN DISTRICT OF WASHINGTON OR THE APPROPRIATE STATE COURT LOCATED IN KING COUNTY, WASHINGTON. THESE OFFICIAL RULES ARE GOVERNED BY THE LAWS OF THE STATE OF WASHINGTON, WITHOUT REGARD TO CHOICE OF LAW OR CONFLICT OF LAW RULES. YOU WAIVE ANY AND ALL OBJECTIONS TO JURISDICTION AND VENUE IN THESE COURTS AND HEREBY SUBMIT TO THE JURISDICTION OF THOSE COURTS.

15. General Conditions: Entrants agree to be bound by these Official Rules and by the decisions of Sponsor, which are final and binding in all respects. Sponsor reserves the right to change these Official Rules at any time, in its discretion, and to suspend or cancel the Contest or any entrant's participation in the Contest should viruses, bugs, unauthorized human intervention or other causes beyond Sponsor's reasonable control affect the administration, security or proper play of the Contest, or Sponsor otherwise becomes (in its discretion) incapable of running the Contest as planned. Entrants who violate these Official Rules, violate any law, rule or regulation in connection with participation in the Contest, tamper with the operation of the Contest or engage in any conduct that is detrimental or unfair to Sponsor, the Contest or any other entrant (as determined in Sponsor's discretion) are subject to disqualification from the Contest. Sponsor reserves the right to lock out persons whose eligibility is in question or who have been disqualified or are otherwise ineligible to enter the Contest. Sponsor's failure to enforce any term of these Official Rules will not constitute a waiver of that provision. Any provision of these Official Rules deemed unenforceable will be enforced to the extent permissible, and the remainder of these Official Rules will remain in effect. The use of the terms "include" or "including" in these Official Rules is illustrative and not limiting.

16. Sponsor: Vulcan, Inc., 505 5th Ave S #900, Seattle WA 98104.

Prizes

A prize of $3,000 will be awarded to the highest scoring entrant on each of the competition's two subtasks.

Development

Start: June 1, 2017, midnight

Description: This phase is for model development and to practice submitting to the competition (it uses the validation data to score)

Testing

Start: June 9, 2017, midnight

Description: This phase scores performance on the blind test set and will determine the winners

Competition Ends

July 16, 2017, midnight

You must be logged in to participate in competitions.

Sign In

Top Three

Rank Username Score
1 TheDonald 0.3641
2 gurkirtsingh 0.3039
3 stroud 0.2966