THOR Challenge - CVPR 2017 Workshop

Organized by dustins - Current server time: Dec. 18, 2017, 3:24 a.m. UTC

Current

Development
June 1, 2017, midnight UTC

Next

Testing
June 26, 2017, midnight UTC

Thor Challenge 2017

In this challenge, the task is to navigate towards a set of objects in indoor scenes based only on visual input. AI2-THOR enables navigation in a set of near-photo-realistic indoor scenes, and we will use this framework for the challenge. The agents that the challenge participants provide receive visual input from THOR and should perform an action based on the visual input.

Track Information

 

The challenge has two settings: (1) Discrete (2) Continuous, with separate leader boards and winners for each.

Discrete Navigation: By default, the framework operates in discrete mode. This means that the agent navigates on a predefined grid. The agent is only able to look up and down in 30 degree increments and rotate in 90 degree increments. If an action is issued that pushes the agent into a wall, the agent will remain on the grid, but the lastActionSuccess value will be set to False.

Continuous Navigation: In continuous mode, the agent is free to rotate, look up/down and move in any amount. This allows the agent to collide with walls, move towards an object in fewer steps and rotate with greater precision.

Evaluation Criteria

The evaluation score is calculated as the average of the optimum number of actions (ot) over the predicted set of actions (pt) over all the tasks:
      
If the predicted set of actions do not place the agent in the correct location for a task, the score for that task will be zero. The tasks include finding objects in different configurations of a scene from different starting locations.

Submission Format

 

The participants should submit a file that includes the taken actions. The code below shows how to use the API. While running against the validation scenes, the target_found() method will never return True. The agent must decide when it has reached the goal and cease performing actions. Once the agent has iterated through all the targets defined in the targets file, you can save the actions to a JSON file and submit to the CodaLab challenge. If submitting to the discrete track, you should name this file submission.json. If submitting to the continious track, the file should be named submission_continuous.json. The file should then be zipped and uploaded:

zip -r submission.zip submission.json

You can submit to both tracks at the same time by zipping both submission files:

zip -r submission.zip submission.json submission_continuous.json

Submissions to the continuous track won't be scored on codalab but instead will evaluated and recorded on the leaderboard on the THOR competition page. If your score doesn't appear there within 1-2 days of submission, please contact us.


#!/usr/bin/env python
import robosims
import json
env = robosims.controller.ChallengeController(
    # Use unity_path=thor-201705011400-OSXIntel64.app/Contents/MacOS/thor-201705011400-OSXIntel64 for OSX
    unity_path='projects/thor-201705011400-Linux64',
    record_actions=True
)
env.start()
with open("thor-challenge-targets/targets-val.json") as f:
    t = json.loads(f.read())
    for target in t:
        env.initialize_target(target)
        # navigate to the target
        event = env.step(action=dict(action='MoveLeft'))

    env.save_actions('submission-discrete.json')

 

VU Challenge

OFFICIAL RULES

Important: Please read these Official Rules before entering this Challenge (the "Contest"). By participating in this Contest, you agree to be bound by these Official Rules and represent that you satisfy all of the eligibility requirements below.

NO PURCHASE IS NECESSARY TO ENTER THIS CONTEST. Contest void where prohibited by law. Subject to applicable federal, state, and local laws.

1. Eligibility: Contest open only to individuals (and individuals representing teams) that meet the following requirement upon entry: Are at least 18 years old (or the age of majority in their jurisdiction of residence, whichever is older) at the time of entry.

Directors, officers, members, managers, and employees of Vulcan, Inc. ("Sponsor") and any of its respective parent companies, affiliate companies, subsidiaries, agents, professional advisors, advertising, promotional, public relations, and fulfillment agencies, consultants, legal counsel, website providers, web masters, and immediate family members (spouses, parent, child, sibling, grandparent, and "step" child, wherever they may live) of each are not eligible to win any prize. IF YOU DO NOT MEET ANY OF THESE REQUIREMENTS, OR ANY OTHER ELIGIBILITY REQUIREMENTS IN THESE OFFICIAL RULES, YOU ARE NOT ELIGIBLE TO WIN A PRIZE. To be eligible to win a prize, entries must be completed and received by Sponsor in the manner and format designated below.

2. Contest Period: The Contest starts at 11:59 p.m. Pacific Time (PT) on June 1st and ends at 11:59 p.m. PT on June 30th, 2017 (the "Contest Period"). All entries must be received during the Contest Period and meet other requirements in these Official Rules to be eligible to win a prize. Sponsor's or its designee's computer is the official time-keeping device for the Contest.

3. How to Enter: There will be two rounds of Submissions during the Contest Period and Submissions for each round will be scored separately. The first round will be the Validation Phase and the second round will be the Final Test Phase. To enter the Validation Phase, complete the following steps between 11:59 p.m. PT on June 1st and 11:59 p.m. PT on June 26, 2017: Round 1 - Validation Phase:

To enter the Charades Challenge:

  • Step 1: First, visit http://vuchallenge.org/Charades and follow the link to the competition page. At the Sign In screen, enter the username and password for your individual or team account. To create an account, click on the "Sign Up" link and follow the instructions to create a username and password. (There is no charge to create an account.)
  • Step 2: Next, click on the "Learn the Details" tab to review the prompts and Evaluation Criteria for the Multi-Label Video Classification Task and the Multi-Label Action Localization Task.
  • Step 3: Then, prepare a solution to the chosen subtask, either the Multi-Label Video Classification Task or the Multi-Label Action Localization Task.
  • Step 4: Finally, upload your solution to the evaluation server in the manner and format described in the Evaluation Criteria for the chosen subtask.


To enter the Textbook Answering Challenge:

  • Step 1: Step 1: First, visit http://vuchallenge.org/TQA and follow the link to the competition page. At the "Sign In" screen, enter the username and password for your individual or team account. To create an account, click on the "Sign Up" link and follow the instructions to create a username and password. (There is no charge to create an account.)
  • Step 2: Next, click on the "Learn the Details" tab to review the prompt and Evaluation Criteria for the Multi-Modal Machine Comprehension Task.
  • Step 3: Then, prepare a solution to the Multi-Modal Machine Comprehension Task.
  • Step 4: Finally, upload your solution to the evaluation server in the manner and format described in the Evaluation Criteria.

By submitting an entry during the Validation Phase as instructed above, you will be eligible to participate in the Final Test Phase. The Final Test Phase task will not be open to entrants until 11:59 p.m. PT on June 26, 2017. Round 2 - Final Test Phase: To enter the Final Test Phase, complete the following steps between 12:01 a.m. PT on June 26, 2017 and 11:59 p.m. on July 7, 2017:

To enter the THOR Challenge:

  • Step1: First, visit http://vuchallenge.org/THOR and click on the "Participate" tab. At the "Sign In" screen, enter the username and password for your individual or team account.
  • Step 2: Next, click on the "Learn the Details" tab top review the prompt and Evaluation Criteria for the Final Test Phase Task.
  • Step 3: Then, prepare a solution to the Final Test Phase Task.
  • Step 4: Finally, upload your solution to the evaluation server in the manner and format described in the


Evaluation Criteria before the expiration of the Contest Period. By submitting an entry or entries as instructed above ("Submission"), you agree that your Submission conforms to the Guidelines detailed below. Sponsor may choose to remove any Submission or disqualify you from the Contest if it believes that your Submission fails to conform with the Guidelines. Upon completion of all required entry steps and compliance with the Guidelines, you will automatically receive one entry into the Contest. Eligible participants may enter as one of the following categories of participants:

 

Individual Entries: If you enter as an individual, you must agree to these Official Rules and complete the entry process as instructed above.

Team Entries: If you enter to represent a team ("Team Leader"), you must agree on behalf of yourself and your team members that everyone on your team has read and agreed to the Official Rules, and that they agree to comply with the terms of this Contest. A Team Leader is solely responsible for obtaining all necessary consents and confirmations from each team member. A team's Submission must be made solely by the Team Leader. If a team is selected as a potential winner, each individual team member may be required to sign the Declaration of Eligibility and Liability/Publicity Release required by Sponsor (or Sponsor's designee) in order to be eligible to potentially win a prize. For any winning team, the prize will be given to the Team Leader who submitted the Submission for the team. Sponsor will have no responsibility thereafter for distribution of the prize to the individual team members.

Limit: LIMIT OF 20 ENTRIES PER INDIVIDUAL OR TEAM PER DAY DURING THE VALIDATION PHASE. LIMIT OF THREE ENTRIES PER INDIVIDUAL OR TEAM DURING THE FINAL TEST PHASE. However, no individual person may obtain more than 100 total entries into the Contest. For example, an individual is not permitted to enter 100 times on behalf of him or herself and then submit additional entries on behalf of his or her team. Likewise, no team may obtain more than 100 total Contest entries.

Submission Guidelines: You represent and warrant that: (i) the Submission does not violate any applicable law, rule, regulation, contract, or third-party rights; (ii) for team entries, that each of your team members has authorized the Submission and expressly agrees to these Official Rules; (iii) the Submission complies fully with the website terms of use, policies, and guidelines at http://www.vulcan.com/ and any other policies or guidelines specified by Sponsor in connection with the entry process; and (iv) the Submission complies fully with all requirements of these Official Rules.

4. Identity of Entrant: As described above in Section 3, all Submissions must be submitted by the individual entrant for an individual entry or the Team Leader for a team entry. Bulk or automated entries will be disqualified (including entries made using any script, macro, bot, or promotional service). Multiple participants are not permitted to share the same email account. Any attempt to obtain additional entries through fraud, bots, or other illegitimate means will result in disqualification, at Sponsor's sole discretion. If there is a dispute about the identity of the entrant, the associated entries will be declared made by the authorized account holder of the email address associated with the account on which the Submission is made. An "authorized account holder" means the natural person who is assigned to an e-mail address by an Internet access provider, online service provider or other organization (e.g., business, educational institution, etc.) that is responsible for assigning e-mail addresses for the domain associated with the submitted e-mail address. Each potential winner may be required to provide Sponsor with proof that the potential winner is the authorized account holder of the email account associated with the winning entry.

5. Selection and Notification of Winner: On or about July 7th, 2017, Sponsor will evaluate all eligible Submissions for the Multi-Label Video Classification Task, the Multi-Label Action Localization Task, and the Multi-Modal Machine Comprehension Task for the Validation Phase. Submissions for each task will be evaluated separately, with a separate leaderboard and winner for each task. On or about July 7, 2017, Sponsor will evaluate all eligible Submissions for the Final Test Phase Task. Sponsor will score each Submission based on the following criteria ("Scoring Criteria"):

  • TQA: Accuracy
  • Charades: Mean average precision
  • THOR: Average of the optimum number of actions


The top scoring Submission for each task will be deemed the potential winner, for a total of 3 potential winners. In the unlikely event of a tie, the winner will be determined by the Submission's score in the following criteria: performance on the validation set. Odds of winning a prize depend on number and quality of eligible entries received. Each potential winner will be notified via email on or about July 7, 2017. To the fullest extent permitted by applicable law, Sponsor may require each potential winner (including all team members if a team is selected as a potential winner) to sign and return, within seven days of being notified, an Affidavit of Eligibility and Liability/Publicity Release in order to claim the prize. If (i) the attempted notification is returned as undeliverable without a forwarding address; (ii) any required documents are not returned within seven days, such as the Affidavit of Eligibility and Liability/Publicity Release; or (iii) if potential winner is otherwise unable to accept the prize as stated, the prize will be forfeited and may be awarded to an alternate winner accordance with the Judging Criteria. Only one alternate winner will be selected for each prize. If, by reason of a print or other error, more prizes are claimed than the number set forth in these Official Rules, an alternate winner will be selected in accordance with the above selection process from among all eligible claimants making purportedly valid claims to award the prizes available. Verification of Potential Winner: EACH POTENTIAL WINNER IS SUBJECT TO VERIFICATION BY SPONSOR, WHOSE DECISIONS ARE FINAL AND BINDING IN ALL MATTERS RELATED TO THE CONTEST. AN ENTRANT IS NOT A PRIZE WINNER UNTIL ENTRANT'S ELIGIBILITY HAS BEEN VERIFIED AND ENTRANT HAS BEEN NOTIFIED THAT VERIFICATION IS COMPLETE AND NO FURTHER ACTION IS NEEDED.

 

6. Prize: Six winners will each receive $3,000. Total ARV of all prizes: $18,000. Prize Restrictions: ALL FEDERAL, STATE AND LOCAL TAXES ASSOCIATED WITH THE RECEIPT OR USE OF ANY PRIZE ARE THE SOLE RESPONSIBILITY OF THE WINNER. Prizes are not transferable. No substitutions or exchanges of any prize (including for cash) will be permitted, except that Sponsor reserves the right to substitute a prize of comparable or greater value for any prize. Sponsor will not replace any lost or stolen prizes. Only the number of prizes stated in these Official Rules are available to be won in the Contest.

7. Disclaimer: Sponsor, and any of its respective parent companies, subsidiaries, affiliates, directors, officers, professional advisors, consultants, contractors, legal counsel, public relations firms, employees and advertising, fulfillment and marketing agencies (collectively, the "Released Parties") will not be responsible for: (a) any late, lost, misrouted, garbled or distorted or damaged transmissions or entries; (b) phone, electronic, hardware, software, network, Internet, or other computer or communications-related malfunctions or failures beyond Sponsor or the Released Parties' reasonable control; (c) any Contest disruptions, injuries, losses or damages caused by events beyond the reasonable control of Sponsor or the Released Parties by non-authorized human intervention; or (d) any printing or typographical errors in any materials associated with the Contest.

8. Limitations of Liability; Releases: By entering the Contest, you release Sponsor and all Released Parties from any liability whatsoever, and waive any and all causes of action, related to any claims, costs, injuries, losses, or damages of any kind arising out of or in connection with the Contest or delivery, misdelivery, acceptance, possession, use of or inability to use any prize (including claims, costs, injuries, losses and damages related to rights of publicity or privacy, defamation or portrayal in a false light, whether intentional or unintentional), whether under a theory of contract, tort (including negligence), warranty or other theory. TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL THE SPONSOR OR THE RELEASED PARTIES BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOSS OF USE, LOSS OF PROFITS OR LOSS OF DATA, WHETHER IN AN ACTION IN CONTRACT, TORT (INCLUDING NEGLIGENCE) OR OTHERWISE, ARISING OUT OF OR IN ANY WAY CONNECTED TO YOUR PARTICIPATION IN THE CONTEST OR USE OR INABILITY TO USE ANY EQUIPMENT PROVIDED FOR USE IN THE CONTEST OR ANY PRIZE, EVEN IF A RELEASED PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL THE AGGREGATE LIABILITY OF THE RELEASED PARTIES (JOINTLY) ARISING OUT OF OR RELATING TO YOUR PARTICIPATION IN THE CONTEST OR USE OF OR INABILITY TO USE ANY EQUIPMENT PROVIDED FOR USE IN THE CONTEST OR ANY PRIZE EXCEED $10. THE LIMITATIONS SET FORTH IN THIS SECTION WILL NOT EXCLUDE OR LIMIT LIABILITY FOR PERSONAL INJURY OR PROPERTY DAMAGE CAUSED BY PRODUCTS RENTED FROM THE SPONSOR, OR FOR THE RELEASED PARTIES' GROSS NEGLIGENCE, INTENTIONAL MISCONDUCT, OR FOR FRAUD.

9. Use of Use of Winner's Name, Likeness, etc.: To the fullest extent permitted by applicable law, entry into the Contest constitutes permission to use your name, hometown, aural and visual likeness, persona, and prize information for advertising, marketing, and promotional purposes without further permission or compensation (including in a public-facing winner list). As a condition of being awarded any prize, except where prohibited by law, winner may be required to execute a consent to the use of their name, hometown, aural and visual likeness, persona, and prize information for advertising, marketing, and promotional purposes without further permission or compensation. By entering this Contest, you consent to being contacted by Sponsor for any purpose in connection with this Contest.

10. License to Submission: By entering the Contest and submitting your Submission to Sponsor, you grant Sponsor a temporary, internal use only, royalty-free, paid-up, personal, non-transferable, non-sublicensable right and license to use, copy, modify, execute and make derivative works of your Submission solely to judge the Contest and otherwise evaluate your Submission in connection with the Contest. Sponsor will discard your Submission in a reasonable time after the completion of the Contest. Notwithstanding the above, you agree and acknowledge that Sponsor may retain copy(s) of your Submission as an incidental result of its normal information technology backup processes. Sponsor will have no obligation to publish or use or retain any Submission you submit or to return any such Submission to you. You agree that it is your sole responsibility to obtain all permissions and releases necessary for the grant of the rights contained in this paragraph. You agree to take, at Sponsor's expense, any further action (including execution of affidavits, tax forms, and other documents) reasonably requested by Sponsor to effect, perfect or confirm Sponsor's rights as set forth above in this paragraph. You will not be entitled to compensation for any use by Sponsor, or its agents, licensees or assignees, of your Submission

11. Privacy: See Sponsor's Privacy Policy at http://www.vulcan.com/Privacy-Policy/.

12. Winner List; Rules Request: For a copy of the winner list, send an email to vuchallenge@allenai.org after July 14, 2017 and before July 31, 2018 to the address listed in Section 16 below, Attn: Vulcan Challenge. To obtain a copy of these Official Rules, visit [insert URL] or send a stamped, self-addressed business-size envelope to the address listed in Section 16 below, Attn: Vulcan Challenge. VT residents may omit return postage.

13. Intellectual Property Notice: VULCAN and the Vulcan Logo are trademarks of Vulcan, Inc. The Contest and all accompanying materials are copyright (c) 2017 by Vulcan, Inc. All rights reserved.

14. Disputes: EACH ENTRANT AGREES THAT ANY DISPUTES, CLAIMS, AND CAUSES OF ACTION ARISING OUT OF OR CONNECTED WITH THIS CONTEST OR ANY PRIZE AWARDED WILL BE RESOLVED INDIVIDUALLY, WITHOUT RESORT TO ANY FORM OF CLASS ACTION, AND EXCLUSIVELY BY THE UNITED STATES DISTRICT COURT FOR THE WESTERN DISTRICT OF WASHINGTON OR THE APPROPRIATE STATE COURT LOCATED IN KING COUNTY, WASHINGTON. THESE OFFICIAL RULES ARE GOVERNED BY THE LAWS OF THE STATE OF WASHINGTON, WITHOUT REGARD TO CHOICE OF LAW OR CONFLICT OF LAW RULES. YOU WAIVE ANY AND ALL OBJECTIONS TO JURISDICTION AND VENUE IN THESE COURTS AND HEREBY SUBMIT TO THE JURISDICTION OF THOSE COURTS.

15. General Conditions: Entrants agree to be bound by these Official Rules and by the decisions of Sponsor, which are final and binding in all respects. Sponsor reserves the right to change these Official Rules at any time, in its discretion, and to suspend or cancel the Contest or any entrant's participation in the Contest should viruses, bugs, unauthorized human intervention or other causes beyond Sponsor's reasonable control affect the administration, security or proper play of the Contest, or Sponsor otherwise becomes (in its discretion) incapable of running the Contest as planned. Entrants who violate these Official Rules, violate any law, rule or regulation in connection with participation in the Contest, tamper with the operation of the Contest or engage in any conduct that is detrimental or unfair to Sponsor, the Contest or any other entrant (as determined in Sponsor's discretion) are subject to disqualification from the Contest. Sponsor reserves the right to lock out persons whose eligibility is in question or who have been disqualified or are otherwise ineligible to enter the Contest. Sponsor's failure to enforce any term of these Official Rules will not constitute a waiver of that provision. Any provision of these Official Rules deemed unenforceable will be enforced to the extent permissible, and the remainder of these Official Rules will remain in effect. The use of the terms "include" or "including" in these Official Rules is illustrative and not limiting.

16. Sponsor: Vulcan, Inc., 505 5th Ave S #900, Seattle WA 98104.

Prizes

A prize of $3,000 will be awarded to the highest scoring entrant on each of the competition's two subtasks.

Development

Start: June 1, 2017, midnight

Description: This phase is for model development and to practice submitting to the competition (it uses the validation data to score)

Testing

Start: June 26, 2017, midnight

Description: This phase scores performance on the blind test set and will determine the winners

Competition Ends

July 16, 2017, midnight

You must be logged in to participate in competitions.

Sign In

Top Three

Rank Username Score
1 JamesChuang 0.10144
2 AI2_admin 0.08646
3 phunghx 0.01455