Textbook Question Answering - CVPR 2017 Workshop

Organized by dustins - Current server time: Feb. 24, 2018, 12:17 p.m. UTC

Previous

Development
June 1, 2017, midnight UTC

Current

Testing
June 26, 2017, midnight UTC

End

Competition Ends
July 16, 2017, midnight UTC

Textbook Question Answering Challenge 2017


The TQA challenge encourages work on the task of Multi-Modal Machine Comprehension (M3C) task. The M3C task builds on the popular Visual Question Answering (VQA) and Machine Comprehension (MC) paradigms by framing question answering as a machine comprehension task, where the context needed to answer questions is provided and composed of both text and images. The dataset constructed to showcase this task has been built from a middle school science curriculum that pairs a given question to a limited span of knowledge needed to answer it. The phenomenon explained and tested on in middle school science can be fairly complex, and many questions require more than a simple look-up. Complex parsing of the source material and reasoning are often both required, as is connecting information provided jointly in text and diagrams. Our experiments have shown that extensions of the state-of-the-art methods from MC and VQA perform poorly on this task. New ideas and techniques are needed to address the challenges introduced by this dataset.

At a high level, the M3C task is to read a multi-modal context along with a multi-modal question and provide an answer. VQA tasks typically require common sense knowledge to answer many questions, in addition to the image itself. In contrast, the this task is meant to be self-contained. All of the information needed to answer a question is included in the context.

Track Information

The TQA dataset consists of two types of questions: Diagram Questions and Text Questions. These questions are distinguished by the modalities of a context needed to answer them.

Diagram Questions need an accompanying diagram to answer in addition to the the text and diagrammatic content in the context.
Text Questions can be answered with only the textual portion of context.


The TQA challenge has 2 tracks, one for each of these question types:
Track 1- Text Questions Track
Track 2- Diagram Questions Track

Separate leader boards are maintained for performance on Diagram and Text Questions, and winners will be declared for both. Diagram and text questions are easily separated in the dataset, and details for doing so are given here.

Researchers are greatly encouraged to submit a model that works well on both types of questions.

Evaluation Criteria

For the textbook question answering competition, accuracy is used as the evaluation metric. Separate leader boards are kept for performance on text-only questions, as well as diagram questions.

Submission Format

Participants should submit a zip file containing a single json with the string 'tqa' in its name, and a file of this form:

    {
	"Question_ID": "letter corresponding to answer choice"
    }

where the Question_ID is the global key of the question being answered, and the value is a lowercase letter denoting one of the multiple choice answer options for that question (e.g. a, b, c, ... ). The filename should contain the string "tqa" to be recognized by the evaluation server, but it is otherwise flexible.

 

 

Text only Question Answering

Evaluation and submission for this subtask are identical to the diagram question task, except that the text questions are scored. Questions requiring a diagram are not scored, and can be left out of the submission file if you would like to participate only in this subtask. Questions are separated by type in the dataset (nested under diagramQuestions and nonDiagramQuestions), and they also have global id prefixes to indicate an associated diagram (DQ_* and NDQ_*).

 

VU Challenge

OFFICIAL RULES

Important: Please read these Official Rules before entering this Challenge (the "Contest"). By participating in this Contest, you agree to be bound by these Official Rules and represent that you satisfy all of the eligibility requirements below.

NO PURCHASE IS NECESSARY TO ENTER THIS CONTEST. Contest void where prohibited by law. Subject to applicable federal, state, and local laws.

1. Eligibility: Contest open only to individuals (and individuals representing teams) that meet the following requirement upon entry: Are at least 18 years old (or the age of majority in their jurisdiction of residence, whichever is older) at the time of entry.

Directors, officers, members, managers, and employees of Vulcan, Inc. ("Sponsor") and any of its respective parent companies, affiliate companies, subsidiaries, agents, professional advisors, advertising, promotional, public relations, and fulfillment agencies, consultants, legal counsel, website providers, web masters, and immediate family members (spouses, parent, child, sibling, grandparent, and "step" child, wherever they may live) of each are not eligible to win any prize. IF YOU DO NOT MEET ANY OF THESE REQUIREMENTS, OR ANY OTHER ELIGIBILITY REQUIREMENTS IN THESE OFFICIAL RULES, YOU ARE NOT ELIGIBLE TO WIN A PRIZE. To be eligible to win a prize, entries must be completed and received by Sponsor in the manner and format designated below.

2. Contest Period: The Contest starts at 11:59 p.m. Pacific Time (PT) on June 1st and ends at 11:59 p.m. PT on June 30th, 2017 (the "Contest Period"). All entries must be received during the Contest Period and meet other requirements in these Official Rules to be eligible to win a prize. Sponsor's or its designee's computer is the official time-keeping device for the Contest.

3. How to Enter: There will be two rounds of Submissions during the Contest Period and Submissions for each round will be scored separately. The first round will be the Validation Phase and the second round will be the Final Test Phase. To enter the Validation Phase, complete the following steps between 11:59 p.m. PT on June 1st and 11:59 p.m. PT on June 26, 2017: Round 1 - Validation Phase:

To enter the Charades Challenge:

  • Step 1: First, visit http://vuchallenge.org/Charades and follow the link to the competition page. At the Sign In screen, enter the username and password for your individual or team account. To create an account, click on the "Sign Up" link and follow the instructions to create a username and password. (There is no charge to create an account.)
  • Step 2: Next, click on the "Learn the Details" tab to review the prompts and Evaluation Criteria for the Multi-Label Video Classification Task and the Multi-Label Action Localization Task.
  • Step 3: Then, prepare a solution to the chosen subtask, either the Multi-Label Video Classification Task or the Multi-Label Action Localization Task.
  • Step 4: Finally, upload your solution to the evaluation server in the manner and format described in the Evaluation Criteria for the chosen subtask.


To enter the Textbook Answering Challenge:

  • Step 1: Step 1: First, visit http://vuchallenge.org/TQA and follow the link to the competition page. At the "Sign In" screen, enter the username and password for your individual or team account. To create an account, click on the "Sign Up" link and follow the instructions to create a username and password. (There is no charge to create an account.)
  • Step 2: Next, click on the "Learn the Details" tab to review the prompt and Evaluation Criteria for the Multi-Modal Machine Comprehension Task.
  • Step 3: Then, prepare a solution to the Multi-Modal Machine Comprehension Task.
  • Step 4: Finally, upload your solution to the evaluation server in the manner and format described in the Evaluation Criteria.

By submitting an entry during the Validation Phase as instructed above, you will be eligible to participate in the Final Test Phase. The Final Test Phase task will not be open to entrants until 11:59 p.m. PT on June 26, 2017. Round 2 - Final Test Phase: To enter the Final Test Phase, complete the following steps between 12:01 a.m. PT on June 26, 2017 and 11:59 p.m. on July 7, 2017:

To enter the THOR Challenge:

  • Step1: First, visit http://vuchallenge.org/THOR and click on the "Participate" tab. At the "Sign In" screen, enter the username and password for your individual or team account.
  • Step 2: Next, click on the "Learn the Details" tab top review the prompt and Evaluation Criteria for the Final Test Phase Task.
  • Step 3: Then, prepare a solution to the Final Test Phase Task.
  • Step 4: Finally, upload your solution to the evaluation server in the manner and format described in the


Evaluation Criteria before the expiration of the Contest Period. By submitting an entry or entries as instructed above ("Submission"), you agree that your Submission conforms to the Guidelines detailed below. Sponsor may choose to remove any Submission or disqualify you from the Contest if it believes that your Submission fails to conform with the Guidelines. Upon completion of all required entry steps and compliance with the Guidelines, you will automatically receive one entry into the Contest. Eligible participants may enter as one of the following categories of participants:

 

Individual Entries: If you enter as an individual, you must agree to these Official Rules and complete the entry process as instructed above.

Team Entries: If you enter to represent a team ("Team Leader"), you must agree on behalf of yourself and your team members that everyone on your team has read and agreed to the Official Rules, and that they agree to comply with the terms of this Contest. A Team Leader is solely responsible for obtaining all necessary consents and confirmations from each team member. A team's Submission must be made solely by the Team Leader. If a team is selected as a potential winner, each individual team member may be required to sign the Declaration of Eligibility and Liability/Publicity Release required by Sponsor (or Sponsor's designee) in order to be eligible to potentially win a prize. For any winning team, the prize will be given to the Team Leader who submitted the Submission for the team. Sponsor will have no responsibility thereafter for distribution of the prize to the individual team members.

Limit: LIMIT OF 20 ENTRIES PER INDIVIDUAL OR TEAM PER DAY DURING THE VALIDATION PHASE. LIMIT OF THREE ENTRIES PER INDIVIDUAL OR TEAM DURING THE FINAL TEST PHASE. However, no individual person may obtain more than 100 total entries into the Contest. For example, an individual is not permitted to enter 100 times on behalf of him or herself and then submit additional entries on behalf of his or her team. Likewise, no team may obtain more than 100 total Contest entries.

Submission Guidelines: You represent and warrant that: (i) the Submission does not violate any applicable law, rule, regulation, contract, or third-party rights; (ii) for team entries, that each of your team members has authorized the Submission and expressly agrees to these Official Rules; (iii) the Submission complies fully with the website terms of use, policies, and guidelines at http://www.vulcan.com/ and any other policies or guidelines specified by Sponsor in connection with the entry process; and (iv) the Submission complies fully with all requirements of these Official Rules.

4. Identity of Entrant: As described above in Section 3, all Submissions must be submitted by the individual entrant for an individual entry or the Team Leader for a team entry. Bulk or automated entries will be disqualified (including entries made using any script, macro, bot, or promotional service). Multiple participants are not permitted to share the same email account. Any attempt to obtain additional entries through fraud, bots, or other illegitimate means will result in disqualification, at Sponsor's sole discretion. If there is a dispute about the identity of the entrant, the associated entries will be declared made by the authorized account holder of the email address associated with the account on which the Submission is made. An "authorized account holder" means the natural person who is assigned to an e-mail address by an Internet access provider, online service provider or other organization (e.g., business, educational institution, etc.) that is responsible for assigning e-mail addresses for the domain associated with the submitted e-mail address. Each potential winner may be required to provide Sponsor with proof that the potential winner is the authorized account holder of the email account associated with the winning entry.

5. Selection and Notification of Winner: On or about July 7th, 2017, Sponsor will evaluate all eligible Submissions for the Multi-Label Video Classification Task, the Multi-Label Action Localization Task, and the Multi-Modal Machine Comprehension Task for the Validation Phase. Submissions for each task will be evaluated separately, with a separate leaderboard and winner for each task. On or about July 7, 2017, Sponsor will evaluate all eligible Submissions for the Final Test Phase Task. Sponsor will score each Submission based on the following criteria ("Scoring Criteria"):

  • TQA: Accuracy
  • Charades: Mean average precision
  • THOR: Average of the optimum number of actions


The top scoring Submission for each task will be deemed the potential winner, for a total of 3 potential winners. In the unlikely event of a tie, the winner will be determined by the Submission's score in the following criteria: performance on the validation set. Odds of winning a prize depend on number and quality of eligible entries received. Each potential winner will be notified via email on or about July 7, 2017. To the fullest extent permitted by applicable law, Sponsor may require each potential winner (including all team members if a team is selected as a potential winner) to sign and return, within seven days of being notified, an Affidavit of Eligibility and Liability/Publicity Release in order to claim the prize. If (i) the attempted notification is returned as undeliverable without a forwarding address; (ii) any required documents are not returned within seven days, such as the Affidavit of Eligibility and Liability/Publicity Release; or (iii) if potential winner is otherwise unable to accept the prize as stated, the prize will be forfeited and may be awarded to an alternate winner accordance with the Judging Criteria. Only one alternate winner will be selected for each prize. If, by reason of a print or other error, more prizes are claimed than the number set forth in these Official Rules, an alternate winner will be selected in accordance with the above selection process from among all eligible claimants making purportedly valid claims to award the prizes available. Verification of Potential Winner: EACH POTENTIAL WINNER IS SUBJECT TO VERIFICATION BY SPONSOR, WHOSE DECISIONS ARE FINAL AND BINDING IN ALL MATTERS RELATED TO THE CONTEST. AN ENTRANT IS NOT A PRIZE WINNER UNTIL ENTRANT'S ELIGIBILITY HAS BEEN VERIFIED AND ENTRANT HAS BEEN NOTIFIED THAT VERIFICATION IS COMPLETE AND NO FURTHER ACTION IS NEEDED.

 

6. Prize: Six winners will each receive $3,000. Total ARV of all prizes: $18,000. Prize Restrictions: ALL FEDERAL, STATE AND LOCAL TAXES ASSOCIATED WITH THE RECEIPT OR USE OF ANY PRIZE ARE THE SOLE RESPONSIBILITY OF THE WINNER. Prizes are not transferable. No substitutions or exchanges of any prize (including for cash) will be permitted, except that Sponsor reserves the right to substitute a prize of comparable or greater value for any prize. Sponsor will not replace any lost or stolen prizes. Only the number of prizes stated in these Official Rules are available to be won in the Contest.

7. Disclaimer: Sponsor, and any of its respective parent companies, subsidiaries, affiliates, directors, officers, professional advisors, consultants, contractors, legal counsel, public relations firms, employees and advertising, fulfillment and marketing agencies (collectively, the "Released Parties") will not be responsible for: (a) any late, lost, misrouted, garbled or distorted or damaged transmissions or entries; (b) phone, electronic, hardware, software, network, Internet, or other computer or communications-related malfunctions or failures beyond Sponsor or the Released Parties' reasonable control; (c) any Contest disruptions, injuries, losses or damages caused by events beyond the reasonable control of Sponsor or the Released Parties by non-authorized human intervention; or (d) any printing or typographical errors in any materials associated with the Contest.

8. Limitations of Liability; Releases: By entering the Contest, you release Sponsor and all Released Parties from any liability whatsoever, and waive any and all causes of action, related to any claims, costs, injuries, losses, or damages of any kind arising out of or in connection with the Contest or delivery, misdelivery, acceptance, possession, use of or inability to use any prize (including claims, costs, injuries, losses and damages related to rights of publicity or privacy, defamation or portrayal in a false light, whether intentional or unintentional), whether under a theory of contract, tort (including negligence), warranty or other theory. TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL THE SPONSOR OR THE RELEASED PARTIES BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOSS OF USE, LOSS OF PROFITS OR LOSS OF DATA, WHETHER IN AN ACTION IN CONTRACT, TORT (INCLUDING NEGLIGENCE) OR OTHERWISE, ARISING OUT OF OR IN ANY WAY CONNECTED TO YOUR PARTICIPATION IN THE CONTEST OR USE OR INABILITY TO USE ANY EQUIPMENT PROVIDED FOR USE IN THE CONTEST OR ANY PRIZE, EVEN IF A RELEASED PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL THE AGGREGATE LIABILITY OF THE RELEASED PARTIES (JOINTLY) ARISING OUT OF OR RELATING TO YOUR PARTICIPATION IN THE CONTEST OR USE OF OR INABILITY TO USE ANY EQUIPMENT PROVIDED FOR USE IN THE CONTEST OR ANY PRIZE EXCEED $10. THE LIMITATIONS SET FORTH IN THIS SECTION WILL NOT EXCLUDE OR LIMIT LIABILITY FOR PERSONAL INJURY OR PROPERTY DAMAGE CAUSED BY PRODUCTS RENTED FROM THE SPONSOR, OR FOR THE RELEASED PARTIES' GROSS NEGLIGENCE, INTENTIONAL MISCONDUCT, OR FOR FRAUD.

9. Use of Use of Winner's Name, Likeness, etc.: To the fullest extent permitted by applicable law, entry into the Contest constitutes permission to use your name, hometown, aural and visual likeness, persona, and prize information for advertising, marketing, and promotional purposes without further permission or compensation (including in a public-facing winner list). As a condition of being awarded any prize, except where prohibited by law, winner may be required to execute a consent to the use of their name, hometown, aural and visual likeness, persona, and prize information for advertising, marketing, and promotional purposes without further permission or compensation. By entering this Contest, you consent to being contacted by Sponsor for any purpose in connection with this Contest.

10. License to Submission: By entering the Contest and submitting your Submission to Sponsor, you grant Sponsor a temporary, internal use only, royalty-free, paid-up, personal, non-transferable, non-sublicensable right and license to use, copy, modify, execute and make derivative works of your Submission solely to judge the Contest and otherwise evaluate your Submission in connection with the Contest. Sponsor will discard your Submission in a reasonable time after the completion of the Contest. Notwithstanding the above, you agree and acknowledge that Sponsor may retain copy(s) of your Submission as an incidental result of its normal information technology backup processes. Sponsor will have no obligation to publish or use or retain any Submission you submit or to return any such Submission to you. You agree that it is your sole responsibility to obtain all permissions and releases necessary for the grant of the rights contained in this paragraph. You agree to take, at Sponsor's expense, any further action (including execution of affidavits, tax forms, and other documents) reasonably requested by Sponsor to effect, perfect or confirm Sponsor's rights as set forth above in this paragraph. You will not be entitled to compensation for any use by Sponsor, or its agents, licensees or assignees, of your Submission

11. Privacy: See Sponsor's Privacy Policy at http://www.vulcan.com/Privacy-Policy/.

12. Winner List; Rules Request: For a copy of the winner list, send an email to vuchallenge@allenai.org after July 14, 2017 and before July 31, 2018 to the address listed in Section 16 below, Attn: Vulcan Challenge. To obtain a copy of these Official Rules, visit [insert URL] or send a stamped, self-addressed business-size envelope to the address listed in Section 16 below, Attn: Vulcan Challenge. VT residents may omit return postage.

13. Intellectual Property Notice: VULCAN and the Vulcan Logo are trademarks of Vulcan, Inc. The Contest and all accompanying materials are copyright (c) 2017 by Vulcan, Inc. All rights reserved.

14. Disputes: EACH ENTRANT AGREES THAT ANY DISPUTES, CLAIMS, AND CAUSES OF ACTION ARISING OUT OF OR CONNECTED WITH THIS CONTEST OR ANY PRIZE AWARDED WILL BE RESOLVED INDIVIDUALLY, WITHOUT RESORT TO ANY FORM OF CLASS ACTION, AND EXCLUSIVELY BY THE UNITED STATES DISTRICT COURT FOR THE WESTERN DISTRICT OF WASHINGTON OR THE APPROPRIATE STATE COURT LOCATED IN KING COUNTY, WASHINGTON. THESE OFFICIAL RULES ARE GOVERNED BY THE LAWS OF THE STATE OF WASHINGTON, WITHOUT REGARD TO CHOICE OF LAW OR CONFLICT OF LAW RULES. YOU WAIVE ANY AND ALL OBJECTIONS TO JURISDICTION AND VENUE IN THESE COURTS AND HEREBY SUBMIT TO THE JURISDICTION OF THOSE COURTS.

15. General Conditions: Entrants agree to be bound by these Official Rules and by the decisions of Sponsor, which are final and binding in all respects. Sponsor reserves the right to change these Official Rules at any time, in its discretion, and to suspend or cancel the Contest or any entrant's participation in the Contest should viruses, bugs, unauthorized human intervention or other causes beyond Sponsor's reasonable control affect the administration, security or proper play of the Contest, or Sponsor otherwise becomes (in its discretion) incapable of running the Contest as planned. Entrants who violate these Official Rules, violate any law, rule or regulation in connection with participation in the Contest, tamper with the operation of the Contest or engage in any conduct that is detrimental or unfair to Sponsor, the Contest or any other entrant (as determined in Sponsor's discretion) are subject to disqualification from the Contest. Sponsor reserves the right to lock out persons whose eligibility is in question or who have been disqualified or are otherwise ineligible to enter the Contest. Sponsor's failure to enforce any term of these Official Rules will not constitute a waiver of that provision. Any provision of these Official Rules deemed unenforceable will be enforced to the extent permissible, and the remainder of these Official Rules will remain in effect. The use of the terms "include" or "including" in these Official Rules is illustrative and not limiting.

16. Sponsor: Vulcan, Inc., 505 5th Ave S #900, Seattle WA 98104.

Prizes

A prize of $3,000 will be awarded to the highest scoring entrant on each of the competition's two subtasks.

Development

Start: June 1, 2017, midnight

Description: This phase is for model development and to practice submitting to the competition (it uses the validation data to score)

Testing

Start: June 26, 2017, midnight

Description: This phase scores performance on the blind test set and will determine the winners

Competition Ends

July 16, 2017, midnight

You must be logged in to participate in competitions.

Sign In

Top Three

Rank Username Score
No data