Competition

How to Participate In The Evaluation

Participation is a two-phase process:

1. Participate in the Development/Dry Run (optional but highly recommended)
2. Participate in the Evaluation

Participation in each phase is more or less the same:

1. Train a generation system and run it on the appropriate test set.
2. Create an answer file for the test set containing the sentences. The answer file must follow the appropriate form:
• The sentences should be in the same order as the AMRs in the test set.
• There should be a single line of text for each AMR and the total number of lines in the file should be equal to the number of AMRs.
• Automated scores are case insensitive but references are not tokenized.
• Unlike the parsing subtask, there may not be empty lines in between generated texts.
• Unlike the parsing subtask, lines prefixed with "#" or any other symbol will not be ignored.
3. Create a submission package. This is a .zip file with a single file named `answer.txt.'
4. Navigate to the `Participate' tab and the `Submit/View Results' subtab. Enter any information into the box and click `Submit' to upload your submission package.
5. Refresh the page periodically until the status of your system is `Finished.' If something goes wrong you may wish to look at the various output logs to debug, including your scores.
• During the Development/Dry Run Phase you may resubmit unlimited multiple times.
• During the Evaluation Phase you may only submit twice, to discourage hillclimbing on the test data. Your last submission will be considered your official submission.

Evaluation Criteria

Please note that all evaluation criteria are subject to change at the whim of the task organizer

The primary trophy-determining metric for this subtask will be a human judgement obtained by the union of (possibly empty) sets of SemEval participants, other NLP researchers, other individuals known to the task organizer, and crowdsourced workers. The TrueSkill algorithm, as described in the WMT 2016 findings paper, will be used to produce a numerical metric.

Automated metrics, which may include but are not limited to BLEU, will be used in the online submission system. These metrics are not official.

We welcome the proposal of human and automated metrics for this task, since it is not at all clear that the above proposed methods are in fact the best way to evaluate systems. That being said, unless otherwise indicated by the task organizer, the trophy-determining metric is that listed above.

Terms and Conditions

By submitting to the 'Evaluation' phase of this track you agree to the public release of your submissions' scores at the SemEval 2017 workshop and in the associated publicly available proceedings, at the task organizer's discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and other metrics as the task organizer sees fit. You accept that the ultimate decision of metric choice and score value is that of the task organizer. You further agree that your system will be named according to the team name provided at the time of submission or to a suitable shorthand, as determined by the task organizer. You agree that the task organizer is under no obligation to release scores and that scores may be withheld if it is the task organizer's judgement that the submission was incomplete, deceptive, or violated the letter or spirit of the competition's rules. Inclusion or exclusion of a submission's scores is not an endorsement or unendorsement of a team or individual's submission, system, or science. You further acknowledge that all trophy-making decisions are made at the sole discretion of the task organizer and that the organizer may present zero or more trophies. The definition of what constitutes a trophy is up to the task organizer.