Competition

How to Participate In The Evaluation

Participation is a two-phase process:

1. Participate in the Development/Dry Run (optional but highly recommended)
2. Participate in the Evaluation

Participation in each phase is more or less the same:

1. Train a parser and run it on the appropriate test set.
2. Create an answer file for the test set containing the parses. The answer file must follow the appropriate form (the development and training data follows this form):
• The parses should be in the same order as the sentences in the test set.
• Extra whitespace and newlines within parses is ignored. That is, a parse may exist on one or multiple (contiguous) lines.
• There must be one or more empty lines in between parses.
• Any number of lines prefixed with "#" may come before or after a parse; these will be ignored.
3. Create a submission package. This is a .zip file with at least one but possibly more files in a flat hierarchy (no subdirectories).
• For the Development/Dry Run Phase, the news/forum test set must be named `nf_answer.txt.' The Biomedical test set must be named `bio_answer.txt.' A submission may contain either or both files.
• For the Evaluation Phase a single test set is used and must be named `bio_answer.txt.'
4. Navigate to the `Participate' tab and the `Submit/View Results' subtab. Enter any information into the box and click `Submit' to upload your submission package.
5. Refresh the page periodically until the status of your system is `Finished.' If something goes wrong you may wish to look at the various output logs to debug, including your scores.
• During the Development/Dry Run Phase you may resubmit unlimited multiple times.
• During the Evaluation Phase you may only submit twice, to discourage hillclimbing on the test data. Your last submission will be considered your official submission.

Evaluation Criteria

Please note that all evaluation criteria are subject to change at the whim of the task organizer

The primary trophy-determining metric for this subtask will be the automated metric Smatch, commit fda1c9ea564a142d2dd0a6455627e69348662c9b, the version from 2016-11-14, located at https://github.com/snowblink14/smatch. This is subject to change.

Other automated metrics, such as other versions of Smatch, and sub-evaluations such as Smatch calculated on specific parts of the corpus will also be displayed in the online submission system and may be part of SemEval workshop proceedings. These metrics are not official and not trophy-determining.

We welcome the proposal of other human and automated metrics for this task, since it is not at all clear that the above proposed methods are in fact the best way to evaluate systems. That being said, unless otherwise indicated by the task organizer, the trophy-determining metric is that listed above.

Terms and Conditions

By submitting to the 'Evaluation' phase of this track you agree to the public release of your submissions' scores at the SemEval 2017 workshop and in the associated publicly available proceedings, at the task organizer's discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and other metrics as the task organizer sees fit. You accept that the ultimate decision of metric choice and score value is that of the task organizer. You further agree that your system will be named according to the team name provided at the time of submission or to a suitable shorthand, as determined by the task organizer. You agree that the task organizer is under no obligation to release scores and that scores may be withheld if it is the task organizer's judgement that the submission was incomplete, deceptive, or violated the letter or spirit of the competition's rules. Inclusion or exclusion of a submission's scores is not an endorsement or unendorsement of a team or individual's submission, system, or science. You further acknowledge that all trophy-making decisions are made at the sole discretion of the task organizer and that the organizer may present zero or more trophies. The definition of what constitutes a trophy is up to the task organizer.