Forums

> Important: Some final remarks before the evaluation phase

Dear participants,

as the beginning of the evaluation phase is drawing closer, we are writing
to make a few final announcements and to remind you about some details:

* The evaluation phase starts on January 8 with the release of the test
data and ends on January 29. Results submitted after January 29 will be
posted on a post-evaluation phase leader board and are *not* considered
for the shared task ranking.
* Only the *last* submitted result per team/participant during the
evaluation phase will count towards the shared task ranking.
* Our evaluation script will *not* provide results for your submission
during the evaluation phase. The leader board will be *private*, so you
will not see your own results nor the results of other participants.
* We are planning to allow *2 submissions per day*, and a total of 40
submissions per participant/team.

PLEASE also read the following *important* notes carefully:

*Important 1*: If you plan to participate in the evaluation phase, we
*strongly* recommend to make a test submission during the practice phase
in order to get familiar with the submission format. If a submission
during the evaluation phase fails, this will not count towards your limit,
and there will be an error message telling you what went wrong. However,
the evaluation script cannot account for all types of errors, so *you*
should check if your submission results in reasonable numbers. As a rough
guideline, we found a simple baseline using word overlap to achieve 56%
accuracy, while a human upper bound lies at about 98%.

During the evaluation phase, we will monitor submissions and contact
participants if we suspect a submission to be faulty. After the evaluation
phase, we will make the leader board public and we will release the gold
standard annotation. Please check your performance at that point and
contact us *immediately* if you think that something went wrong.

*Important 2:* By submitting a system description paper after the
evaluation phase, you will agree to reviewing system descriptions by two
other task participants. The goal of this peer-review process is to ensure
the quality of submitted papers.

If there are any open questions, please don't hesitate to drop us a message.

Last but not least, we wish all participants a happy holiday season!