Forums

> End of the Evaluation Phase

Dear participants,

the evaluation phase is now officially over and we'd like to thank everyone
who participated! We are very glad that so many people were interested in
our task and are looking forward to your system descriptions.

With this post, we want to inform you about the next steps. A couple of you
already realized that it is currently not possible to submit results for
the post-evaluation phase. This is intended, since we first want to check
all submissions again and publish the evaluation phase leader board, before
allowing submissions for the new phase. The leader board (and thus, the
final ranking) will be published on *February 5*. Please do check your
scores at that point and contact us if you think that something is wrong
and we'll try to find a solution. The gold data will be released on
*February 9*. After this date, we cannot accept any re-submissions or
corrections, even if you think that something went wrong with your system
during the evaluation phase.

While waiting for the results, we kindly ask everyone who participated to
fill out the participant survey here:
https://docs.google.com/forms/d/e/1FAIpQLSctJqTRGOvm6C85--aGI6vR_jBasjrsbPs7GbTAse8OUY-CcQ/viewform?usp=sf_link
In this survey, you have to briefly summarize the details of your model. We
need this information in order to write up a task description paper that
includes an overview of the participant systems and the methods and
resources that they used. Please take the time to fill this out until
*February 9*!

Afterwards, it is your responsibility to write up a 4-page system
description paper that describes the technology and (additional) data used
in your system in more detail. We encourage you to use the gold data,
which will be available by then, in order to identify interesting test
cases and provide a deeper (error/success) analysis of your model. The
submission is not anonymous. Style and submission guidelines can be found
on the SemEval website
(http://alt.qcri.org/semeval2018/index.php?id=papers). Please note that
system description papers are due on *February 26*. As mentioned before,
you will have to review other participants' papers -- we'll provide more
information on this step soon.

Of course, the drive to have a top-ranked submission can be productive, but
keep in mind that scoring among the top 3 is not everything. A thorough
analysis of your system and its performance on the data has a high value
for understanding the task and data better. Thus, reporting your results
and elaborating on how you achieved them is very important, irrespective of
your rank. We will not reject any papers as long as you provide clear and
adequate details of your submission -- thus SemEval is also a great place
to record negative results, i.e. ideas that seemed promising but did not
work out.

The most important points summed up again:
* Please fill out the participant form until *Feb 9*!
* The final ranking is released on *Feb 5*
* Gold data are released on *Feb 9*
* System description papers are due *Feb 26*

Thanks again for your participation!