Dear participants,
The official results are available at the following GitHub page:
https://github.com/habernal/semeval2018-task12-results/tree/master
We opted for publishing the results this way as it gives everybody a full open access to the final submissions as well as evaluation scripts, detailed error analysis, and so on. The gold labels will be released on the same page soon.
Should you have any questions, comments, or suggestions, don't hesitate to contact us.
The next steps are as follows:
* Each task participant team should write a 4-to-6-page system description paper. The paper should describe the technology used in the system, highlighting any novel contributions, and should use the provided gold data to analyze interesting (both positive and negative) behavior. Submission and format details are provided here: http://alt.qcri.org/semeval2018/index.php?id=papers. Note that SemEval submission is not anonymous; author names should be included. Papers are due Mon., 26 February by 23:59 GMT -12:00
* Each task participant should review other teams’ system description papers, using the START system. We (organizers) will be responsible for enrolling participants as reviewers, assigning reviewers to papers, and ensuring that reviews are completed in a timely manner. Reviews are due Mon., 19 Mar.
* SemEval seeks to have all participants publish a paper. In the rare event that a paper is deemed unacceptable in its current form, reviewers should outline a list of errors that must be fixed and you should contact system authors and help them to get their papers into shape. After this we (organizers) will notify system paper authors of their acceptance. This should be done by Mon., 2 Apr. Camera-ready papers are due Mon., 16 Apr.
* All task participant teams should prepare a poster for display at SemEval. Selected teams will be asked to prepare a short talk. Details will be provided at a later date.
Finally, while the drive to have the top-ranked submission can be productive, it is worth noting that it is not everything. More important is the analysis to help improve our collective understanding of the task. Thus, irrespective of one’s rank, it is useful to test different hypotheses and report results. All papers are accepted except for system papers that fail to provide clear and adequate details of their submission. Thus SemEval is also a great place to record negative results -- ideas that seemed promising but did not work out.
Thanks everybody for taking part in this tough competition!
Ivan
Posted by: ivan.habernal @ Feb. 7, 2018, 12:30 p.m.Hi,
I was wondering about the length of the system description paper. As I recall from previous years, SemEval is pretty relaxed about the paper length.
However, this year, http://alt.qcri.org/semeval2018/index.php?id=papers lists exactely four pages as the length of a system description paper outlining a single system. Does your 4-6 pags limit take preference or the general 4 page limit? And is there going to be additional space for addressing comments/reviews?
Best regards
Matthias
Dear all,
the general page limit of a system description paper is 4 pages. However, there may be reasons for why more pages are needed, as also described here:
http://alt.qcri.org/semeval2018/index.php?id=papers
If you have other reasons for needing up to 6 pages, let me know. We can make exceptions where appropriate.
Similar holds for addressing comments from the reviews. Usually, you should try to still comply with the given page limit. But if there are reasons for why this doesn't seem possible, let me know.
Hope that helps!
Henning
Dear Henning,
Is it OK if the Conclusion spills onto the references page (i.e. the 5th page)? (That is, no 6th page).
And if we wished to provide a comprehensive results table, are we allowed an appendix page for that?
Posted by: hanshan @ March 4, 2018, 3:47 p.m.Our conclusion and acknowledgement sections also spill over into the left column of page 5 due to the multiple figures in our paper. We hope that is OK.
Posted by: Liebeck @ March 5, 2018, 9:11 a.m.Dear all,
my deep apologies for replying too late (i.e., after the deadline) to your last requests. As a result of the sudden change of responsibilities on our side on the last mile, we oversaw that I do not get informed myself about new forum entries.
I checked all submissions, finding three that slightly exceed the 4-page limit. This will not lead to rejection, so no worries.
Besides I'm not fully sure whether all of you are already aware of the review process now to come. In particular, each of you will be assigned papers from the other participants for peer-reviewing. Details on the process will follow soon in a separate message.
Sorry again and best regards,
Henning
No worries, thanks, Henning!
Posted by: hanshan @ March 6, 2018, 10:25 p.m.