Evaluation

For development purposes, we provide an evaluation script here. The script takes two files as input, a gold-standard file (such as trial.labels) and a prediction file in the same format.

Official Competition Metric: the evaluation will be based on macro-averaged F1-score. Secondary Evaluation Metrics: Apart from the official competition metric described above, some additional metrics will also be calculated for your submissions. These are intended to provide a different perspective on the results:

• Micro Precision
• Micro Recall
• Micro F1-score
• Macro Precision
• Macro Recall

Terms and Conditions

By participating in this task you agree to these terms and conditions. If, however, one or more of this conditions is a concern for you, send us an email and we will consider if an exception can be made.

By submitting results to this competition, you consent to the public release of your scores at this website, at the WASSA 2018 website, the Codalab website and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers. You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science. A participant can be involved in exactly one team (no more). If there are reasons why it makes sense for you to be on more than one team, then email us before the evaluation period begins. In special circumstances this may be allowed.

Each team must create and use exactly one CodaLab account.

Team constitution (members of a team) cannot be changed after the evaluation period has begun. No participant can be part of more than one team.

During the evaluation period:

Each team can submit as many as fifty submissions. However, only the final submission will be considered as the official submission to the competition.

You will not be able to see results of your submission on the test set.

You will be able to see any warnings and errors for each of your submission.

Leaderboard is disabled

Once the competition is over, we will release the gold labels and you will be able to determine results on various system variants you may have developed. We encourage you to report results on all of your systems (or system variants) in the system-description paper. However, we will ask you to clearly indicate the result of your official submission.

We will make the final submissions of the teams public at some point after the evaluation period.

The organizers and their affiliated institutions makes no warranties regarding the datasets provided, including but not limited to being correct or complete. They cannot be held liable for providing access to the datasets or the usage of the datasets.

The dataset should only be used for scientific or research purposes. Any other use is explicitly prohibited.

The datasets must not be redistributed or shared in part or full with any third party. Redirect interested parties to this website.

If you use any of the datasets provided in the shared task, you agree to cite the associated paper. Information will be provided later.

You can contact all organizers of the shared task at iest@wassa2018.com

Organizers of the shared task:

Roman Klinger, Evgeny Kim
Institut für Maschinelle Sprachverarbeitung
University of Stuttgart
roman.klinger@ims.uni-stuttgart.de, evgeny.kim@ims.uni-stuttgart.de

Alexandra Balahur
European Commission Joint Research Centre
Directorate I - Competences Text and Data Mining Unit (I3)
alexandra.balahur@jrc.ec.europa.eu

Saif M. Mohammad
National Research Council Canada
saif.mohammad@nrc-cnrc.gc.ca

Veronique Hoste, Orphee de Clercq
Ghent University, Department of Translation, Interpreting and Communication
LT³ - Language and Translation Technology Team
veronique.hoste@ugent.be, orphee.declercq@ugent.be

Sample, Training, Development and Test Datasets

• Sample data (published on 2018-02-08, disclaimer: the preprocessing of the data changed from sample to training, trial and test data.)
• Training data (updated version 2018-06-18)
  • To access the training data, fill the form at https://goo.gl/forms/HXe6rloBkhbsdv6l2. Make sure that you mention your real name and affiliation. An individual password will be sent to you a couple of days after you filled the form.
  • Tweet-IDs will be made publicly available after the shared task.
• Development data (updated version 2018-06-18)
• Test data without labels (available)

Submission format

System submissions for CodaLab are zip-compressed folders containing a predictions file called predictions.txt.

The evaluation script will check whether the file contains the correct number of instances

IMPORTANT: a lot of competitions are run on CodaLab, and just a certain number of submissions can be handled at a given time, due to which your submission may be 'stuck' (i.e. status remains 'submitted' even after refreshing) for a certain time. In this case, patiently try again until your submission does get processed. Please make sure that you do not wait until the very last moment to submit your final system to avoid stress and missing the deadline.

Training and Evaluation phase

During the training phase (now - July 2, 2018), teams can upload a submission by means of development. They can upload predictions for all instances in the development data in the same way as for the official evaluation phase. During the training phase, submissions will be evaluated against the gold-standard labels of the development data.

Find below a step-by-step guideline to upload your submission on CodaLab during both the development and evaluation phase:

• the output of your system should be saved in a file named "predictions.txt". We expect a file in which the first column (tab separated from everything else) contains the prediction labels as strings in the right order. Everything else than this one column is optional and will be ignored.
• compress this file into a zipfile "predictions.zip" (for Mac users: ensure that this zipfile contains no _macosx file).
• on CodaLab, navigate to 'Participate'zip > Submit/View Results and upload your "predictions.zip" file
• click 'Refresh status' until your submission receives the "Finished" status
• click 'Submit to leaderboard' to push your results to the official scoring board. Please note: as soon as the official evaluation period starts, the scoring board will not be made visible until the official announcements of the final results (16th of July 2018).

Schedule:

• 9th of February 2018: Publication of sample data set
• 15th of March: Publication of training data, development data, and evaluation script
• 1st of June: Codalab competition website goes online (latest)
• 2nd of July 2018: Evaluation period/phase begins (test data released)
• 9th of July 2018: Evaluation period/phase ends
• 16th of July 2018: Announcement of final results
• 6th of August 2018: Submission of system description papers for review
• 17th of August 2018: Reviews due (teams review each others submissions)
• 31st of August 2018: Camera Ready Deadline of System Description: (same deadline as WASSA main workshop papers)
• 31st of October 2018 or 1st of November 2018: Workshop in Brussels at EMNLP 2018

Step 1: Submit your team's predictions

Submit the prediction through the CodaLab platform before 9 July 23:59 UCT

More information about the submission format can be found here.

Step 2: Create a START account

All teams can write a system description paper and submit it to the WASSA 2018 workshop. But first, all team members are to make sure they have a softconf account. New softconf users can register for an account by following the new user link here: https://www.softconf.com/emnlp2018/WASSA-2018/

Step 3: Fill in the Google Form (ASAP)

In order for the task organizers to write a task description paper, they first require some additional information from each team. To this purpose, a Google Form has been created. As soon as the team has submitted predictions and every team member has created a START account, the team leader can access the Google Form and fill in all the necessary information.

Please do this as soon as possible, preferably immediately after you submitted your results and before July 10, 2018.

Step 4: Check your results

On the 16th of July, we will announce the final results and make the gold standard test labels available.

Please note that we will only include teams in the ranking that have actually filled in the Google Form.

Step 5: Write your system description paper

You have time to write your system description papers until the 6th of August 2018. All papers should follow the EMNLP 2018 style files (LaTeX or Word). You can submit your paper by clicking on "Make a new Submission" on the WASSA softconf page, and then select the "WASSA shared task".

Please note that all author names should be included not only in the system paper (pdf), but also in the softconf (START) form when you are uploading paper information. When you click on the link to upload your paper on softconf, first you enter your start login details to login, then you will see a page where you are to enter meta information about the paper, like title, list of authors etc. At that point, you first click "add self" to add yourself, and then add all authors (one by one) via their start account id. Fill in other details. In the same page there is a button to upload the pdf.

We thank you for your participation!