Emotion is a concept that is challenging to describe. Yet, as human beings, we understand the emotional effect situations have or could have on us and other people. How can we transfer this knowledge to machines? Is it possible to learn the link between situations and the emotions they trigger in an automatic way?
In the light of these questions, we propose the Shared Task on Implicit Emotion Recognition, organized as part of WASSA 2018 at EMNLP 2018 aims at developing models which can classify a text into one of the following emotions: Anger, Fear, Sadness, Joy, Surprise, Disgust without having access to an explicit mention of an emotion word.
Join the official task mailing group: firstname.lastname@example.org
Please join the mailing list to receive the latest news and updates. Also note that even if you join the mailing now, you will be able to see all previous messages.
You will be given a tweet from which a certain emotion word is removed. That word is one of the following: "sad", "happy", "disgusted", "surprised", "angry", "afraid" or a synonym of one of them. Your task is to predict the emotion the excluded word expresses: Sadness, Joy, Disgust, Surprise, Anger, or Fear.
With this formulation of the task, we provide data instances which are likely to express an emotion. However, the emotion needs to be inferred from the causal description, which is typically more implicit than an emotion word. We therefore presume that successful systems will take into account world knowledge in a structured or statistical manner.
"It's [#TARGETWORD#] when you feel like you are invisible to others."
"My step mom got so [#TARGETWORD#] when she came home from work and saw that the boys didn't come to Austin with me."
"We are so #[#TARGETWORD#] that people must think we are on good drugs or just really good actors."
The shared task consists of the challenge to build a model which recognizes that [#TARGETWORD#] corresponds to sadness ("sad") in the first two examples and with joy ("happy") in the third.
Participants will be given the opportunity to write a system-description paper that describes their system, resources used, results, and analysis. This paper will be part of the official WASSA-2018 proceedings. The paper is to be four pages long plus two pages at most for references and should be submitted using the EMNLP 2018 Style Files (LaTeX style files or Word template).
Official Competition Metric: the evaluation will be based on macro-averaged F1-score. Secondary Evaluation Metrics: Apart from the official competition metric described above, some additional metrics will also be calculated for your submissions. These are intended to provide a different perspective on the results:
By participating in this task you agree to these terms and conditions. If, however, one or more of this conditions is a concern for you, send us an email and we will consider if an exception can be made.
By submitting results to this competition, you consent to the public release of your scores at this website, at the WASSA 2018 website, the Codalab website and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers. You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science. A participant can be involved in exactly one team (no more). If there are reasons why it makes sense for you to be on more than one team, then email us before the evaluation period begins. In special circumstances this may be allowed.
Each team must create and use exactly one CodaLab account.
Team constitution (members of a team) cannot be changed after the evaluation period has begun. No participant can be part of more than one team.
During the evaluation period:
Each team can submit as many as fifty submissions. However, only the final submission will be considered as the official submission to the competition.
You will not be able to see results of your submission on the test set.
You will be able to see any warnings and errors for each of your submission.
Leaderboard is disabled
Once the competition is over, we will release the gold labels and you will be able to determine results on various system variants you may have developed. We encourage you to report results on all of your systems (or system variants) in the system-description paper. However, we will ask you to clearly indicate the result of your official submission.
We will make the final submissions of the teams public at some point after the evaluation period.
The organizers and their affiliated institutions makes no warranties regarding the datasets provided, including but not limited to being correct or complete. They cannot be held liable for providing access to the datasets or the usage of the datasets.
The dataset should only be used for scientific or research purposes. Any other use is explicitly prohibited.
The datasets must not be redistributed or shared in part or full with any third party. Redirect interested parties to this website.
If you use any of the datasets provided in the shared task, you agree to cite the associated paper. Information will be provided later.
You can contact all organizers of the shared task at email@example.com
Organizers of the shared task:
Roman Klinger, Evgeny Kim
Institut für Maschinelle Sprachverarbeitung
University of Stuttgart
European Commission Joint Research Centre
Directorate I - Competences Text and Data Mining Unit (I3)
Saif M. Mohammad
National Research Council Canada
Veronique Hoste, Orphee de Clercq
Ghent University, Department of Translation, Interpreting and Communication
LT³ - Language and Translation Technology Team
System submissions for CodaLab are zip-compressed folders containing a predictions file called predictions.txt.
The evaluation script will check whether the file contains the correct number of instances
IMPORTANT: a lot of competitions are run on CodaLab, and just a certain number of submissions can be handled at a given time, due to which your submission may be 'stuck' (i.e. status remains 'submitted' even after refreshing) for a certain time. In this case, patiently try again until your submission does get processed. Please make sure that you do not wait until the very last moment to submit your final system to avoid stress and missing the deadline.
During the training phase (now - July 2, 2018), teams can upload a submission by means of development. They can upload predictions for all instances in the development data in the same way as for the official evaluation phase. During the training phase, submissions will be evaluated against the gold-standard labels of the development data.
Find below a step-by-step guideline to upload your submission on CodaLab during both the development and evaluation phase:
More information about the submission format can be found here.
All teams can write a system description paper and submit it to the WASSA 2018 workshop. But first, all team members are to make sure they have a softconf account. New softconf users can register for an account by following the new user link here: https://www.softconf.com/emnlp2018/WASSA-2018/
In order for the task organizers to write a task description paper, they first require some additional information from each team. To this purpose, a Google Form has been created. As soon as the team has submitted predictions and every team member has created a START account, the team leader can access the Google Form and fill in all the necessary information.
Please do this as soon as possible, preferably immediately after you submitted your results and before July 10, 2018.
On the 16th of July, we will announce the final results and make the gold standard test labels available.
Please note that we will only include teams in the ranking that have actually filled in the Google Form.
You have time to write your system description papers until the 6th of August 2018. All papers should follow the EMNLP 2018 style files (LaTeX or Word). You can submit your paper by clicking on "Make a new Submission" on the WASSA softconf page, and then select the "WASSA shared task".
Please note that all author names should be included not only in the system paper (pdf), but also in the softconf (START) form when you are uploading paper information. When you click on the link to upload your paper on softconf, first you enter your start login details to login, then you will see a page where you are to enter meta information about the paper, like title, list of authors etc. At that point, you first click "add self" to add yourself, and then add all authors (one by one) via their start account id. Fill in other details. In the same page there is a button to upload the pdf.
Start: March 15, 2018, midnight
Start: July 2, 2018, midnight
Start: July 10, 2018, midnight
Nov. 2, 2018, midnight
You must be logged in to participate in competitions.Sign In