SemEval19 Task 3 : EmoContext

Organized by emocontext_organizers - Current server time: April 26, 2019, 3:57 a.m. UTC

First phase

Training Data Release
Aug. 21, 2018, midnight UTC

End

Competition Ends
Jan. 26, 2019, 11:59 p.m. UTC

EmoContext

A Shared Task on Contextual Emotion Detection in Text

We routinely experience emotions such as happiness, anger, sadness etc. As humans, on reading "Why don't you ever text me!", we can either interpret it as a sad or an angry emotion in absence of context; and the same ambiguity exists for machines as well. Lack of facial expressions and voice modulations make detecting emotions in text a challenging problem. However, as we increasingly communicate using text messaging applications and digital agents, contextual emotion detection in text is gaining importance to provide emotionally aware responses to users. Our shared task aims to bring more research to the problem of contextual emotion detection in text. 

We also realize that many of the participants might be first time Codalab users, and/or not advanced Machine Learning experts. Our task is aimed to help you get your feet wet in the domain, just follow our "How to Get Started?" section, and we will help you through. 

How to Get Started?


Step 1: Get Data Sets
To get the data set, please join this LinkedIn group of EmoContext and you will be able to get the data link inside.

 
Step 2: Get Starting kit
To make it easier to get your first emotion context model trained, we have provided a starting kit in Section Participate -> Files. 
Please download it and get started. 
Staring kit contains scripts, which you can run on Step 1 data sets and it generates the final test set in the format which is accepted by Step 3.
 
Step 3: Submit the test set for Leaderboard score evaluation
To submit the test set for evaluation and leaderboard calculation, Please go to Section Participate -> Submit/ View Results. 
 
References:
Organizers also have a relevant paper in this domain, which can be a useful read to get more insights. 
A Sentiment-and-Semantics-Based Approach for Emotion Detection in Textual Conversations

Terms and Conditions

By participating in this task you agree to these terms and conditions. If, however, one or more of this conditions is a concern for you, send us an email and we will consider if an exception can be made.

  • By submitting results to this competition, you consent to the public release of your scores at this website and at SemEval-2019 workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.
  • You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.
  • A participant can be involved in exactly one team (no more). If there are reasons why it makes sense for you to be on more than one team, then email us before the evaluation period begins. In special circumstances this may be allowed.

  • Each team must create and use exactly one CodaLab account.
  • Team constitution (members of a team) cannot be changed after the evaluation period has begun.

  • During the evaluation period:

    • Each team can submit as many as thirty submissions . The best submission will be considered as the official submission to the competition.

    • You will be able to see results of your submission on the test set.

    • You will be able to see any warnings and errors for each of your submission.

    • Leaderboard with one F1 score is visible to all users.

  • Once the competition is over, we will release the gold labels or reopen Codalab and you will be able to determine results on various system variants you may have developed.

  • We encourage you to report results on all of your systems (or system variants) in the system-description paper.  

  • We will make the final submissions of the teams public at some point after the evaluation period.

  • The organizers and their affiliated institutions makes no warranties regarding the datasets provided, including but not limited to being correct or complete. 

  • Each task participant will be assigned another teams’ system description papers for review, using the EasyChair system. The papers will thus be peer reviewed.
  • The dataset release is governed by terms and conditions mentioned with the dataset release. 

  • If you use any of the datasets provided here, you must attribute to the organizers. We will post exact details once the task paper is available. 

Evaluation Details

Evaluation will be done by calculating microaveraged F1 score (F1µ) for the three emotion classes i.e. Happy, Sad and Angry on the submissions made with predicted class of each sample in the Test set. To be precise, we define the scoring as following:

Pµ = ΣTPi / Σ(TPi + FPi)∀i {Happy,Sad,Angry} 
Rµ = ΣTPi / Σ(TPi + FNi)∀i {Happy,Sad,Angry}

where TPi is the number of samples of class i which are correctly predicted, FNi and FPi are the counts of Type-I and Type-II errors respectively for the samples of class i.

Our final metric F1µ will be calculated as the harmonic mean of Pµ and Rµ. 

Talk to Organizers

The organizers of this task are a group of Applied Scientists at Microsoft AI, Hyderabad, India. Find more about them below:

Niranjan Nayak

Puneet Agrawal

Meghana Joshi

Kedhar Nath Narahari

Ankush Chatterjee

Contact the Organizers

We encourage you to post your questions on the LinkedIn group of EmoContext and one of the organizers will respond. This allows for an active discussion with participants and we hope to encourage the same. 

You may alternatively also reach the organizers at emocontext-semeval19@microsoft.com 

 

Data Set Format

The Training dataset is a .txt file containing 5 columns :

  1. ID - Contains a unique number to identify each training sample
  2. Turn 1 - Contains the first turn in the three turn conversation, written by User 1
  3. Turn 2 - Contains the second turn, which is a reply to the first turn in conversation and is written by User 2
  4. Turn 3 - Contains the third turn, which is a reply to the second turn in the conversation, which is written by User 1
  5. Label - Contains the human judged label of Emotion of Turn 3 based on the conversation for the given training sample. It is always one of the four values - 'happy', 'sad, 'angry' and 'others'

The Dev set and the Test set contains the first 4 columns as mentioned above. The 5th column - 'Label' is absent from those 2 sets.

Please note an important difference which you may note in the Training set vs Dev or Test set. Training data consists of about 5k samples each from 'angry', 'sad', 'happy' class, and 15k samples from 'others' class, whereas, both Dev and Test sets have a real life distribution, which is about 4% each of 'angry', 'sad', 'happy' class and the rest is 'others' class. This will significantly change your F1 scores, so please keep this important note in mind.

Two example training samples are given below:

id turn1 turn2 turn3 label
156 You are funny  LOL I know that. :p  😊  happy
187 Yeah exactly  Like you said, like brother like sister ;)  Not in the least  

others

 

 

 

 

 

 

 

Submission Format

The data to be submitted should be zip of a text file with name test.txt and containing 5 columns (tab separated) :

  1. ID - Contains the unique number assigned to the cconversation in the test data
  2. Turn 1 - Contains the first turn in the three turn conversation, written by User 1
  3. Turn 2 - Contains the second turn, which is a reply to the first turn in conversation and is written by User 2
  4. Turn 3 - Contains the third turn, which is a reply to the second turn in the conversation, which is written by User 1
  5. Label - Contains the predicted label of Emotion of Turn 3 based on the conversation for the given sample. It has to be one of the four values - 'happy', 'sad, 'angry' and 'others'

The first row in the submission set must contain the line - 'id turn1 turn2 turn3 label', each word separated by a <tab>.

Please note an important difference which you may note in the Training set vs Dev or Test set. Training data consists of about 5k samples each from 'angry', 'sad', 'happy' class, and 15k samples from 'others' class, whereas, both Dev and Test sets have a real life distribution, which is about 4% each of 'angry', 'sad', 'happy' class and the rest is 'others' class. This will significantly change your F1 scores, so please keep this important note in mind. 

 

Check list to check before submitting a file -

1. The file inside the zip should be named as "test.txt"

2. The file should be in utf-8 format

3. The file should have headers  'id turn1 turn2 turn3 label', each word separated by a <tab>

4. The file should contain all four labels - happy, sad, angry and others. Otherwise it will result into divide by zero error while doing F1 calculation. 

 

Important Dates

Date Phase
21 Aug 2018 Task Release - Baseline system, train set (with labels) and dev set (without labels) will be provided
10 Dec 2018 Pre evaluation - Dev set (with labels) release
18 Jan 2019 Evaluation start - Test Set (without labels) release
26 Jan 2019 Evaluation end
30 Jan 2019 Results and guidelines for system description paper posted 
23 Feb 2019 System description paper submissions due by 23:59 GMT -12:00
16 Mar 2019 Paper reviews due
29 Mar 2019 Author notifications
5 Apr 2019 Camera ready submissions due

System-Description Papers

Participants who made a submission on the CodaLab website during the official evaluation period and had a better F1 score than the official baseline (under the name of emocontext_organizers) are given the opportunity to submit a system-description paper that describes their system, resources used, results, and analysis. This paper, if accepted, will be part of the official SemEval-2019 proceedings.

See details on number of pages, format of the paper provided by SemEval here.
Link to submit your system description paper is here. All the authors of the paper should register on the SoftConf site.

Papers are due 23 February, 2019, by 23:59 GMT -12:00.

The title of your paper should be : 

<GroupName> at SemEval-2019 Task 3: <Subtitle>

 

You can choose your own Group Name and Subtitle but the rest is fixed.

You do not have to repeat details of the task and data. Just cite the Task paper (details below), quickly summarize the task you made submission to and then you can get into the details of the related work, your systems, experiments, and results.

Cite the task paper as shown below:

Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. 2019. Semeval-2019 Task 3: EmoContext: Contextual Emotion Detection in Text. In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval-2019), Minneapolis, Minnesota, 2019.

@InProceedings{SemEval2019Task3, author = {Chatterjee, Ankush and Narahari, Kedhar Nath and Joshi, Meghana and Agrawal, Puneet}, title = {SemEval-2019 Task 3: EmoContext: Contextual Emotion Detection in Text}, booktitle = {Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval-2019)}, address = { Minneapolis, Minnesota}, year = {2019} }

A draft of this task description paper will be made available in early March. This is after the deadline for your paper submission but you will be able to see this paper well-before the camera-ready deadline. So after access to the task paper, you can still update your paper as you see fit.

Additionally, if you used or compared against the SS-BED model provided as a reference in Codalab's "How to get Started?" section, we encourage you to cite the below paper.

 @article{chatterjee2019understanding, title={Understanding Emotions in Text Using Deep Learning and Big Data}, author={Chatterjee, Ankush and Gupta, Umang and Chinnakotla, Manoj Kumar and Srikanth, Radhakrishnan and Galley, Michel and Agrawal, Puneet}, journal={Computers in Human Behavior}, volume={93}, pages={309--317}, year={2019}, publisher={Elsevier} }

 

Important Notes:

  • You are not obligated to submit a system-description paper, however, we strongly encourage all eligible teams to do so.
  • SemEval seeks to have all eligible participants publish a paper, unless the paper does a poor job of describing their system and is not well written. Your system rank and scores will not impact whether the paper is accepted or not.
  • Note that SemEval submission is not anonymous; author names should be included.
  • Later each task participant will be assigned another teams’ system description papers for review, using the SoftConf system. It is mandatory to review the papers assigned to you.
  • All task participant teams should prepare a poster for display at SemEval. One selected team will be asked to prepare a short talk. Details will be provided at a later date.
  • Please do not dwell too much on rankings. Focus instead on analysis and the research questions that your system can help address.
  • It may also be helpful to look at some of the papers from past SemEval competitions, e.g., from https://aclweb.org/anthology/S/S16/.

 

What to include in a system-description paper?

Here are some key pointers:

  • Replicability: Present all details that will allow someone else to replicate your system.
  • Analysis: Focus more on results and analysis and less on discussing rankings. Report results on several variants of your system; present sensitivity analysis of your system's parameters, network architecture, etc.; present ablation experiments showing usefulness of different features and techniques; show comparisons with baselines.
  • Related work: Place your work in context of previously published related work. Cite all data and resources used in your submission.

Training Data Release

Start: Aug. 21, 2018, midnight

Description: In this phase, participants will be provided with (a) Training data set to train their model (b) Dev data set (without the labels) and a mechanism to test their model on this data set. (c) A starter kit consisting of a baseline system, as well as scripts to read data, to run evaluation etc.

Final Evaluation on Test Set

Start: Jan. 18, 2019, midnight

Description: In this phase, participants will be given access to the test set without labels to let them evaluate their final system. All times mentioned are in UTC.

Competition Ends

Jan. 26, 2019, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In