SemEval19 Task 3 : EmoContext

Organized by emocontext_organizers - Current server time: Dec. 15, 2018, 4:35 a.m. UTC

Current

Training Data Release
Aug. 21, 2018, midnight UTC

Next

Final Evaluation on Test Set
Jan. 10, 2019, midnight UTC

End

Competition Ends
Never

EmoContext

A Shared Task on Contextual Emotion Detection in Text

We routinely experience emotions such as happiness, anger, sadness etc. As humans, on reading "Why don't you ever text me!", we can either interpret it as a sad or an angry emotion in absence of context; and the same ambiguity exists for machines as well. Lack of facial expressions and voice modulations make detecting emotions in text a challenging problem. However, as we increasingly communicate using text messaging applications and digital agents, contextual emotion detection in text is gaining importance to provide emotionally aware responses to users. Our shared task aims to bring more research to the problem of contextual emotion detection in text. 

We also realize that many of the participants might be first time Codalab users, and/or not advanced Machine Learning experts. Our task is aimed to help you get your feet wet in the domain, just follow our "How to Get Started?" section, and we will help you through. 

How to Get Started?


Step 1: Get Data Sets
To get the data set, please join this LinkedIn group of EmoContext and you will be able to get the data link inside.

 
Step 2: Get Starting kit
To make it easier to get your first emotion context model trained, we have provided a starting kit in Section Participate -> Files. 
Please download it and get started. 
Staring kit contains scripts, which you can run on Step 1 data sets and it generates the final test set in the format which is accepted by Step 3.
 
Step 3: Submit the test set for Leaderboard score evaluation
To submit the test set for evaluation and leaderboard calculation, Please go to Section Participate -> Submit/ View Results. 
 
References:
Organizers also have a relevant paper in this domain, which can be a useful read to get more insights. 
A Sentiment-and-Semantics-Based Approach for Emotion Detection in Textual Conversations

Terms and Conditions

By participating in this task you agree to these terms and conditions. If, however, one or more of this conditions is a concern for you, send us an email and we will consider if an exception can be made.

  • By submitting results to this competition, you consent to the public release of your scores at this website and at SemEval-2019 workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.
  • You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.
  • A participant can be involved in exactly one team (no more). If there are reasons why it makes sense for you to be on more than one team, then email us before the evaluation period begins. In special circumstances this may be allowed.

  • Each team must create and use exactly one CodaLab account.
  • Team constitution (members of a team) cannot be changed after the evaluation period has begun.

  • During the evaluation period:

    • Each team can submit as many as fifty submissions . However, only the final submission will be considered as the official submission to the competition.

    • You will not be able to see results of your submission on the test set.

    • You will be able to see any warnings and errors for each of your submission.

    • Leaderboard is disabled.

  • Once the competition is over, we will release the gold labels and you will be able to determine results on various system variants you may have developed. We encourage you to report results on all of your systems (or system variants) in the system-description paper. However, we will ask you to clearly indicate the result of your official submission.

  • We will make the final submissions of the teams public at some point after the evaluation period.

  • The organizers and their affiliated institutions makes no warranties regarding the datasets provided, including but not limited to being correct or complete. 

  • Each task participant will be assigned another teams’ system description papers for review, using the EasyChair system. The papers will thus be peer reviewed.
  • The dataset release is governed by terms and conditions mentioned with the dataset release. 

  • If you use any of the datasets provided here, you must attribute to the organizers. We will post exact details once the task paper is available. 

Evaluation Details

Evaluation will be done by calculating microaveraged F1 score (F1µ) for the three emotion classes i.e. Happy, Sad and Angry on the submissions made with predicted class of each sample in the Test set. To be precise, we define the scoring as following:

Pµ = ΣTPi / Σ(TPi + FPi)∀i {Happy,Sad,Angry} 
Rµ = ΣTPi / Σ(TPi + FNi)∀i {Happy,Sad,Angry}

where TPi is the number of samples of class i which are correctly predicted, FNi and FPi are the counts of Type-I and Type-II errors respectively for the samples of class i.

Our final metric F1µ will be calculated as the harmonic mean of Pµ and Rµ. 

Talk to Organizers

The organizers of this task are a group of Applied Scientists at Microsoft AI, Hyderabad, India. Find more about them below:

Niranjan Nayak

Puneet Agrawal

Meghana Joshi

Kedhar Nath Narahari

Ankush Chatterjee

Contact the Organizers

We encourage you to post your questions on the LinkedIn group of EmoContext and one of the organizers will respond. This allows for an active discussion with participants and we hope to encourage the same. 

You may alternatively also reach the organizers at emocontext-semeval19@microsoft.com 

 

Data Set Format

The Training dataset is a .txt file containing 5 columns :

  1. ID - Contains a unique number to identify each training sample
  2. Turn 1 - Contains the first turn in the three turn conversation, written by User 1
  3. Turn 2 - Contains the second turn, which is a reply to the first turn in conversation and is written by User 2
  4. Turn 3 - Contains the third turn, which is a reply to the second turn in the conversation, which is written by User 1
  5. Label - Contains the human judged label of Emotion of Turn 3 based on the conversation for the given training sample. It is always one of the four values - 'happy', 'sad, 'angry' and 'others'

The Dev set and the Test set contains the first 4 columns as mentioned above. The 5th column - 'Label' is absent from those 2 sets.

Please note an important difference which you may note in the Training set vs Dev or Test set. Training data consists of about 5k samples each from 'angry', 'sad', 'happy' class, and 15k samples from 'others' class, whereas, both Dev and Test sets have a real life distribution, which is about 4% each of 'angry', 'sad', 'happy' class and the rest is 'others' class. This will significantly change your F1 scores, so please keep this important note in mind.

Two example training samples are given below:

id turn1 turn2 turn3 label
156 You are funny  LOL I know that. :p  😊  happy
187 Yeah exactly  Like you said, like brother like sister ;)  Not in the least  

others

 

 

 

 

 

 

 

Submission Format

The data to be submitted should be zip of a text file with name test.txt and containing 5 columns (tab separated) :

  1. ID - Contains the unique number assigned to the cconversation in the test data
  2. Turn 1 - Contains the first turn in the three turn conversation, written by User 1
  3. Turn 2 - Contains the second turn, which is a reply to the first turn in conversation and is written by User 2
  4. Turn 3 - Contains the third turn, which is a reply to the second turn in the conversation, which is written by User 1
  5. Label - Contains the predicted label of Emotion of Turn 3 based on the conversation for the given sample. It has to be one of the four values - 'happy', 'sad, 'angry' and 'others'

The first row in the submission set must contain the line - 'id turn1 turn2 turn3 label', each word separated by a <tab>.

Please note an important difference which you may note in the Training set vs Dev or Test set. Training data consists of about 5k samples each from 'angry', 'sad', 'happy' class, and 15k samples from 'others' class, whereas, both Dev and Test sets have a real life distribution, which is about 4% each of 'angry', 'sad', 'happy' class and the rest is 'others' class. This will significantly change your F1 scores, so please keep this important note in mind. 

Important Dates

Date Phase
21 Aug 2018 Task Release - Baseline system, train set (with labels) and dev set (without labels) will be provided
10 Dec 2018 Pre evaluation - Dev set (with labels) release
17 Jan 2019 Evaluation start - Test Set (without labels) release
25 Jan 2019 Evaluation end
05 Feb 2019 Results posted
28 Feb 2019 System description paper submissions due by 23:59 GMT -12:00
14 Mar 2019 Paper reviews due
06 Apr 2019 Author notifications
20 Apr 2019 Camera ready submissions due

Training Data Release

Start: Aug. 21, 2018, midnight

Description: In this phase, participants will be provided with (a) Training data set to train their model (b) Dev data set (without the labels) and a mechanism to test their model on this data set. (c) A starter kit consisting of a baseline system, as well as scripts to read data, to run evaluation etc.

Final Evaluation on Test Set

Start: Jan. 10, 2019, midnight

Description: In this phase, participants will be given access to the test set without labels to let them evaluate their final system.

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In
# Username Score
1 jogonba2 0.7723
2 mfzszgs 0.7679
3 rafalposwiata 0.7660