Competition

Evaluation

Systems are evaluated using standard evaluation metrics, including accuracy, precision, recall and F₁-score. The submissions will be ranked by F₁-score.

Note that while accuracy provides insights into the system performance for all classes, the latter three measures will be calculated for the positive class only (subtask A) or will be reported per class label or macro-averaged (subtask B). Macro-averaging of the F₁-score implies that all class labels have equal weight in the final score.

The metrics will be calculated as follows:

Scoring script

The official evaluation script is available at GitHub.

Terms and conditions

By submitting results to this competition, you consent to the public release of your scores at the SemEval-2018 workshop and in the associated proceedings, at the task organisers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organisers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organisers.

You further agree that the task organisers are under no obligation to release scores and that scores may be withheld if it is the task organisers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.

You further agree that your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organisers.

You agree not to redistribute the test data except in the manner prescribed by its licence.

Schedule

Competition setup

• Mon 21 Aug 2017: CodaLab competition website ready and made public. Should include basic task description and mailing group information for the task. Trial data ready. Evaluation script ready for participants to download and run on the trial data.
• Mon 25 Sep 2017: Training data ready. Development data ready. CodaLab competition website updated to include an evaluation script uploaded as part of the competition so that participants can upload submissions on the development set and the script immediately checks the submission for format and computes the results on the development set. Benchmark system should be made available to participants. Also, the organisers should run the submission created with the benchmark system on CodaLab, so that participants can see its results on the Leaderboard.

Competition and evaluation

• Mon 15 Jan 2018: Evaluation start Task A
• Mon 22 Jan 2018: Evaluation end Task A, Evaluation start Task B
• Mon 29 Jan 2018: Evaluation end Task B
• Mon 05 Feb 2018: Results Task A & Task B posted
• Fri 09 Feb 2018: Gold labels Task A & Task B released
• Mon 26 Feb 2018: System description paper submissions due by 23:59 GMT -12:00
• Mon 05 Mar 2018: Task description paper submissions due by 23:59 GMT -12:00
• Mon 19 Mar 2018: Paper reviews due (for both systems and tasks)
• Mon 02 Apr 2018: Author notifications
• Mon 16 Apr 2018: Camera ready submissions due

Task organisers

Cynthia Van Hee is a post-doctoral researcher at the LT3 Language and Translation Technology Team at Ghent University, active in the field of computational linguistics and machine learning. In the framework of her PhD, she created a theoretic framework of irony and developed a state-of-the-art irony detection system. Her other research interests include sentiment and emotion analysis in social media text and the detection of cyberbullying, which was one of the use cases of the AMiCA (Automatic Monitoring for Cyberspace Applications) project in which she was actively involved. Related to the first topic, she participated in the SemEval-2014 and SemEval-2015 tasks "Sentiment Analysis on Twitter" and "Sentiment Analysis of Figurative Language in Twitter", respectively.

Els Lefever is assistant professor Terminology and Computational Semantics. She holds a PhD in computer science from Ghent University on "ParaSense: Parallel Corpora for Word Sense Disambiguation" (2012). She has a strong expertise in machine learning of natural language and multilingual natural language processing, with a special interest for computational semantics, cross-lingual word sense disambiguation and automatic terminology extraction & taxonomy learning. Els organised two runs of the SemEval "Crosslingual Word Sense Disambiguation" task and co-organised the SemEval-2014 task on "L2 Writing Assistant" and the SemEval-2016 task on "Taxonomy Extraction Evaluation (TexEval)". She also participated to 5 other SemEval tasks (2007: Web People Search, 2014: Sentiment Analysis in Twitter, 2015: Sentiment Analysis of Figurative Language in Twitter, Aspect Based Sentiment Analysis, and Taxonomy Extraction Evaluation).

Véronique Hoste is full professor and head of LT3. She holds a PhD in linguistics on optimisation issues in machine learning of coreference resolution (2005). She has a strong expertise in machine learning of natural language, and more specifically in coreference resolution, word sense disambiguation, terminology extraction, text classification, classifier optimisation, readability prediction, sentiment mining, etc. Véronique already participated in the Senseval 2 and Senseval-3 competitions early 2000 on all-words word sense disambiguation. In the SemEval successor, she co-organised tasks on coreference resolution (2010), cross-lingual word sense disambiguation (2010 and 2013), L2 writing (2014) and multilingual aspect-based sentiment analysis (2016) on which she also published several articles.

The task organisers are members of the LT³ (Language and Translation Technology) team at the Faculty of Arts and Philosophy at Ghent University.
Contact: firstname.lastname@ugent.be

Data

Training and test datasets are provided for two subtasks A and B on irony detection in tweets. The training dataset for task A has tweets with a binary value score (0 or 1) indicating whether the tweet is ironic. The training data for subtask B includes tweets with a numeric value corresponding to one of the subcategories (i.e. ironic by clash, other irony, situational irony, non-ironic). The test data includes only the tweet text. Gold irony labels will be released after the evaluation period. Download links and further details on the construction of the dataset are provided below.

Corpus

We constructed a data set of 3,000 English tweets, by searching Twitter for the hashtags #irony, #sarcasm and #not. All tweets were collected between 01/12/2014 and 04/01/2015 and represent 2,676 unique users. To minimize the noise introduced by groundless irony-related hashtags, all tweets were manually labeled using a fine-grained annotation scheme for irony (Van Hee et al., 2016c). Prior to data annotation, the entire corpus was cleaned by removing retweets, duplicates and non-English tweets, and replacement of XML-escaped characters (e.g. &).

Annotation

The entire corpus was annotated by three students in linguistics and second-language speakers of English, which each annotated one third of the corpus. Brat (Stenetorp et al., 2012). was used as the annotation tool. To assess the reliability of the annotations, an annotation agreement study was carried out on a subset of the corpus (100 instances), resulting in Fleiss Kappa scores of 0.72 for recognising (the different forms of) ironic instances. Annotation of the corpus resulted in the following distribution of the different classes:

• Verbal irony by means of a polarity contrast: 1,728 instances
• Other types of verbal irony: 267 instances
• Situational irony: 401 instances
• Non-ironic: 604 instances

Train and test corpus

Based on the annotations, 2,396 instances are ironic (1,728 + 267 + 401) while 604 are not. To balance class representation in the corpus, 1,792 non-ironic tweets were added from a background corpus. The tweets were manually checked to ascertain that they are non-ironic and are devoid of irony-related hashtags. This brings the total amount of data to 4,792 tweets (2,396 ironic + 2,396 non-ironic). For the SemEval-2018 competition, this corpus will be randomly split into a training (80% or 3,833 instances) and test (20%, or 958 instances) set. In addition, we will provide a second manually annotated test set containing approximately 1,500 tweets.

Submission format

System submissions for CodaLab are zip-compressed folders containing a predictions file called predictions-taskA.txt or predictions-taskB.txt, for subtask A and B, respectively. Remember that your submission should only concern one task at a time. There will be separate evaluation periods for the two subtasks.

The evaluation script will check whether the files contain the relevant labels for each task. The script, as well as a sample predictions file can be found on GitHub.

IMPORTANT: a lot of competitions are run on CodaLab, and just a certain number of submissions can be handled at a given time, due to which your submission may be 'stuck' (i.e. status remains 'submitted' even after refreshing) for a certain time. In this case, patiently try again until your submission does get processed. Please make sure that you do not wait until the very last moment to submit your final system to avoid stress and missing the deadline :-). 

Practice phase

During the practice phase, teams can upload an example submission for Subtask A (i.e. binary irony classification). This submission is optional and just by means of practice for participants that are new to CodaLab (i.e. to find out how submitting and evaluating system output on CodaLab works) and does not require any development. The sample file for this phase can be downloaded from GitHub and contains predictions for 10 instances of the training data for Task A. Please note that the file (like the prediction files for final evaluation) should be named 'predictions-taskA.txt' and should be uploaded in a zipped folder named 'submission'.

Want to try?

• navigate to 'Participate';
• click 'Submit / View Results';
• upload your zipped submission folder (*);
• refresh the status to see the evaluation status;
• click 'download evaluation output from scoring step' to see the results.

(*) Make sure that your submission is a zip-file named submission.zip (for Mac users: ensure that it contains no __macosx file).

Development and evaluation phase

During the development phase (25/09/2017 - 15/01/2018), teams can upload a submission for Subtask A (i.e. binary irony classification) by means of development. They can upload predictions (e.g. obtained via the baseline system) for all instances in the training data for Task A in the same way as for the official evaluation phase. During the development phase, submissions will be evaluated against the gold-standard labels of the training data for subtask A.

Find below a step-by-step guideline to upload your submission on CodaLab during both the development and evaluation phase:

• via the command line, navigate on your local pc to a folder named 'submission' containing the output of your system (i.e. a file named 'predictions-taskA.txt' or 'predictions-taskB.txt', depending on the task). Make sure that there is 1 prediction per line and that the original order of the posts is left unchanged.
• inside the folder, execute the following command: zip -r ../submission.zip *
• on CodaLab, navigate to 'Participate' > 'Evaluation Task A' or 'Evaluation Task B' > Submit/View Results and upload your submission.zip file
• click 'Refresh status' until your submission receives the 'Finished' status
• click 'Submit to leaderboard' to push your results to the official scoring board. Please note: as soon as the official evaluation period starts, the scoring board will not be made visible until the end of the evaluation period (end of January 2018).

Baseline system

You are free to build a system from scratch using any available software packages and resources, as long as they are not against the spirit of fair competition. In order to assist testing of ideas, we also provide a baseline irony detection system that you can build on.

The use of this system is completely optional and is available on GitHub, as well as instructions for using the system with the task data. The system is based on token unigram features and outputs predictions for subtask A or subtask B and a score (obtained through 10-fold cross validation on the training sets).

Directions, notes and reminders for the official evaluation phase

Time schedule

• Evaluation Task A: 15/01 - 22/01 (i.e. submissions can be made until Sunday 21/01, 23:59)
• Evaluation Task B: 22/01 - 29/01 (i.e. submissions can be made until Sunday 28/01, 23:59)

Important

• A participant can be involved in exactly one team. If there are reasons why it makes sense for you to participate in more than one team, then email us before the evaluation period starts. In special circumstances this may be allowed.
• All registered members of this task are asked to indicate their username, email and team name in a Google spreadsheet (invitation via email). Team constitutions cannot be changed once the evaluation period has started.

System submission

• Participants can download the test data via GitHub (when released) and run their system on the data to generate a submission file.
• Participants can submit for each task a constrained (only the provided training data were used to develop the system) and an unconstrained (additional training data were used) run for each task (but: only the last submission will appear on the official leader board).
• Submission files should be named 'predictions-taskA.txt' or 'predictions-taskB.txt' and zipped in a folder named 'submission_constrained' or 'submission_unconstrained', depending on the flavour of your system. Submission folders that are named 'submission.zip' will be considered constrained.
• The predictions file consists of n predictions (one per line/for each instance in the test set) in the relevant order. Only labels should be included in the predictions file, without any text, metadata or headers.
• When uploading their system, participants will need to fill in some metadata, including the name of their team, a brief description of the method used, etc.
• Only the last submission (it does not matter whether this is constrained/unconstrained) will be shown on the leaderboard (this will be the official ranking on CodaLab), but the task description paper will list a separate ranking of the best constrained and unconstrained system runs. It will therefore be important to clearly indicate, at the moment of submission, whether the run is constrained or unconstrained.
• After the evaluation phase has finished, each team will be asked to fill in a form to provide details about thehir submission so that all systems can be accurately described in the task description paper. 

Practical details about the leader board

• The leader board will be disabled until the end of the evaluation period, so you will not be able to see the results of your submission. You can, however, see whether the upload of your submission was succesful or not (if not, consult the error log) and you can of course evaluate your system on the training data by making use of the evaluation script (more details below).

• The scores of the last submission of each team will be displayed on the leader board after the evaluation period ends. Whether that is the constrained or unconstrained system (in case you have both), is up to the choice of the team.

• For each team, there is one team member that submits the system. Submissions for one team that are made by different people will not be considered and only the final submission will be considered as the team's official submission. Important: there is as a maximum of 10 submission attempts (i.e. in case something goes wrong wile uploading, there are problems with the file format, etc.), uploading more than 10 submissions is not possibile.

Use of the evaluation script

During development, participants can evaluate their system on the training data (on their local device) by making use of the official scoring script. The same scoring script runs on CodaLab to evaluate the submissions.

To simulate the scoring of your submission on your local PC, execute the following steps:

1. Create a folder (e.g. named 'demo') with two subfolders: 'ref' and 'res'
2. Put the gold-standard file in the 'ref' folder and your system output in the 'res' folder
3. Run the script with the following command: python evaluate.py demo .
4. Consult the output in the 'scores.txt' file that was generated from step 3