Shared Task on Sarcasm and Sentiment Detection in Arabic (Subtask 1 - Sarcasm Detection)

Organized by iaf7 - Current server time: Jan. 25, 2021, 11:46 a.m. UTC

Previous

Training phase
Jan. 1, 2021, midnight UTC

Current

Test phase
Jan. 21, 2021, midnight UTC

End

Competition Ends
Feb. 1, 2021, 11:59 p.m. UTC

Shared Task on Sarcasm and Sentiment Detection in Arabic (Subtask 1- Sarcasm Detection)

Sarcasm detection is the process of identifying whether a piece of text is sarcastic or not. Sarcasm is one of the main challenges for sentiment analysis systems. The reason for this is that a sarcastic sentence usually carries a negative implicit sentiment, while it is expressed using positive expressions. This contradiction between the surface sentiment and the intended one creates a complex challenge for sentiment analysis systems.
Sarcasm detection received attention in other languages, but Arabic still lags behind. There have been few efforts on Arabic sarcasm detection such as the works of (Karoui et al., 2017; Ghanem et al., 2020) and the shared task held by (Ghanem et al., 2019). There have been some recent efforts to build standard datasets for this task such as (Abbes et al., 2020; Abu Farha and Magdy, 2020). The shared task on Sarcasm and Sentiment Detection in Arabic will be held with WANLP@EACL2021. The shared task will focus on analysing tweets and identifying their sentiment and whether a tweet is sarcastic or not.

There are two subtasks in this shared task:

  • Subtask 1 (Sarcasm Detection): Identifying whether a tweet is sarcastic or not, this is a binary classification task.
  • Subtask 2 (Sentiment Analysis): Identifying the sentiment of a tweet and assigning one of three labels (Positive, Negative, Neutral), multiclass classification task.

Data: For initial experimentation, participants can use the ArSarcasm dataset.
The training will be available here for participants.

Evaluation Criteria:

The following will be used for the evaluation:

  • Subtask 1: The evaluation metrics will include precision/recall/f-score/accuracy. F-score of the sarcastic class will be the official metric.
  • Subtask 2: The evaluation metrics will include precision/recall/f-score/accuracy. F-PN (Macro average of the F-score of the positive and negative classes) will be the official metric.

Submission Format:

Classifications of test dataset (labels only) should be submitted as separate files in the following format with a label for each corresponding tweet (i.e. the label in line x in the submission file corresponds to the tweet in line x in the test file):
For Subtask 1, it should be whether a tweet is sarcastic or not as follows:
        TRUE (or FALSE)\n

For Subtask 2, it should be the sentiment class positive (POS), negative (NEG), or neutral (NEU):
        POS (or NEG or NEU)\n

Sumbission filename should be in the following format:

ParticipantTeamName_Subtask_<1/2>.zip (a plain .txt file inside each .zip file)

Ex: SMASH_Subtask_1.zip (the results for Subtask 1 for test dataset from SMASH team)
Ex: HBKU_Subtask_2.zip (the results for Subtask 2 for test dataset from HBKU team)

Contact:
For any questions related to the shared task, please contact the organizers using this email address arsarcasm.shared.task@gmail.com or contact Ibrahim Abu Farha (i.abufarha@ed.ac.uk)

Terms and Conditions

The data in this competition is licensed under a Creative Commons Attribution license (CC-BY).

Training phase

Start: Jan. 1, 2021, midnight

Test phase

Start: Jan. 21, 2021, midnight

Competition Ends

Feb. 1, 2021, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In