Offensive Language Identification in Dravidian Languages-EACL 2021

Organized by dravidianlangtech - Current server time: Jan. 25, 2021, 11:36 a.m. UTC

First phase

First phase
Nov. 20, 2020, midnight UTC

End

Competition Ends
Jan. 15, 2021, 8:07 a.m. UTC

 

Offensive language identification is classification task in natural language processing (NLP) where the aim is to moderate and minimise offensive content in social media. It has been an active area of research in both academia and industry for the past two decades. There is an increasing demand for offensive language identification on social media texts which are largely code-mixed. Code-mixing is a prevalent phenomenon in a multilingual community and the code-mixed texts are sometimes written in non-native scripts. Systems trained on monolingual data fail on code-mixed data due to the complexity of code-switching at different linguistic levels in the text. This shared task presents a new gold standard corpus for offensive language identification of code-mixed text in Dravidian languages (Tamil-English, Malayalam-English, and Kannada-English).

The goal of this task is to identify offensive language content of the code-mixed dataset of comments/posts in Dravidian Languages ( (Tamil-English, Malayalam-English, and Kannada-English)) collected from social media. The comment/post may contain more than one sentence but the average sentence length of the corpora is 1. Each comment/post is annotated at the comment/post level. This dataset also has class imbalance problems depicting real-world scenarios.

The participants will be provided development, training and test dataset.

Task:

This is a comment/post level classification task. Given a Youtube comment, systems have to classify it into Not-offensive, offensive-untargeted, offensive-targeted-individual, offensive-targeted-group, offensive-targeted-other, or Not-in-indented-language. To download the data and participate, go to the "Participate" tab.

As far as we know, this is the first shared task on offensive language identificaiton in Dravidian languages.

More details at https://dravidianlangtech.github.io/2021/

Paper  name format should be:   TEAM_NAME@DravidianLangTech-EACL2021: Title of the paper.

Example: NUIG_ULD@Dravidian-CodeMix-FIRE2020: Offensive Language Identificaiton on Multilingual Code Mixing Text.
 
 
Following are some general guidelines to keep in mind while submitting the working notes.
- Basic sanity check for grammatical errors and reported results
- Papers should have sufficient information for reproducing the mentioned results- Papers should follow the appropriate style (We will use EACL 2021 style: details below)
- Check the papers for text reuse / Plagiarism. This includes self-plagiarism as well. We would like to stress this point as EACL is quite strict about it. Any paper found to have plagiarized content should be rejected without further consideration.
 
- Please ensure the author names do not have any salutations like Dr., Prof., etc in the final version

 
All submissions should be in Double column EACL 2021 format. Authors should use one of the EACL 2021 Templates below:
In general, apart from plagiarism related concerns, we would not be rejecting any working notes paper.
 
References:
[1] Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N. and Kumar, R., 2019. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983.

Evaluation Criteria

We accept test result only through google form.
 
Format of the submission file should be like below:
 
 
id textlabel
1  Ennik mathram aano 200 cr clubil ee movie keerum enn thoniyath ? Not_offensive
2  Ouru Lúcifer erangi athe polle thanne enni kore padomm erakuum myrreeee Offensive_Targeted_Insult_Other

3

Orumathiri cheenja abhinayam... Manju warrier Offensive_Targeted_Insult_Individual

4

 Asooyamootha laalunnikal kuthiyirunnu dislike adikkunnu .poy chaavinadaa Offensive_Targeted_Insult_Group
5  This movie deserve double Oscar not_Tamil / not_Malayalam / not_Kannada
6 Itinum dislike adikkan aalundallo...nallatine angeegarikkan padikk... duranthangal... Offensive_Untargetede
  • label column should only have labels in the form mentioned in the training data i.e. 'Not-offensive, offensive-untargetede, Offensive_Targeted_Insult_Individual, Offensive_Targeted_Insult_Group, Offensive_Targeted_Insult_Other, or Not-in-indented-language.
id column is the index of the row. (keep the sequence of the YouTube comments same as provided in the test data)
 
Submission should be a single zip file with your team name containing csv files for individual languages -  'teamname_language.csv' e.g. the zip file may contain teamname_tamil.csv, teamname_malayalam.csv etc.
 
It is not mandatory to submit results for all language pairs, but csv files corresponding to the languages which you intend to submit results have to be included in the zip folder.
 
Please keep in mind that there is no limit on the number of submissions but only the latest one(zip folder) will be considered as final.
 
The submission will be evaluated with weighted average F1-score.
 
Submit results in google form given below.
 https://docs.google.com/forms/d/1sBt4GO-KzfIX_tWTYnu8-t8ayg-fBEWVC90GF6-Vif4/edit?usp=sharing
 
 RESULTS
 
 

Classification system’s performance was measured in terms of weighted averaged Precision, weighted averaged Recall and weighted averaged F-Score across all the classes. Weighted averaged are averaging the support-weighted mean per label. Participants are encouraged to check their system with Sklearn classification report https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html

Terms and Conditions

By downloading the data or by accessing it any manner, you agree not to redistribute the data except for non-commercial and academic-research purposes. The data must not be used for providing surveillance, analyses or research that isolates a group of individuals or any single individual for any unlawful or discriminatory purpose.

You should cite these papers if you are using our data.

@inproceedings{dravidianoffensive-eacl,
  title={Findings of the Shared Task on {O}ffensive {L}anguage {I}dentification in {T}amil, {M}alayalam, and {K}annada},
  author={Chakravarthi, Bharathi Raja  and
Priyadharshini, Ruba and
Jose, Navya  and
M, Anand Kumar and
Mandl, Thomas and
Kumaresan, Prasanna Kumar and
Ponnusamy, Rahul and
V,Hariharan and  
Sherly, Elizabeth and  
McCrae, John Philip },
    booktitle = "Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages",
    month = April,
    year = "2021",
    publisher = "Association for Computational Linguistics",
year={2021}
}

@inproceedings{chakravarthi-etal-2020-sentiment,

    title = "A Sentiment Analysis Dataset for Code-Mixed {M}alayalam-{E}nglish",

    author = "Chakravarthi, Bharathi Raja and

      Jose, Navya and

      Suryawanshi, Shardul and

      Sherly, Elizabeth and

      McCrae, John Philip",

    booktitle = "Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL)",

    month = may,

    year = "2020",

    address = "Marseille, France",

    publisher = "European Language Resources association",

    url = "https://www.aclweb.org/anthology/2020.sltu-1.25",

    pages = "177--184",

    abstract = "There is an increasing demand for sentiment analysis of text from social media which are mostly code-mixed. Systems trained on monolingual data fail for code-mixed data due to the complexity of mixing at different levels of the text. However, very few resources are available for code-mixed data to create models specific for this data. Although much research in multilingual and cross-lingual sentiment analysis has used semi-supervised or unsupervised methods, supervised methods still performs better. Only a few datasets for popular languages such as English-Spanish, English-Hindi, and English-Chinese are available. There are no resources available for Malayalam-English code-mixed data. This paper presents a new gold standard corpus for sentiment analysis of code-mixed text in Malayalam-English annotated by voluntary annotators. This gold standard corpus obtained a Krippendorff{'}s alpha above 0.8 for the dataset. We use this new corpus to provide the benchmark for sentiment analysis in Malayalam-English code-mixed texts.",

    language = "English",

    ISBN = "979-10-95546-35-1",

}}

 

@inproceedings{chakravarthi-etal-2020-corpus,

    title = "Corpus Creation for Sentiment Analysis in Code-Mixed {T}amil-{E}nglish Text",

    author = "Chakravarthi, Bharathi Raja and

      Muralidaran, Vigneshwaran and

      Priyadharshini, Ruba and

      McCrae, John Philip",

    booktitle = "Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL)",

    month = may,

    year = "2020",

    address = "Marseille, France",

    publisher = "European Language Resources association",

    url = "https://www.aclweb.org/anthology/2020.sltu-1.28",

    pages = "202--210",

    abstract = "Understanding the sentiment of a comment from a video or an image is an essential task in many applications. Sentiment analysis of a text can be useful for various decision-making processes. One such application is to analyse the popular sentiments of videos on social media based on viewer comments. However, comments from social media do not follow strict rules of grammar, and they contain mixing of more than one language, often written in non-native scripts. Non-availability of annotated code-mixed data for a low-resourced language like Tamil also adds difficulty to this problem. To overcome this, we created a gold standard Tamil-English code-switched, sentiment-annotated corpus containing 15,744 comment posts from YouTube. In this paper, we describe the process of creating the corpus and assigning polarities. We present inter-annotator agreement and show the results of sentiment analysis trained on this corpus as a benchmark.",

    language = "English",

    ISBN = "979-10-95546-35-1",

}

@inproceedings{hande-etal-2020-kancmd,
    title = "{K}an{CMD}: {K}annada {C}ode{M}ixed Dataset for Sentiment Analysis and Offensive Language Detection",
    author = "Hande, Adeep  and
      Priyadharshini, Ruba  and
      Chakravarthi, Bharathi Raja",
    booktitle = "Proceedings of the Third Workshop on Computational Modeling of People's Opinions, Personality, and Emotion's in Social Media",
    month = dec,
    year = "2020",
    address = "Barcelona, Spain (Online)",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.peoples-1.6",
    pages = "54--63",
    abstract = "We introduce Kannada CodeMixed Dataset (KanCMD), a multi-task learning dataset for sentiment analysis and offensive language identification. The KanCMD dataset highlights two real-world issues from the social media text. First, it contains actual comments in code mixed text posted by users on YouTube social media, rather than in monolingual text from the textbook. Second, it has been annotated for two tasks, namely sentiment analysis and offensive language detection for under-resourced Kannada language. Hence, KanCMD is meant to stimulate research in under-resourced Kannada language on real-world code-mixed social media text and multi-task learning. KanCMD was obtained by crawling the YouTube, and a minimum of three annotators annotates each comment. We release KanCMD 7,671 comments for multitask learning research purpose.",
}

Important Dates for shared task:

Task announcement: Nov 20, 2020
Release of Trail data: Nov 20, 2020
Release of Training data: Nov 20, 2020  
Release of Test data: Jan 2, 2021
Run submission deadline: Jan 15, 2021
Results declared: Jan 16, 2021
Paper submission: Jan 30, 2021
Peer review notification: Feb 18, 2021
Camera-ready paper due: Mar 1, 2021
Workshop Dates: April 19-20, 2021

Organizers: 

Student Volunteers

  • Navya Jose, Indian Institute Of Information Technology and Management-Kerala

  • Prasanna Kumar Kumaresan, Indian Institute Of Information Technology and Management-Kerala

  • Rahul Ponnsamy, Indian Institute Of Information Technology and Management-Kerala

  • Hariharan V, National Institute of Technology Karnataka Surathkal 

 

Email: dravidianlangtech@gmail.com and bharathiraja.akr@gmail.com

First phase

Start: Nov. 20, 2020, midnight

Competition Ends

Jan. 15, 2021, 8:07 a.m.

You must be logged in to participate in competitions.

Sign In