Competition

Dravidian-CodeMix - FIRE 2020

Organized by dravidiancodemixed - Current server time: March 30, 2025, 1:13 p.m. UTC

First phase

First phase
June 11, 2020, midnight UTC

End

Competition Ends
Dec. 31, 2020, 8:07 a.m. UTC

DravidianCodeMix FIRE 2020

Sentiment analysis is the task of identifying subjective opinions or responses about a given topic. It has been an active area of research in the past two decades in both academia and industry. There is an increasing demand for sentiment analysis on social media texts which are largely code-mixed. Code-mixing is a prevalent phenomenon in a multilingual community and the code-mixed texts are sometimes written in non-native scripts. Systems trained on monolingual data fail on code-mixed data due to the complexity of code-switching at different linguistic levels in the text. This shared task presents a new gold standard corpus for sentiment analysis of code-mixed text in Dravidian languages (Malayalam-English and Tamil-English).

Malayalam is one of the Dravidian languages spoken in the southern region of India with official recognition in the Indian state of Kerala and the Union Territories of Lakshadweep and Puducherry. There are nearly 38 million Malayalam speakers in India and other countries. Malayalam is a deeply agglutinative language. Tamil is a Dravidian language spoken by Tamil people in India, Sri Lanka and by the Tamil diaspora around the world, with official recognition in India, Sri Lanka and Singapore. The Malayalam script is the Vatteluttu alphabet extended with symbols from the Grantha alphabet. The Tamil script evolved from the Brahmi script, Vatteluttu alphabet, and Chola-Pallava script. It has 12 vowels, 18 consonants, and 1 āytam (voiceless velar fricative). Minority languages such as Saurashtra, Badaga, Irula, and Paniya are also written in the Tamil script. Both Tamil and Malayalam scripts are alpha-syllabic, belonging to a family of the abugida writing system that is partially alphabetic and partially syllable-based. However, social media users often adopt Roman script for typing because it is easy to input. Hence, the majority of the data available in social media for these under-resourced languages are code-mixed.

The goal of this task is to identify sentiment polarity of the code-mixed dataset of comments/posts in Dravidian Languages (Malayalam-English and Tamil-English) collected from social media. The comment/post may contain more than one sentence but the average sentence length of the corpora is 1. Each comment/post is annotated with sentiment polarity at the comment/post level. This dataset also has class imbalance problems depicting real-world scenarios. Our proposal aims to encourage research that will reveal how sentiment is expressed in code-mixed scenarios on social media.

The participants will be provided development, training and test dataset.

Task:

This is a message-level polarity classification task. Given a Youtube comment, systems have to classify it into positive, negative, neutral, mixed emotions, or not in the intended languages. To download the data and participate, go to the "Participate" tab.

As far as we know, this is the first shared task on Sentiment Analysis in Dravidian Code-Mixed text.

Keynote Speakers:

  1. Dr. Elizabeth Sherly, Professor (HAG) of Indian Institute of Information Technology and Management-Kerala (IIITM-K), will give a talk about Dravidian Languages
  2. Dr. Monojit Choudhury, Researcher in Microsoft Research Lab India, will give a talk about Code-Mixing

More details at https://dravidian-codemix.github.io/2020

 

Paper (Sentiment Analysis for Davidian Languages in Code-Mixed Text) name format should be:   TEAM_NAME@Dravidian-CodeMix-FIRE2020: Title of the paper.
Example: NUIG_ULD@Dravidian-CodeMix-FIRE2020: Sentiment Analysis on Multilingual Code Mixing Text.
 
 
Following are some general guidelines to keep in mind while submitting the working notes.
- Basic sanity check for grammatical errors and reported results
- Papers should have sufficient information for reproducing the mentioned results- Papers should follow the appropriate style (We will use CEUR style: details below)
- Check the papers for text reuse / Plagiarism. This includes self-plagiarism as well. We would like to stress this point as CEUR is quite strict about it. Any paper found to have plagiarized content should be rejected without further consideration.
- It has been commonly observed that several teams write more than one working notes (for e.g. separate submissions for separate subtasks) and reuse a substantial portion of the text in these multiple submissions. Keeping this in mind, we will NOT be allowing multiple working notes from the same set of authors. They should be asked to merge them into one.
- Please ensure the author names do not have any salutations like Dr., Prof., etc in the final version
- Each paper should have a copyright clause included in the paper (See the"Author agreement variants" at http://ceur-ws.org/HOWTOSUBMIT.html)
- Each author should also submit a copyright agreement signed by the authors. (Partially filled agreement will be shared shortly).
 
All submissions should be in Single column CEUR format. Authors should use one of the CEUR Templates below:
In general, apart from plagiarism related concerns, we would not be rejecting any paper.
 
Rank list:

Evaluation Criteria

We accept test result only through google form.
Format of the submission file should be like below:
 
 
id textlabel
ta_sent_1 Yarayellam FDFS ppga ippove ready agitinga Positive
ta_sent_2 Ennada viswasam mersal sarkar madhri time la likes and views create pannalayae Negative
-  -  -
  • label column should only have labels in the form mentioned in the training data i.e. 'not-Tamil', 'unknown_state', 'Positive', 'Mixed_feelings', 'Negative'.
  • id column is the index of the row. (keep the sequence of the youtuebe comments same as provided in the test data)
 
Submission should be a zip file with your team name containing a tsv file with name 'teamname_language.tsv'.. The submission will be evaluated with weighted average F1-score. Submit results in google form https://docs.google.com/forms/d/e/1FAIpQLSelTfEE3L_xHpNuGMGRl3WYZNdX1juSEfIRgozp7Kt_6WfCkw/viewform?usp=sf_link

 

Classification system’s performance will be measured in terms of weighted averaged Precision, weighted averaged Recall and weighted averaged F-Score across all the classes. Weighted averaged are averaging the support-weighted mean per label. Participants are encouraged to check their system with Sklearn classification report https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html

Terms and Conditions

By downloading the data or by accessing it any manner, you agree not to redistribute the data except for non-commercial and academic-research purposes. The data must not be used for providing surveillance, analyses or research that isolates a group of individuals or any single individual for any unlawful or discriminatory purpose.

You should cite these papers if you are using our data.

@inproceedings{chakravarthi-etal-2020-sentiment,

    title = "A Sentiment Analysis Dataset for Code-Mixed {M}alayalam-{E}nglish",

    author = "Chakravarthi, Bharathi Raja and

      Jose, Navya and

      Suryawanshi, Shardul and

      Sherly, Elizabeth and

      McCrae, John Philip",

    booktitle = "Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL)",

    month = may,

    year = "2020",

    address = "Marseille, France",

    publisher = "European Language Resources association",

    url = "https://www.aclweb.org/anthology/2020.sltu-1.25",

    pages = "177--184",

    abstract = "There is an increasing demand for sentiment analysis of text from social media which are mostly code-mixed. Systems trained on monolingual data fail for code-mixed data due to the complexity of mixing at different levels of the text. However, very few resources are available for code-mixed data to create models specific for this data. Although much research in multilingual and cross-lingual sentiment analysis has used semi-supervised or unsupervised methods, supervised methods still performs better. Only a few datasets for popular languages such as English-Spanish, English-Hindi, and English-Chinese are available. There are no resources available for Malayalam-English code-mixed data. This paper presents a new gold standard corpus for sentiment analysis of code-mixed text in Malayalam-English annotated by voluntary annotators. This gold standard corpus obtained a Krippendorff{'}s alpha above 0.8 for the dataset. We use this new corpus to provide the benchmark for sentiment analysis in Malayalam-English code-mixed texts.",

    language = "English",

    ISBN = "979-10-95546-35-1",

}}

 

@inproceedings{chakravarthi-etal-2020-corpus,

    title = "Corpus Creation for Sentiment Analysis in Code-Mixed {T}amil-{E}nglish Text",

    author = "Chakravarthi, Bharathi Raja and

      Muralidaran, Vigneshwaran and

      Priyadharshini, Ruba and

      McCrae, John Philip",

    booktitle = "Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL)",

    month = may,

    year = "2020",

    address = "Marseille, France",

    publisher = "European Language Resources association",

    url = "https://www.aclweb.org/anthology/2020.sltu-1.28",

    pages = "202--210",

    abstract = "Understanding the sentiment of a comment from a video or an image is an essential task in many applications. Sentiment analysis of a text can be useful for various decision-making processes. One such application is to analyse the popular sentiments of videos on social media based on viewer comments. However, comments from social media do not follow strict rules of grammar, and they contain mixing of more than one language, often written in non-native scripts. Non-availability of annotated code-mixed data for a low-resourced language like Tamil also adds difficulty to this problem. To overcome this, we created a gold standard Tamil-English code-switched, sentiment-annotated corpus containing 15,744 comment posts from YouTube. In this paper, we describe the process of creating the corpus and assigning polarities. We present inter-annotator agreement and show the results of sentiment analysis trained on this corpus as a benchmark.",

    language = "English",

    ISBN = "979-10-95546-35-1",

}

Task announcement: Starting now

Release of Trail data: 10 June

Release of Training data: 10 June

Release of Test data: 1 August

Run submission deadline: 20 August

Results declared: 31 August

Paper submission: 20 September

Revised paper: 30 October.

10th-13th December - FIRE 2020

Dr.Bharathi Raja Chakravarthi, Researcher, Insight SFI Research Centre for Data Analytics, Data Science Institute, National University of Ireland Galway

Vigneshwaran Muralidaran, Research Software Engineer, School of Computer Science and Informatics, Cardiff University, United Kingdom

Ruba Priyadharshini, Assistant Professor, Saraswathi Narayanan College, Madurai, India

Dr John P. McCrae, Lecturer-above-the-bar, Insight SFI Research Centre for Data Analytics, Data Science Institute, National University of Ireland Galway

Prof. Elizabeth Sherly, Senior Professor, Indian Institute of Information Technology and Management-Kerala, India

Student Volunteers

Navya Jose, M.Sc. Indian Institute of Information Technology and Management-Kerala, India

Shardul Suryawanshi, PhD Researcher, Insight SFI Research Centre for Data Analytics, Data Science Institute, National University of Ireland Galway, Ireland

Email: dravidiancodemixed@gmail.com and bharathiraja.akr@gmail.com

First phase

Start: June 11, 2020, midnight

Competition Ends

Dec. 31, 2020, 8:07 a.m.

You must be logged in to participate in competitions.

Sign In