Hope Speech Detection for Equality, Diversity, and Inclusion-EACL 2021

Organized by dravidianlangtech - Current server time: Jan. 25, 2021, 11:07 a.m. UTC

First phase

First phase
Nov. 24, 2020, midnight UTC

End

Competition Ends
Jan. 15, 2021, 8:07 a.m. UTC

 Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion at LT-EDI 2021- EACL 2021

Hope is considered significant for the well-being, recuperation and restoration of human life by health professionals. Hope speech reflects the belief that one can discover pathways to their desired objectives and become roused to utilise those pathways[1-5]. Our work aims to change the prevalent way of thinking by moving away from a preoccupation with discrimination, loneliness or the worst things in life to building the confidence, support and good qualities based on comments by individuals. The goal of this task is to identify whether a given comment contains hope speech or not. We define the hope speech for our problem as "YouTube comments / posts that offer support, reassurance, suggestions, inspiration and insight". A comment / post within the corpus may contain more than one sentence but the average sentence length of the corpora is 1. The annotations in the corpus are made at a comment / post level.

The participants will be provided development, training and test dataset in English, Tamil, and Malayalam. To download the data and participate, go to codalab and click “Participate" tab. As far as we know, this is the first shared task on Hope Speech Detection.

Task:

This is a comment / post level classification task. Given a Youtube comment, the systems submitted by the participants should classify it into 'Hope speech', 'Not hope speech' and 'Not in intended language'. To download the data and participate, go to the Participate tab.

As far as we know, this is the first shared task on Hope Speech detection. More details at https://sites.google.com/view/lt-edi-2021

Results:

Tamil

Malayalam

English

Paper  name format should be: TEAM_NAME@LT-EDI-EACL2021: Title of the paper. 

Example: NUIG_ULD@LT-EDI-EACL2021: Hope Speech Detection for Equality, Diversity, and Inclusion

Following are some general guidelines to keep in mind while submitting the working notes.
- Basic sanity check for grammatical errors and reported results
- Papers should have sufficient information for reproducing the mentioned results- Papers should follow the appropriate style (We will use EACL 2021 style: details below)
- Check the papers for text reuse / Plagiarism. This includes self-plagiarism as well. We would like to stress this point as EACL is quite strict about it. Any paper found to have plagiarized content should be rejected without further consideration.
- Please ensure the author names do not have any salutations like Dr., Prof., etc in the final version
 
All submissions should be in Double column EACL 2021 format. Authors should use one of the EACL 2021 Templates below:
 
 
Reference:
[1] Harvey Milk. 1997. The hope speech. We are everywhere: A historical sourcebook of gay and lesbian politics,pages 51–53
[2] Edward  C.  Chang.   1998.   Hope,  problem-solving  ability,  and  coping  in  a  college  student  population:  Some implications for theory and practice. Journal of Clinical Psychology, 54(7):953–962
[3] Carolyn M. Youssef and Fred Luthans. 2007. Positive organizational behavior in the workplace: The impact of hope, optimism, and resilience. Journal of Management, 33(5):774–80
[4] Rob Cover. 2013. Queer youth resilience: Critiquing the discourse of hope and hopelessness in lgbt suicide representation.M/C Journal, 16(5).
[5]Snyder, C. R., Harris, C., Anderson, J. R., Holleran, S. A., Irving, L. M., Sigmon, S. T., et al.(1991). The will and the ways: Development and validation of an individual-differences measure of hope. Journal of Personality and Social Psychology, 60, 570-585.

Evaluation Criteria

We accept the test results only through the google form. The google form can be accessed from here.
Format of the submission file should be like below:
 
id textlabel
1 காற்றில் பரவும் வாய்ப்பு உள்ளது Non_hope_speech
2 Mrs.Sudha moorthy pathi pesungalen... Hope_speech
3 Realme india product not-Tamil
  • label column should only have labels in the form mentioned in the training data i.e. 'Hope_speech', 'Non_hope_speech', 'not-Tamil/not-English/not-Malayalam'.
  • id column is the index of the row. (Retain the sequence of the Youtuebe comments in the same order as provided in the test data)
 
Submission should be a zip file with your team name containing tsv files for individual langauges -  'teamname_language.tsv' e.g. the zip file may contain teamname_tamil.tsv, teamname_malayalam.tsv etc. The submission will be evaluated with weighted average F1-score. The results should be submitted on the google form.

Classification system’s performance will be measured in terms of weighted averaged Precision, weighted averaged Recall and weighted averaged F-Score across all the classes. Weighted averaged are averaging the support-weighted mean per label. Participants are encouraged to check their system with Sklearn classification report https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html

Results

Tamil

Malayalam

English

Terms and Conditions

By downloading the data or by accessing it any manner, you agree not to redistribute the data except for non-commercial and academic-research purposes. The data must not be used for providing surveillance, analyses or research that isolates a group of individuals or any single individual for any unlawful or discriminatory purpose.

You should cite this papers if you are using our data.

https://www.aclweb.org/anthology/2020.peoples-1.5/

@inproceedings{chakravarthi-2020-hopeedi,
title = "{H}ope{EDI}: A Multilingual Hope Speech Detection Dataset for Equality, Diversity, and Inclusion",
author = "Chakravarthi, Bharathi Raja",
booktitle = "Proceedings of the Third Workshop on Computational Modeling of People's Opinions, Personality, and Emotion's in Social Media",
month = dec,
year = "2020",
address = "Barcelona, Spain (Online)",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.peoples-1.5",
pages = "41--53",
abstract = "Over the past few years, systems have been developed to control online content and eliminate abusive, offensive or hate speech content. However, people in power sometimes misuse this form of censorship to obstruct the democratic right of freedom of speech. Therefore, it is imperative that research should take a positive reinforcement approach towards online content that is encouraging, positive and supportive contents. Until now, most studies have focused on solving this problem of negativity in the English language, though the problem is much more than just harmful content. Furthermore, it is multilingual as well. Thus, we have constructed a Hope Speech dataset for Equality, Diversity and Inclusion (HopeEDI) containing user-generated comments from the social media platform YouTube with 28,451, 20,198 and 10,705 comments in English, Tamil and Malayalam, respectively, manually labelled as containing hope speech or not. To our knowledge, this is the first research of its kind to annotate hope speech for equality, diversity and inclusion in a multilingual setting. We determined that the inter-annotator agreement of our dataset using Krippendorff{'}s alpha. Further, we created several baselines to benchmark the resulting dataset and the results have been expressed using precision, recall and F1-score. The dataset is publicly available for the research community. We hope that this resource will spur further research on encouraging inclusive and responsive speech that reinforces positiveness.",
}

 

@inproceedings{dravidianhopespeech-eacl,
title={Findings of the Shared Task on {H}ope {S}peech {D}etection for {E}quality, {D}iversity, and {I}nclusion},
author={Chakravarthi, Bharathi Raja and
Muralidaran, Vigneshwaran},
booktitle = "Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion",
month = April,
year = "2021",
publisher = "Association for Computational Linguistics",
year={2021}
}

Important Dates for shared task:

Task announcement: Nov 20, 2020

Release of Trail data: Nov 20, 2020

Release of Training data: Nov 20, 2020

Release of Test data: Jan 2, 2021

Run submission deadline: Jan 10, 2021

Results declared: Jan 15, 2021

Paper submission: Jan 30, 2021

Peer review notification: Feb 18, 2021

Camera-ready paper due: Mar 1, 2021

Workshop Dates: April 19-20, 2021

Bharathi Raja Chakravarthi, Insight SFI Research Centre for Data Analytics, Data Science Institute, National University of Ireland Galway

Vigneshwaran Muralidaran,  School of Computer Science and Informatics, Cardiff University, United Kingdom

Email: equalitydiversityinclusion.lang@gmail.com and bharathiraja.akr@gmail.com

First phase

Start: Nov. 24, 2020, midnight

Competition Ends

Jan. 15, 2021, 8:07 a.m.

You must be logged in to participate in competitions.

Sign In