Memotion Analysis

Organized by semeval2019 - Current server time: Oct. 22, 2019, 5:43 a.m. UTC


First phase
Sept. 4, 2019, midnight UTC


Competition Ends
Jan. 10, 2020, 11:32 p.m. UTC

Information on social media comprises of various modalities such as textual, visual and audio. NLP and Computer Vision communities often leverage only one prominent modality in isolation to study social media. However, computational processing of Internet memes needs a hybrid approach. The growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter further suggests that we can not ignore such multimodal content anymore. To the best of our knowledge, there is not much attention towards meme emotion analysis. The objective of this proposal is to bring the attention of the research community towards the automatic processing of Internet memes. The task Memotion analysis will release 8K annotated memes - with human annotated tags namely sentiment, and type of humor that is, sarcastic, humorous, or offensive.

The Multimodal Social Media
In the last few years, the growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter has become a topic of immense interest. Memes, one of the most typed English words (Sonnad, 2018) in recent times. Memes are often derived from our prior social and cultural experiences such as TV series or a popular cartoon character (think: One Does Not Simply - a now immensely popular meme taken from the movie Lord of the Rings). These digital constructs are so deeply ingrained in our Internet culture that to understand the opinion of a community, we need to understand the type of memes it shares. (Gal et al., 2016) aptly describes them as performative acts, which involve a conscious decision to either support or reject an ongoing social discourse. Online Hate - A brutal Job: The prevalence of hate speech in online social media is a nightmare and a great societal responsiblity for many social media companies. However, the latest entrant Internet memes (Williams et al., 2016) has doubled the challenge. When malicious users upload something offensive to torment or disturb people, it traditionally has to be seen and flagged by at least one human, either an user or a paid worker. Even today, companies like Facebook and Twitter rely extensively on outside human contractors from start-ups like CrowdFlower, or companies in the Philippines. But with the growing volume of multimodal social media it is becoming impossible to scale. The detection of offensive content on online social media is an ongoing struggle. OffenseEval (Zampieri et al., 2019) is a shared task which is being organized since the last two years at SemEval. But, detecting an offensive meme is more complex than detecting an offensive text – it involves visual cue and language understanding. This is one of the motivating aspects which encourages us to propose this task. Multimodal Social Media Analysis - The Necessity: Analogous to textual content on social media, memes also need to be analysed and processed to extract the conveyed message. A few researchers have tried to automate the meme generation (Peirson et al., 2018; Oliveira et al., 2016) process, while a few others tried to extract its inherent sentiment (French, 2017) in the recent past. Nevertheless, a lot more needs to be done to distinguish their finer aspects such as type of humor or offense. We hope Memotion analysis - the task will bring research attention towards the topic and the forum will be the place to continue relevant discussions on the topic among researchers.

The Memotion Analyis Task

Task A- Sentiment Classification: Given an Internet meme, the first task is to classify it as positive or negative meme. We presume that a meme is not neutral.

Task B- Humor Classification: Given an Internet meme, the system has to identify the type of humor expressed. The categories are sarcastic, humorous, and offensive meme. If a meme does not fall under any of these categories, then it is marked as a other meme. A meme can have more than one category. For instance, Fig 3 is an offensive meme but sarcastic too.

Task C- Scales of Semantic Classes: The third task is to quantify the extent to which a particular effect is being expressed. Details of such quantifications is reported in the Table 1. Appropriate annotated data will be provided.

Evaluation Criteria
The metric for evaluating the participating systems will be as follows. For the Task-A, and Task-B, we will use averaged F1 score across all the classes (Positive/Negative-Sentiment and Sarcasm/Humor/Offensive/Regular- Semantic Analysis) with macro-averaged recall, since the latter has better theoretical properties than the former (Esuli and Sebastiani, 2015), and provides better consistency. For the Task-C, we will use averaged Mean Absolute Error across all the classes. Baseline: A baseline system with clear description of the system - ML model details, features etc. will be made available through Github repository. We will host the task in Codalab.

Trial data ready: July 31, 2019
Training data ready: September 4, 2019
Test data ready: December 3, 2019
Evaluation start: January 10, 2020
Evaluation end: January 31, 2020
Paper submission due: February 23, 2020
Notification to authors: March 29, 2020
Camera-ready due: April 5, 2020
SemEval workshop: Summer 2020

Dr. Amitava Das.
Wipro AI Labs, Bangalore, India
Mahindra École Centrale, Hyderabad, India.

Dr.Tanmoy Chakraborty.
Indraprastha Institute of Information Technology Delhi, India.

Dr.Soujanya Poria.
Nanyang Technological University, Singapore.

Dr. Björn Gambäck.
Norwegian University of Science and Technology, Norway.

Chhavi Sharma.
Indian Institute of Information Technology, Sri City, India.

Student Volunteer

William Scott Paka.
Indraprastha Institute of Information Technology, Delhi.

Deepesh Bhageria.
Indian Institute of Information Technology, Sri City, India.

Terms & Conditions

By submitting results to this competition, you consent to the public release of your scores at the SemEval workshop and in the associated proceedings, at the task organizers' discretion. Scores may include but are not limited to, automatic and manual quantitative judgments, qualitative judgments, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.

You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgment that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.

You further agree that your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.

By downloading the data or by accessing it any manner, You agree not to redistribute the data except for the purpose of non-commercial and academic-research. The data must not be used for providing surveillance, analyses or research that isolates a group of individuals or any single individual for any unlawful or discriminatory purpose.

For any queries contact us on Email: 

First phase

Start: Sept. 4, 2019, midnight

Competition Ends

Jan. 10, 2020, 11:32 p.m.

You must be logged in to participate in competitions.

Sign In