Competition

_{Important update: Trial test labels released on January 5th 2019. Trial phase submissions not anymore accepted on CodaLab. You can trial evaluate your systems using the evaluation script provided in the task GitHub repo.}

Introduction

Welcome to the pilot challenge on suggestion mining! 

Suggestion mining can be defined as the extraction of suggestions from unstructured text, where the term 'suggestions' refers to the expressions of tips, advice, recommendations etc. Consumer opinions towards commercial entities like brands, services, and products are generally expressed through online reviews, blogs, discussion forums, or social media platforms. These opinions largely express positive and negative sentiments towards a given entity, but also tend to contain suggestions for improving the entity or advice to the fellow consumers. Traditional opinion mining systems mainly focus on calculating the sentiment distribution by means of Sentiment Analysis methods. A suggestion mining component can therefore extend the capabilities of traditional opinion mining systems, leading to a number of new use cases. Such systems can empower both public and private sectors by extracting suggestions which are spontaneously expressed on various online platforms, enabeling organisations to collect suggestions from much larger and varied sources. 

Suggestion mining remains a relatively young area as compared to Sentiment Analysis, especially in the context of recent advancements in neural network based approaches for learning feature representations. This research could drive the engagement of both commercial entities, as well as the research communities working on problems like opinion mining, supervised learning, representation learning, etc. From a linguistic viewpoint, suggestion mining includes extra propositional aspects like mood and modality, sarcasm, compound sentences, etc. It is observed that in some cases the grammatical properties of a sentence can alone decide its label, while at times semantics can play a significant role. 

In this pilot SemEval task, we introduce suggestion mining as a simple task of classifying given sentences into suggestion and non-suggestion classes. With this task, we aim to evaluate the submitted systems in two settings, domain specific and cross domain training. We are also looking forward to systems which employ transfer learning methods, in light of a limited training dataset.

Suggestions mining from online reviews and forums

Considering the domain of discussion forums, all the posts within a single thread are centered around answering a question or replying to a topic defined in the first post of the thread. A number of times these questions/topics are advice seeking, for example, "Advice for our week in Vienna" thread on a travel discussion forum. The answers on discussion forums are conversational and may contain additional contextual and information than what the first post sought. A suggestion mining system can extract the exact sentences where the advice is expressed, which renders a suggestion mining system as a suggestion summarisation system for discussion forums. Another type of discussions forums are dedicated suggestion forums pertaining to a commercial entity. For example, a suggestion forum to share the platform capability requests and general ideas for improving the Windows developer platform. Such forums operate by developers or customers posting messages explaining the improvisations they want to see in the product, which is different from online reviews where the main objective is to provide positive or negative ratings and rest of the information is additional. In the case of developer suggestion forums, the contextual text describing the functionality of the product gets repetitive over a large number of posts and a suggestion mining system would really be helpful to extract sentences containing concrete suggestions. A suggestion post is shown in the image below, where only first and last sentences express suggestions while the rest can be considered as the context.

 Some examples of suggestions found among the text from different opinion platforms are listed below. 

Challenges

Some of the observed challenges in suggestion mining are:

• Class imbalance: Suggestions appear sparsely in domains like hotel reviews (6% - 13%), which leads to higher data annotation costs as well as results in a skewed class distribution for model training.
• Figurative expressions: The text from social media and other sources usually contain figurative use of language. For example, `Try asking for extra juice at breakfast - its 22 euros!!!!!' is more of a sarcasm than a suggestion. Therefore, a sentence in the form of suggestions may not always be a suggestion, and vice versa. 
• Context dependency: At times, context plays a major role in determining whether a sentence is a suggestion or not. For example, `There is a parking garage on the corner of the Forbes showroom.' can be perceived as a suggestion (for parking space) when it appears in a restaurant review and the human annotator gets to read the full review, while the same sentence would not be labeled as a suggestion if it is present in the description of the locality of a Forbes showroom.
• Long and complex sentences: Often, a suggestion is only expressed in one part of a long sentence, or appears as a very long sentence, like, `I think that there should be a nice feature where you can be able to slide the status bar down and view all the push notifications that you got but you didn't view, just like android and IOS, but the best part is that it fixes many problems like when people wanted a short cut to turn WiFi on and off and data on and off so that would be a nice feature to have 2'. This poses challenges to the training algorithms for learning effective features, as well as for certain pre-processing steps like part of speech tagging. 

Relevant Publications

Task Organisers

• Sapna Negi

• Tobias Daudert

• Paul Buitelaar

• Saeedeh Shekarpour