Competition

Evaluation Criteria 

For evaluation, the classic Precision and Recall metrics over each emoji are used. The official results will be based on Macro F-score, as the fundamental idea of this task is to encourage systems to perform well overall, which would inherently mean a better sensitivity to the use of emojis in general, rather than for instance overfitting a model to do well in the  three or four most common emojis of the test data. Macro F-score can be defined as simply the average of the individual label-wise F-scores.  We will also report Micro F-score for informative purposes.

The official evaluation script can be found here.

Terms and Conditions

By submitting results to this competition, you consent to the public release of your scores at the SemEval-2018 workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.

You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.

You further agree that your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.

You agree not to redistribute the test data except in the manner prescribed by its licence.

The dataset can be downloaded here

The zip file includes three subdirectories: train, test, and trial (the latter may be used as development).

Test and Trial tweets along with their corresponding labels are already available, but, due to Twitter restrictions, you will need to download the tweets from the training set yourself. It is a smooth process and all the instructions and commands can be found in the train folder.

The dataset can be cited as:

@InProceedings{semeval2018task2,
  title={{SemEval-2018 Task 2: Multilingual Emoji Prediction}},
  author={Barbieri, Francesco and Camacho-Collados, Jose and Ronzano, Francesco and Espinosa-Anke, Luis and Ballesteros, Miguel and Basile, Valerio and Patti, Viviana and Saggion, Horacio},
  booktitle={Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018)},
  year={2018}, 
  address={New Orleans, LA, United States},
  publisher = {Association for Computational Linguistics}
 } 

The results can be found here.
There are two tabs, one for English and one for Spanish. The columns show:
username, macro f1, precision and recall, accuracy, and f1 for all the emojis (starting from label 0). 

System Description Papers

For those teams that have not completed the survey yet, please do it as soon as possible.
The deadline for the submission of system descriptions is Mon 26 Feb 2018 by 23:59 GMT-12:00. The system descriptions must be submitted through SOFTCONF following the detailed instructions found at this URL: http://alt.qcri.org/semeval2018/index.php?id=papers