Competition

Evaluation

For the evaluation of the results of task A and B different strategies and metrics are applied in order to allow for more fine-grained scores.

TASK A.

Systems will be evaluated using standard evaluation metrics, including accuracy, precision, recall and F1-score. The submissions will be ranked by F1-score.
The metrics will be computed as follows:

• Accuracy = (number of correctly predicted instances) / (total number of instances)
• Precision = (number of correctly predicted instances) / (number of predicted labels)
• Recall = (number of correctly predicted labels) / (number of labels in the gold standard)
• F1-score = (2 * Precision * Recall) / (Precision + Recall)

TASK B.

Systems will be evaluated on the basis of two criteria: partial match and exact match.

• Partial match: each dimension to be predicted (Hate Speech HS, Target TR and Aggressiveness AG) will be evaluated independently of the others using standard evaluation metrics, including accuracy, precision, recall and F1-score as defined above. The report for each participant will include all the measures and a summary of the performance in terms of macro-average F1-score, computed as follows:
  
  
  
• Exact match: all the dimensions to be predicted will be jointly considered computing the Exact Match Ratio (Kazawa, 2005). Given the multi-label dataset consisting of n multi-label samples (x_i,Y_i), where x_i denotes the i-th instance and Y_i represents the corresponding set of labels to be predicted (HS ∈ {0,1}, TR ∈ {0,1} and AG ∈ {0,1}), the Exact Match Ratio (EMR) will be computed as follows:

            where Z_i denotes the set of labels predicted for the i-th instance and I is the indicator function.

           The submissions will be ranked by EMR. This choice is motivated by the willingness of capturing the most difficult task of capturing the entire phenomena, and therefore to identify the most dangerous behaviours against the targets.

Scoring program

The evaluation script is available in this GitHub repository:

https://github.com/msang/hateval/tree/master/SemEval2019-Task5/evaluation

NOTE

During the Practice phase, the prediction files submitted by participants to the task page will be evaluated for the task A, and for demonstration purposes only; if participants wish to test the script on prediction files for task B as well, they could use the version available in the GitHub repository.

For the Development and Evaluation phases, the script will provide a complete evaluation for each language and task for any submitted file, provided that the latter meet the submission requirements (see Submission Instructions).

Submission Instructions

The official hatEval evaluation script takes one single prediction file as input, for each task and for each language, that MUST be a TSV file structured as follows:

Task A

id[tab]{0|1}

e.g.

101[tab]1

102[tab]0

103[tab]1

Task B

id[tab]{0|1}[tab]{0|1}[tab]{0|1}

e.g.

101[tab]1[tab]1[tab]1

102[tab]0[tab]0[tab]0

103[tab]1[tab]1[tab]0

104[tab]1[tab]0[tab]0

105[tab]1[tab]0[tab]1

Contrary to the trial and training set, the submission files do NOT have the header in the first line.

File names

When submitting predictions to the task page in Codalab, one single file should be uploaded for each task and language, as a zip-compressed file, and it should be named according to the language and task predictions are submitted for, therefore:

• en_a.tsv for predictions for taskA-English
• es_a.tsv for predictions for taskA-Spanish
• en_b.tsv for predictions for taskB-English
• es_b.tsv for predictions for taskB-Spanish

NOTE: 

For the Practice phase, more than one submission is allowed, BUT for the task A only. While during the Development and Evaluation phases, participants are free to submit their system's predictions for each language and task separately.

For the Development phase participants will be able to make more than one submission for each language and task, while for the Evaluation phase, a maximum of 2 submissions has been set for each language and for both task A and B, but please note that only the final valid one is taken as the official submission for the competition.

Terms and conditions

By submitting results to this competition, you consent to the public release of your scores at the SemEval-2019 workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.

You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.

You further agree that your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.

You agree not to redistribute the test data except in the manner prescribed by its licence.