SemEval-2020 Task5: Modelling Causal Reasoning in Language: Detecting Counterfactuals Forum

Go back to competition Back to thread list Post in this thread

> Explanation of Labels in the subtask 1

a. Does the Labels 1-5 denote the five human annotators?
b. In the evaluation task, the predicted label is matched for the similarity with the human annotators (label 1-5)?

Posted by: utkarshani @ Nov. 20, 2019, 10:24 p.m.

I guess what you mean is the trial data we released before to show the format of our dataset as required by SemEval. We just released the training data for task 1 & 2, please use the new data for the competition.
In the trail data, besides the label 1 to label 5, we provided 'gold_label', and the models only need to predict 'gold_label'. Actually we kept label 1 to label 5 annotated by annotators only to make sure the agreement and we have omitted them in our newly published training data, and we only provided 'gold_label' now.

Posted by: Ariel_yang @ Nov. 21, 2019, 1:13 a.m.
Post in this thread