Just to confirm, should the final submission format should be a 784 Line file with either a 1 or a 0 on every line? Additionally, does it matter if the file is terminated with a newline?
Posted by: muakasan @ Jan. 21, 2018, 8:05 p.m.Hi
A little late, but this is also relevant for Task B: indeed, there should be 1 label per line (1/0 for Task A and 0/1/2/3 for Task B).
Each label should be followed by a newline, so the file should count 785 lines in total.
Kind regards
Cynthia
Hi organiser,
In the practice phase when we add any extra line then there are different results. Now in the evaluation phase, there are total 784 instances in task A and output file should be having 784 lines. There is no header or empty first line. Please explain why there should be 785 lines if the number of instances is 784? Kindly explain it.
Hi Ahmed
"In the practice phase when we add any extra line then there are different results" >> this can only be true if you added an extra line with metadata, or too many predictions.
The length of the predictions file should be 784 lines indeed, one line/prediction per instance in the test file. Whether or not a newline is added to the last label does not matter. Apologies for the confusion.
Kind regards
Cynthia
Thank you for the quick reply,
Yes, in practice when I left the first line empty, then results are not correct. But when I removed the first line and added one line per prediction then I get result accordingly. Yes, the length of the file should be 784 as same as the number of instances of the test set. Kindly let me know!
1- Will the result of task_A be announced after task B?
2- Is we will see the leaderboard after evlaution phase of both task ?
Yes, the results of both tasks will be announced after the entire evaluation period, further details will be communicated through the SemEval-2018 Task 3 Google group.
Kind regards
Cynthia
Thank you.
Posted by: usmanahmed189 @ Jan. 23, 2018, 6:57 a.m.