SemEval 2020 Task 4 - Commonsense Validation and Explanation (ComVE) Forum

Go back to competition Back to thread list Post in this thread

> Subtask C submission

Hello

I tried to submit file for subtask C, but I got error:

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
ERROR:root:Missing prediction for instance '1'.

I have checked everything and I think that file and format is ok.
Error log indicates that it is probably looking for sample id of '1' which is not present in TEST data.

Posted by: TMLab @ Feb. 28, 2020, 1:18 p.m.

Hi

I have put another submission with BAD ids (ids 1 -> 1000) like in sample submission (from starting kit) and it WORKED,
so like I supposed there is an error in submission interface now because it expects wrong sample ids.

Please correct the interface and I think it would be fair to cancel my both submissions to subtaskC, since those worked only for error detection.

Best!

Posted by: TMLab @ Feb. 28, 2020, 3:30 p.m.

Hi, we are sorry about the error. The issue is now fixed and we also rerun your first submission and get reasonable scores. Your second submission is deleted. Thanks!

Posted by: Shuailong @ Feb. 28, 2020, 6:02 p.m.

Hi! Is the score in subtask C available? How do I know I have a valid submission? WIll the program check the submission format?

Posted by: cdjhz @ March 2, 2020, 4:24 a.m.

Hi cdjhz, the official score will only be available when the evaluation ends, therefore we hide the leaderboard during the evaluation period. You may be able to see your score for Subtask C after UTC March 4 00:00.

For your submission, we noticed that the file name is not correct, therefore it's not properly evaluated. Please use "subtaskC_answers.csv" as the file name. Please refer to the starting kit for detailed instruction.

Posted by: Shuailong @ March 2, 2020, 4:40 a.m.

"You may be able to see your score for Subtask C after UTC March 4 00:00." May I ask why the score for Subtask A is not shown yet.

Posted by: ZhengxianFan @ March 2, 2020, 9:17 a.m.

Correction: Sorry for the misleading information provided above. Actually the final results will be only available after the whole evaluation period ends, which is March 11th. If your submission has any problem we will try our best to let you know in time.

Posted by: Shuailong @ March 2, 2020, 9:31 a.m.

Is there a way to check if our submission is successful or not?
It just shows status FINISHED but there is another column with a TICK on the top (which indicated submission on the leaderboard) and my submission didn't get a tick on it.
Is it something I need to worry about? Let me know If I'm missing anything.

Posted by: anandhperumal02 @ March 2, 2020, 7:25 p.m.

It seems that when the leaderboard is hidden, you cannot choose which submission to submit to leaderboard, unlike in practice mode. So the system will automatically use your last submission and submit to leaderboard.

If you did not receive an error message usually it means your submission is successful. We will also monitor your submissions manually incase your submission is not properly evaluated. Your submission was properly evaluated, so do not worry.

Posted by: Shuailong @ March 3, 2020, 6:38 a.m.
Post in this thread