SemEval-2019 Task 12 - Toponym Resolution in Scientific Papers Forum

Go back to competition Back to thread list Post in this thread

> Questions about the evaluation

Dear Davy,

There are some questions about the evaluation:
1. Can I see the score of my submission?
2. What's the difference between submission status "Finished" and "Submitted"?
3. Can you show the results with higher precision (maybe remain 2-3 digits after decimal point)?
4. Subtask-1 and 3 are in the same phase, where can I see the submission summary of subtask-3?

Posted by: HAONANL5 @ Jan. 11, 2019, 6:06 a.m.

Dear HAONANL5,

1. Can I see the score of my submission?
When the leaderboard is hidden the scores obtained by each participant are not communicated to them. There is only 2 options in codalab, showing or not showing the leaderboard, I can't change this. I will communicate to every participating team their own scores at the end of the evaluation period by mail.
2. What's the difference between submission status "Finished" and "Submitted"?
When you submit your files the sequences of status should be submitting-submitted-running-finished (or submitting-submitted-running-failed if there is any problem when computing the scores). Codalab is running on a shared server, other shared tasks are running in the same time than ours. It may happen that your submission stays in the submitted status for a while before being processed by the server. It may also happen that the submission stays in "submitted" for ever (we had this case when testing codalab before the competition). If you don't see any changes on the submitted status after 1h, please let me know, I can re-run the submission forcing the server to re-queue the file (I guess it is doing that...). If we still can't finish the run I will write an email to codalab team for help.
3. Can you show the results with higher precision (maybe remain 2-3 digits after decimal point)?
Yes, the scoring file will display the scores with 13 decimals. I set up the leaderboard to display 4 decimals, if the leaderboard does not display them, I can't do anything. I cannot change the evaluation metrics without recreating the competition.
4. Subtask-1 and 3 are in the same phase, where can I see the submission summary of subtask-3?
In the results tab on the top you can find the list of the sub-tasks and can click on the one you want to check. For subtask 3 this should be the purpule button "Sub-task 3: Evaluation Resolution". If you cannot see the results this is because the leaderboard is hidden, but I can see that you have already submitted 1 run for task 3

Best regards,
Davy

Posted by: dweissen @ Jan. 11, 2019, 4:13 p.m.
Post in this thread