"The submitted result must be from the same method that generated the last submission to the CodaLab. We will check the consistency. Otherwise, the submission is void."
Does the submitted result mentioned in the rule refer to the test set or the validation set?
Is it necessary to submit the validation set results generated by the final version of the model to CodaLab?
Posted by: Veigar @ March 20, 2021, 2:53 a.m.Dear Veigar,
"The last submission to the CodaLab" refers to the last submission during the testing phase.
You don't need to submit the validation set results.
The development phase submission & validation data PSNR are not in the scoring criteria.
We will check if the validation data accuracy severely deviates from the testing data accuracy but the final ranks are only affected by the test set results.
Best,
Seungjun
Dear Seungjun,
thanks for you reply!
How the validation data accuracy is checked? Is it checked by the source code we submitted via email or by the PSNR on the development leaderboard?
Thank you!
Posted by: Veigar @ March 20, 2021, 5:57 a.m.We will evaluate with the submitted code & results.
Validation data: reproduce ourselves from the submitted code.
Test data: reproduce ourselves from the submitted code & evaluate directly on the submitted results via CodaLab & email. Consistency will be checked.