Hi, I found the some top validation benchmark results use the train + validation data to train the model, such as using pretrained EDVR model trained with validation data.
And I think the better testing benchmark result can be obtained by using the train + validation data to train the model.
If you only check the submitted code, the participant can submit the pretrained model trained by validation data, but remove the code about the training with validation data. The cheating behavior is difficult to observe if the organizer only check the code and does not re-train the model.
Hi chuminglin,
In the 2019 challenges when EDVR was submitted, we didn't have such a data usage policy for training.
In this NTIRE 2021 challenge, we have announced the training rule in multiple ways, and using the validation/test data is not allowed.
By submitting the fact sheet with the statements on the first page, the challenge participants claim themselves to have followed the rules.
We did not allow using the validation data for training for several reasons, mostly to encourage better analysis.
Besides, training with the validation set does not guarantee better accuracy on the test set.
During the testing phase, except for the validity of the submission file, the participants will not get scored feedback.
CodaLab leaderboard will only show zero-ed scores to prevent fitting to the test set by changing the training conditions.
The division of training/validation/test sets is designed to minimize the correlation between the evaluation sets.
Also in terms of size, the validation set is 1/8 of the training set which is not very big.
Using extra data (which is allowed) could be practically more beneficial rather than using the REDS validation data.
Nevertheless, if any submissions with violations are witnessed, they will be penalized.
Most likely the submissions will be rejected and the violating teams will be banned from participating in the following NTIRE/AIM/Mobile AI workshops, etc.
We will continue to check the factual details with the participants after the preliminary test scores are announced.
We will do our best in organizing the challenge.
Best,
Seungjun