NTIRE 2021 Video Super-Resolution Challenge - Track1. Spatial Forum

Go back to competition Back to thread list Post in this thread

> How to judge whether the model uses validation set for training?

Since the ground-truth (GT) of the verification set has been released, it is possible to use it as a training set.
So is it better to have a part of the test set as an online validation assembly?

Posted by: zheng222 @ Jan. 5, 2021, 5:41 a.m.

Hi zheng222,

As mentioned in the evaluation tab, we do not recommend using the validation set for training purposes.

While the official benchmarks on the test data can be used for comparison, most of the analysis including the ablation study is done on validation data.
If validation data is used for training, each method is not eligible to compare with the others.
I believe using validation GT for training is not much helpful for the community.

To check what is done during the training, we are going to look into the details on the fact sheet the source code.

The test set accuracy will not be noticed on CodaLab benchmarks and the participants will not get intermediate feedbacks in terms of PSNR and SSIM. (unless noticed otherwise)
In my experiences, the validation set accuracy had a close correlation with the test set accuracy.
I expect the test set evaluation results will not be necessary to fine-tune the models.


Posted by: SeungjunNah @ Jan. 5, 2021, 10:49 a.m.
Post in this thread