Dear all,
We have been receiving several questions about the usage of validation data and the pre-trained models from the validation data.
Technically, we cannot force someone not to use the validation GT for whatever the purpose as the data has been publicly available from NTIRE 2019.
However, using the validation data for training does not follow the basic idea of having a validation set.
Our main goal of hosting competitions and making the dataset public is to pursue the good for the community.
Validation data exists to let people
1) validate the effectiveness of their methods and the modifications on the data outside the training set.
2) compare their methods without using the test set.
If validation data is used for training, we have no (or little) data to check the generalizability of the developed methods.
If someone chooses to construct their own validation set, the comparison with the other methods gets complicated.
As the environment for training/validation is different, it is difficult to perform scientific analyses between them.
To compare such methods in a fair manner, additional effort must be done: one needs to retrain all the methods in a unified environment.
The community cannot get concrete knowledge without the 3rd party effort.
We understand that such concerns wouldn't have been an issue if we did not release the validation GT.
However, CodaLab online server provides limited power to evaluate on 1/10 of the validation set.
Thus, we decided to release the validation GT to let people analyze their solutions on their own, not to encourage them to use it for training.
We keep our statement: we don't recommend using the validation data for training for the good of the community.
Should a pre-trained model (trained from the validation data) be used, the model can be trained from scratch using the training data and then be used for further development.
Best,
Seungjun
Hi,
Good Job!
But how could you confirm all the participant follow rule strictly?(Including not using validation for training or leaky medidate frame for training video frame interpolation network)
By the way, Could participant use any other external datasets for trainning or pretrain for full or partial model?
Posted by: DeepBlueAI @ Jan. 20, 2021, 10:45 a.m.Dear DeepBlueAI,
We will check the fact sheet for factual details.
For the use of odd-numbered frames, we will check the reproducibility.
We will only provide even-numbered frames as input.
At training, except for the validation set, participants can use extra datasets, too.
In that case, participants should describe in detail what kind of data is used in which manner.
Best,
Seungjun
Hi Seungjun,
Just post to remind that，besides being used as input，
the odd-numbered low resolution frames in the test set can also be used to supervise the interpolation task.
How could one tell if a solution has used the odd-numbered frames as supervision labels or not?
Hi MuteGrab,
In the REDS dataset, we have training/validation/test data.
Training data has both even and odd frames, LR and HR.
All of them can be used for training.
Restriction to the even frames is only applied at inference time.
Validation data also contains both even and odd frames, LR and HR.
You can use the even-numbered LR frames as input.
The HR frames should only be used for local and online evaluation during development.
Test data is meant to provide even-numbered LR frames.
However, due to historic reasons, odd-numbered LR frames are also publicly available.
Challenge participants should only use even-numbered LR frames as input.
The odd-numbered LR frames should not be used for any purposes.
If the odd-numbered LR frames are involved in any part of the development & deployment of a solution, the solution should be submitted to the Video Super-Resolution Track 1. Spatial.
Besides the REDS data, external data can be used for training.
Usage of external data does not affect the rank but should be clearly described in the fact sheet. (what, why, and how)
We will soon make an announcement so that every participant in the final phase acknowledges the rule.
ex) stating the agreement to the rules in the fact sheet