I’ve submitted to the validation server, but it showed significant difference to my local test on a random selected validation dataset. So I want to know how the metrics are calculated on the validation server. I suppose there maybe two ways of doing this.
1) For each reference image, calculate the SRCC and PLCC among its distortions. Then the final SRCC/PLCC is averaged among all reference images.
2) For all the distortion images of all the references, predictions and labels were put together. Then SRCC and PLCC were calculated.
For the two alternatives 1) and 2), which one is the right way to calculate PLCC/SRCC ?
Posted by: SunGaofeng @ March 2, 2021, 4:25 a.m.Hi,
We have a demo code for calculating SRCC and PLCC on the evaluation page. We follow the calculation process used by previous works such as TID. Like you have mentioned: For all the distortion images of all the references, predictions and labels were put together. Then SRCC and PLCC were calculated.
Thanks,
Posted by: JinjinGu @ March 2, 2021, 12:22 p.m.