Dear organizer,
I am wondering if the ranking on the leaderboard meaningful? Because it does not seem to be related to any of the listed measurements like PSNR or SSIM. Is it based on the mIOU of segmentation maps which is not visible to us?
If the ranking is meaningful, it seems to be of users rather than the entries. Is there any way to compare different entries uploaded by the same user?
I am looking forward to your response. Thank you.
Posted by: c.chen @ July 1, 2020, 2:07 p.m.Dear c.chen,
The leaderboard is there to give an indication of performance, we are not using a hidden metric to rank the submissions in the leaderboard. As far as I know, there is no way to compare among the entries of a single participant.
Here are some details as posted in our evaluation page:
"For the final Perceptual ranking we will conduct a user study to compute Mean Opinion Score (MOS) for the top solutions in the challenge. Due to the complexity of the task we will conduct the user study and compute MOS only for the results submitted for the final test phase. Complementary to MOS, we will report also FID, mIoU, PSNR and SSIM.
The validation server provides feedback under the form of PSNR and SSIM scores, the other measures will be computed offline and reported for the final ranking after the end of the challenge."
Best,
Evan