Dear organizer
How exactly is the scoring calculated for one prediction mask against the ground truth? For example when there is no change in any pixel in the ground truth mask and the predicted mask also contains only the label for no change for every pixel is the score then 1 for this data set? And is the calculation for the overall score than sum(score_i)/N for i in N with N number of provided masks?
Hi, for one prediction and gt mask, iou score is calculated per each class seperately and averaged over all occurring classes in the GT. The overall score is averaged exactly how you described.
Hello, I think the evaluation is not happening as described. According to the described method, 1024*1024 png files where values in all the pixels in 0 should result in a score of 0 (as intersection of all classes would be 0, resulting in iOu 0 for all classes) according to the given method. However this type of blank image results in a mIoU of 0.228853 on the validation set. This score of 0.228853 makes no sense to me.
Posted by: Aryal007 @ May 23, 2021, 2:02 a.m.Hi, the evaluation is happening exactly as described. In the description we specify that 0 is the no-change class which is evaluated as all other classes. Meaning yes when you upload 1024x1204 change maps which contains only 0 values (blank images) you will get a score>0 since some of the predictions =class 0 are correct. However it is no different than uploading 1024x1024 change maps which contains only 2 class ids and getting some score. We cannot avoid this here but we will check this for the paper submissions. Also as LBeumer said, if there is no change in any pixel in the ground truth mask and the predicted mask also contain only the label for no change for every pixel, then the score is 1, which does not happen in practice, however.
Posted by: aysim @ May 23, 2021, 8:36 a.m.If that's the case then could it be possible that the best mIoU in the leaderboard is not technically the best performing method? Will the team with the highest score in the leaderboard still win the competition?
Posted by: Aryal007 @ May 24, 2021, 5:02 a.m.