Hello!
As noted :
" Evaluation Metric
We will use pixel-wise Intersection over Union (IoU) score as our evaluation metric, defined as: True Positive / (True Positive + False Positive + False Negative). "
There are two ways to calculate the IoU of the all images：
1. compute the IoU of every single image: IoU(1), IoU(2), ... , IoU(1243),
and then calculate the average .
2. sum the TP, FP and FN over the 1243 images, and then caculate the IoU using the summed TP, FP and FN .
In the first one, the pixels in different images account differently for the final score： the pixels in the image with fewer roads would account more than the pixel in the images with more roads.
It seems that the current valid script is using the first one.
Is it fixed as the final Evaluation Metric?
Thanks!
I'd also love to know the eval metrics (or maybe if the eval script can be released as per my separate thread).
I tried eval metrics on my held out data and I get numbers closer to previous scale (~0.80). I tried both the ways suggested in this thread, both give me ~0.80 value for my model.
I can understand it might be because validation data is significantly different than training data but that seems somewhat unlikely.
Posted by: naman @ April 7, 2018, 6:48 a.m.are you sure you do not count in the background when evaluation.
Posted by: fanneilyang @ April 7, 2018, 5:40 p.m.I think I found my mistake, I forgot to binarize my prediction and was using 255 for truth and 0 for false.
But even after fixing above I can't get close to what the official script is showing on Leaderboard.
Posted by: naman @ April 7, 2018, 5:45 p.m.but for me, i got the close evaluation score on my divided validation data.
Posted by: fanneilyang @ April 7, 2018, 5:47 p.m.but for me, i got the close evaluation score on my divided validation data.
Posted by: fanneilyang @ April 7, 2018, 5:47 p.m.I see, and did you use (1) or (2) mentioned in this thread? For me both are very far.
I am not sure if it's allowed or not, but it'd be great if you can share your eval script. I am sure it's just couple of lines.
Posted by: naman @ April 7, 2018, 5:49 p.m.both of them. i can share it to you, but not sure if it is permitted here.
I see, that's fair. Let's wait for the organizers to reply to one of these two threads.
Thanks for your help.
Posted by: naman @ April 7, 2018, 6:26 p.m.jxd is right - we're using approach 1:
1. compute the IoU of every single image: IoU(1), IoU(2), ... , IoU(1243),
and then calculate the average.
As fanneilyang pointed out, the two approaches shouldn't lead to very different results.
Posted by: jinghuang @ April 10, 2018, 7:11 p.m.