Looks like there is a bug with scoring system (see my 9th submission I do not thin that this score is intended).
Dear organizers,
This is another bug in this competition scorer. Can we look at your scoring code to make sure it works as intended and bug free etc.?
Posted by: rakhlin @ April 20, 2018, 2:52 p.m.I would like to agree with @rakhlin it would be very nice if the scoring code was made public so that it can be verified and so that our own internal testing can be consistent with the way the contest is scored
Posted by: Queuecumber @ April 20, 2018, 6:17 p.m.adavydow: Thanks for reporting it! We'll fix it soon.
The bug is related to
"Please note the Unknown class (0,0,0) is not an active class used in evaluation. Pixels marked as Unknown will simply be ignored."
If you don't have predictions on the unknown class, the score would not be affected.
We'll discuss about releasing the scoring code and will let everyone know.
Posted by: jinghuang @ April 20, 2018, 6:25 p.m.Can you explain what is the intendended behaviour for predictions of unknown class?
Can we use unknown in our predictions?
Q: Can you explain what is the intended behaviour for predictions of unknown class?
A: The intended behaviour: if certain pixel is labeled as unknown in the ground truth, it won't affect the final score whatever label you predict on this pixel.
Q: Can we use unknown in our predictions?
A: Yes you can.
And how is unknow evaluated in our prediction?
My guess is: it gives a false negative to whatever is underground trouth of this pixel and no false positives.
Is it correct?
Q: And how is unknown evaluated in our prediction?
My guess is: it gives a false negative to whatever is underground truth of this pixel and no false positives.
Is it correct?
A: Your understanding is correct.
Here's a simple example of 3 classes (assuming 0 is unknown class), where rows denote the ground truths and columns are predicted labels.
Confusion Matrix:
0 1 2
0 1 2 3
1 2 5 6
2 5 3 14
1 2
TP 5 14
FP 3 6
FN 8 8
IOU 0.3125 0.5
mIOU 0.40625
Posted by: jinghuang @ April 20, 2018, 10:44 p.m.Thank you.
However another question appeared:
Now my cheaty solution is scored at NaN.
Is it intended or a temporary fix?
According to the metric from the previous post it should be scored slightly above zero.
What should we do to avoid NaN scores?
Sorry for misinformation in the previous post, my solution is still scored at 1, but there are 3 new solutions sent from my account.
Am I hacked or is it jury trying to fix the bug?
I can explain what's wrong with the checker and how to fix it if needed, either on the forum or in private.
Q: Is it intended or a temporary fix? What should we do to avoid NaN scores?
A: It just happened that we were testing the fix at the time.
Q: Am I hacked or is it jury trying to fix the bug?
A: It is jury trying to fix the bug by scheduling the re-run and we believe it's fixed now.