Dear all:
The leaderboard currently shows entries of IoU > 1.
Forgive me but I am under the impression that IoU or Jacccard coefficient cannot be greater than 1.
If I am wrong, could you please share your definition of IoU here, so that everyone can adjust their code?
Posted by: ComputerArchaeologist @ June 12, 2021, 3:41 p.m.Yes we are aware of this, and we are investigating.
Thanks,
Posted by: simidjievskin @ June 12, 2021, 3:46 p.m.Hi.
Share you IoU metric function, please.
You are welcome! I use the fast.ai library from Univ. San Francisco. The IoU metric is the *same* as the Jaccard metric (according to wikipedia etc : https://en.wikipedia.org/wiki/Jaccard_index)
https://docs.fast.ai/metrics.html#Jaccard
Their source code is:
https://github.com/fastai/fastai/blob/master/fastai/metrics.py#L157
According to this source code they are just utilizing the scikit-learn metric underneath:
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html
I suggest that you look into Scikit-learn, as probably all participants know it.
Posted by: ComputerArchaeologist @ June 12, 2021, 5:12 p.m.Hi all,
First of all apologies for the inconvenience caused. Second, thanks to @ruslantau for discovering the bug and together with @ComputerArchaeologist for opening the discussion.
Indeed we use a standard IoU score which in fact is IoU= |A and B|/|A or B| (where A and B are boolean matrices) .
In this particular case, however, since we are operating with sparse matrices (due to server constraints), the union (or the logical or) in some cases was miscalculated.
This was evident from @ruslantus IoU (which has a theoretical maximum of 1) to be miscalculated.
This bug now is fixed, and all the submissions that were online were re-evaluated - that is why now it appears that everyone has 'double' of their submissions. In the next days we will also try to correct this and bring it back to 'normal' (wrt the number of submissions).
Once again, apologies and thanks for your understanding.
best,