Discover the mysteries of the Maya @ ECML PKDD 2021- Integrated Image Segmentation Challenge Forum

Go back to competition Back to thread list Post in this thread

> IoU greater than one

Dear all:

The leaderboard currently shows entries of IoU > 1.

Forgive me but I am under the impression that IoU or Jacccard coefficient cannot be greater than 1.

If I am wrong, could you please share your definition of IoU here, so that everyone can adjust their code?

Posted by: ComputerArchaeologist @ June 12, 2021, 3:41 p.m.

Yes we are aware of this, and we are investigating.


Posted by: simidjievskin @ June 12, 2021, 3:46 p.m.

Share you IoU metric function, please.

Posted by: ruslantau @ June 12, 2021, 4:26 p.m.

You are welcome! I use the library from Univ. San Francisco. The IoU metric is the *same* as the Jaccard metric (according to wikipedia etc :

Their source code is:
According to this source code they are just utilizing the scikit-learn metric underneath:

I suggest that you look into Scikit-learn, as probably all participants know it.

Posted by: ComputerArchaeologist @ June 12, 2021, 5:12 p.m.

Hi all,

First of all apologies for the inconvenience caused. Second, thanks to @ruslantau for discovering the bug and together with @ComputerArchaeologist for opening the discussion.
Indeed we use a standard IoU score which in fact is IoU= |A and B|/|A or B| (where A and B are boolean matrices) .
In this particular case, however, since we are operating with sparse matrices (due to server constraints), the union (or the logical or) in some cases was miscalculated.
This was evident from @ruslantus IoU (which has a theoretical maximum of 1) to be miscalculated.

This bug now is fixed, and all the submissions that were online were re-evaluated - that is why now it appears that everyone has 'double' of their submissions. In the next days we will also try to correct this and bring it back to 'normal' (wrt the number of submissions).

Once again, apologies and thanks for your understanding.

Posted by: simidjievskin @ June 12, 2021, 6:28 p.m.
Post in this thread