The bug is that duplicates are not detected and removed in eval.py. In this situation, if the ID of the query image is 'nm_0', and there are three 'nm_0' images (e.g. A, B, C) in the gallery.
The ranking with no duplicates: A, B, C, and other images. The mAP will be (1/3)*(1+1+1)=1.0
Howerver, the ranking with duplicates: A, B, C, A and other images. The mAP will be (1/3)*(1+1+1+1)>1.0
The bug should be fixed in eval.py
Sorry, found the same bug, started the thread and only noticed your post.
Posted by: dagnyt @ July 17, 2018, 7:33 p.m.Thanks very much for your remind! That is the truth. We are quit sorry for that. We will update the code and rerun all the submissions within today. If you are eager to get a right result, you can just change line 26 of eval.py from "submission[key] = ret" to "submission[key] = unique_ret".
Posted by: wider @ July 18, 2018, 2:32 a.m.The bug has been fixed. The evaluation script has been updated and all the submissions have been re-run.
Posted by: wider @ July 18, 2018, 5:50 a.m.