ETCI 2021 Competition on Flood Detection Forum

Go back to competition Back to thread list Post in this thread

> Questions about the test procedure

Organizer, I think you should explain how your test program works to get the final result, the randomly generated mask we submitted is even better than the results we generated from our code!! A regular competition event should be strict with your testing procedures and give instructions,only in this way can the participants have the confidence to persevere.

Posted by: Abel @ June 4, 2021, 9:40 a.m.

I am facing a similar issue. Submitting an untrained model output is giving a higher IOU than a trained one. In CV, the IOU is ~ 0.35. I am wondering what am I missing ? Would request organizers to please clarify

Posted by: anindabitm @ June 17, 2021, 11:32 a.m.

Hey, we sent out an email about it. Sorry in case you missed it. The reason for the low numbers could be multiple:

1. We had updated the evaluation method since some participants uploaded simply 0 arrays that caused the leaderboard to blow up.
2. The model could be underfitting on the test data.
3. Don't worry if we find that this glitch has caused some participants an advantage over others we can create a new test competition with additional dataset to create a level playing field for all.
4. We are simply using an IOU metric calculator. We will also open source the datasets once the competition is over.

I hope it alleviates your concerns.

Posted by: Shubhankar @ June 23, 2021, 8:24 p.m.

Does the organizer fully recognize the ranking of results submitted after May 27? Will the winner be announced directly based on the ranking of submissions after May 27? You seem to be facing an important mistake. Submitting a result of all 1s will still result in the highest score. I regret the setting of your test procedure.

Posted by: Abel @ June 26, 2021, 1:24 a.m.

We had informed about this earlier that the platform doesn't allow the leaderboard to be reset therefore any results currently on the leader board are invalid. We are sorry about the confusion it may have caused. We will have to manually go through the scores and find the top scores. Unfortunately nothing can be done for the leaderboard. We are working on a solution for the lower score issue. We will make an announcement soon.

Posted by: Shubhankar @ June 26, 2021, 1:31 a.m.

Thank you for your efforts, and hope that the new correct test program will be released as soon as possible. Thanks again.

Posted by: Abel @ June 26, 2021, 1:39 a.m.
Post in this thread