Please, use this "budget" wisely to make the final submission. Very sorry for the issues that it causes for you. If you cannot make your "best bet" submission processed after 10 attempts, write to me personally at spirin2 illinois edu.
Posted by: spirinus @ Oct. 3, 2016, 1:58 p.m.I just want to point out that It is unfair to some participants if you change the rules suddenly. Not everyone get the notice in time . If Ｉknew this message earlier, I could try more new models and pick the best one .
Posted by: ls @ Oct. 4, 2016, 4:11 a.m.@ls, sorry about that. We didn't intend to do that and increased the number of submissions only to make sure that everyone got their submission processed at least once by the platform.
Posted by: spirinus @ Oct. 5, 2016, 1:27 a.m.Just kindly remind you about the issue so the organizers won't make the same mistake in the future competitions. You should try to keep rules unchanged, or change the rules as earlier as possible so that all participants have enough time to response.
Take me for example, I have 3 models, which has about 100+, 500+, 1000+ features. Model 1 give me a F1-score of 0.3795929982 in Phase 1. It's not a satisfying score so that I decide to use model 2 in Phase two. Since I didn't know the potential of new features, I make the single bet with model 2 in Phase 2 . Surprisingly it gave me a F1-score of 0.4155+ in Phase 2, which is a huge progress from model 1. Since I think we have only 3 chances to submit our solution, I stop running model 3 and went on vacation(Oct 1 to Oct 7 is our National Holidays). That's why I feel so unhappy to hear that you increase the submission quota to 10. Since I am away from working-computer, the only thing I can do is to tune the number of instance in submission files, which help me make small progress(from 0.4155 to 0.4166).
@ls thank you for the detailed explanation. While it is not an excuse, please also understand my point of view. Since the system sometimes didn't accept valid submissions due to the instability, we increased the number of allowed submissions in Phase 2 to 10 just to make sure that *at least one* submission is accepted. We didn't expect that the participants would do multiple attempts using different models because it was risky to do so.
Imagine a situation when you have three models: a good model (A), a better model (B), and the model that you believe to be the best (C). A gets accepted from the second attempt, B gets accepted after two more attempts (4/10 total) and unfortunately shows lower results than A in Phase 2, C is stuck after 4 submissions. In this case, you have 8 attempts used and 2 more attempts to try in Phase 2. Now the decision is to resubmit the model A (known to be the best so far for Phase 2), try further to submit C one more time (and keep one more attempt to pick the best in the attempt 10). If, eventually, you cannot submit A or C and get stuck with B knowing that at least A was better, you will be equally unhappy but because of the different reason.
Again, I am very sorry about this situation and will embrace your comment for the future. Sincerely appreciate your time sharing the feedback.
Posted by: spirinus @ Oct. 5, 2016, 5:40 p.m.@spirinus makes a good point.
To substantiate his point. This is what happened on our side. Our team only made 5 good submissions in the end. We had a better model at hand that we wanted to try, but with already several failed/error submissions, it would be too risky to do so and therefore, we choose to not submit anymore.
so you do admit you make 5 good submissions in phrase 2.
Posted by: ls @ Oct. 6, 2016, 5:48 a.m.@ls
10 is allowed. We used our budget wisely. No issue with that.
We are now focusing on our report and cleaning up for an open source release. You should too.
And enjoy the remainder of your vacation. :)
Posted by: ytay017 @ Oct. 6, 2016, 6:43 a.m.sure ：）
the most important thing is that we are learning and improving. at last, thanks all the organizers for your hard working. I enjoy the competition
"the most important thing is that we are learning and improving" @ls totally agree. Great comment!
Posted by: spirinus @ Oct. 6, 2016, 3:34 p.m.