Hello,
I have received an email to submit the predictions to the second phase leaderboard. Before, I thought that this is the final scoring dataset, so we won't know what is the score of our submission before the end of the competition and the scores and leaderboard will be released after October 1st. In the email I can find this sentence: 'you could submit and test your predictions and get a score in 'Result' in 'final scoring phase sleep' and 'final scoring phase motor imagery'.'.
1. Does it mean we can get the scores immediately and we can try different solutions on this dataset to select the optimal one?
2. If yes, are the results from the 'final scoring phase sleep' and 'final scoring phase motor imagery' leaderboards the final results of the competition or the solutions will be tested by organizers on an additional dataset? Or maybe leaderboard 2nd phase scores are only partial results and the full results will be used for final competition scoring? Unfortunately, if we can test our solutions on final scores for two weeks it may lead to overfitting to the 2nd phase dataset.
Posted by: ms01 @ Sept. 16, 2021, 7:41 p.m.Thanks for the question.
1. Does it mean we can get the scores immediately and we can try different solutions on this dataset to select the optimal one?
-Yes
2. If yes, are the results from the 'final scoring phase sleep' and 'final scoring phase motor imagery' leaderboards the final results of the competition or the solutions will be tested by organizers on an additional dataset? Or maybe leaderboard 2nd phase scores are only partial results and the full results will be used for final competition scoring? Unfortunately, if we can test our solutions on final scores for two weeks it may lead to overfitting to the 2nd phase dataset.
- Evaluation will be based on the score of the two tasks in phase2 according to 'evaluation' in https://beetl.ai/code. We would like to make the result transparent and give researchers who developed their algorithms for 2-3months a bit of chance to get a feel of how it works for a short period of time instead of giving a result 'you lose'. We also considered the 'overfitting' problem during competition design. That's why phases 1&2 are separated, and phase1 is 2month+ and phase 2 is 2weeks. Also, submission time is limited to 4 times a day, and that's also why there are two data sets with different sampling rates/channels sizes. While I think your consideration is reasonable and I've reduced the test chance to twice a day.
Xiaoxi Wei
Posted by: BEETLCompetition @ Sept. 17, 2021, 2:15 a.m.