We have a few questions regarding the competition:
1) Can we use EEG statistics (e.g. Unsupervised learning) from the test sets (no labels)? For example, get the average covariance matrix on the test set for the purpose of recentering the test subject's distribution?
2) Does the Cybathlon data include few seconds of pre-trial/post-trial EEG?
3) Do we have to use all MI datasets in the second task?
4) Are the Cybathlon dataset electrodes the same 32 electrodes as in MI leaderboard dataset B?
5) Can we use leaderboard data (unlabelled, labelled) during final tests?
6) Is the scoring metric on each task accuracy? Does the score chart on codalab show the total score?
Thanks for your help in advance.
Posted by: StylianosBakas @ July 30, 2021, 9:51 a.m.Thanks for the question:
1) Can we use EEG statistics (e.g. Unsupervised learning) from the test sets (no labels)? For example, get the average covariance matrix on the test set for the purpose of recentering the test subject's distribution?
- yes, you can use the test subject's statistics.
2) Does the Cybathlon data include few seconds of pre-trial/post-trial EEG?
- no we don't have that, the final test set will be similar to the leaderboard test.
3) Do we have to use all MI datasets in the second task?
- you can choose the one you need.
4) Are the Cybathlon dataset electrodes the same 32 electrodes as in MI leaderboard dataset B?
- the final test set will be similar to the leaderboard test, but details will be provided later
5) Can we use leaderboard data (unlabelled, labelled) during final tests?
- we won't provide the true label of the leaderboard data. During the final test, you can use leaderboard data as additional unsupervised sources.
6) Is the scoring metric on each task accuracy? Does the score chart on codalab show the total score?
The shown in the table score of each task is only the score of that task, but not an average of the two. The chart of Codalab is the highest score of a day (both task included, but not average), it's automatically defined by Codalab by default.
Thank you for the quick and insightful answers!
Posted by: StylianosBakas @ July 30, 2021, 10:20 a.m.Is the named leaderboard results table inconsistent with the highest submission score chart? Highest score in the chart is 67.5, but in the leaderboard tables it is 62.0.
Posted by: andacdemir @ Aug. 6, 2021, 3:04 p.m.The score shown in the leaderboard table of each task is only the score of that task. The chart of Codalab is the highest score of a day (and participants may not yet upload that score to the leaderboard table yet, teams need to choose and click which result they have to show in the table). We highly recommend participants update the highest score to the leaderboard when they have new submissions;) In the later stage of the competition, we may also change the backend function to force to do so.
Best,
Xiaoxi Wei