Thank you so much for registering for the MOCHA - ICMI Grand Challenge. The final phase will arrive in a few hours and this message contains important information in this regard.
The system will be open for submissions from July 5th midnight UTC until July 11th (midnight UTC). You can start downloading test videos now, please go to the challenge repository here:
To download the code for downloading test videos you should use the following file: test.zip. This is script will automatically download training and test videos and create data partitions (you may remove the *.csv files associated with training data if you just want to download the test samples). For the final phase you are expected to provide predictions for every video in the test data folder. There are 237 test samples and you should provide a single file with predictions for each of these videos (one row per test sample, see below). Please use the same order for videos as in the “Dataset_ComicMischief_Test_Scenes.csv” file.
* Please note that for this code you should install “pytube==10.8.5”, see the readme file, instead of the pytube version used in the previous code
* It may be possible that a couple of training videos are not available for downloading, but we have checked that all of the test videos are available. Please make sure no test sample is missing.
* Submissions: please provide predictions for the samples in the “Dataset_ComicMischief_Test_Scenes.csv” file, one row per sample and following the same order as in this *.csv file. For the binary task, a single value is required, for the multilabel task please 4 predictions are expected per sample (please use tabs to separate the values in each row). Once the predictions file is ready, please create a zip file containing the prediction file. No other file or folders should be included in the *.zip file. We are including sample submissions for both tasks in the test phase code for your reference.
* You will be able to upload up to 5 submissions to evaluate different settings/variants of your systems. Please note that the last valid submission will be the one that will be considered for the leaderboard.
* The leaderboard may remain hidden until the end of the final phase, so you will not know your scores immediately, but you will be informed if the submission was successfully evaluated.
Looking forward to receiving your submissions!
The organizing team.