Hello organizers,
I've got a few questions regarding the GT and data preparation/submisssion:
1. Among all the annotations for each video, which one is chosen as the GT? Is the f1 score calculated with respect to the rater with the best f1 score among all raters or do we have to take into account all the different annotations?
2. Some videos seem inconsistent with the provided start/end times. For example, video uH4TF9gl21I (bartending) is 50 seconds long on YouTube, which is in conflict with the kinetics400 start/end times, which are 76 and 86 seconds respectively (out of range). Other videos face similar problems, where the trimmed video is shorter than the provided duration. What do we do with those videos? Do we include them in our submission file?
Thank you in advance,
Guillem
Hi Guillem,
Thank you for your interest! Regarding to your questions:
1. You can refer to our evaluation script on Github for details: the detected boundaries for each video will be compared to each rater's annotation and finally the one yields the best f1 score will be taken into further calculation (f1 on the whole dataset). Actually you can either use all the annotations or the one with highest f1 score. There is no limit on this matter.
2. Yes, we have received similar questions recently: https://competitions.codalab.org/forums/27005/5475/. Actually this is an issue from the original Kinetics-400 dataset/annotations. We will improve this and keep you updated. Currently,
2-a. For those start time>original video duration (e.g. uH4TF9gl21I that you mentioned), you can ignore these videos. Also we will remove such videos from our annotation.
2-b. For the trimmed videos which are shorter than 10s, just keep them since our annotators worked on these trimmed videos. It should be fine.
A lot of thanks!
Stan
Thank you for the quick response!
Also, there are some videos in the Kinetics-GEBD Val Set annotations that are not present in the original Kinetics-400 Val Set (for example, video '1438aE9LMgY'). What do we do with them?
Thank you,
Guillem
Hi Guillem,
For these videos, you can try to download them using `youtube-dl` and trim them according to the filename "videoInditifier_start_end.mp4". For the test set, we've verified that all the videos can be downloaded as of March, 2021.
Stan
Posted by: leiwx52 @ April 27, 2021, 1:15 a.m.