Hi,
In the evaluation, the relative distance is measured between the groundtruth timestamp and the predicted timestamp. However in problems like hard cut shot boundary detection and gradual transition shot boundary detection the groundtruth includes a range of timestamps i.e from start to end of a shot change. So for example if a hard cut is a shot change between two adjacent shots and it occurs at frame X then there is a small window of 3 or 4 frames either side of frame X and If the predicted timestamp from a submission lies either side of frame X but within that range then that would be considered as a correctly detected boundary. Similarly a gradual transition between shots like a fade might take 1 second or 25 frames to complete so between frames X and Y, for example, and if a submitted run predicts a gradual shot transition between frames A and B then provided there is a significant overlap between X..Y and A..B then that is rewarded as being a correct submission.
Since micro events also occur over a range of frames or a time span, so for example an athlete does not instantly go from a walk to a run but this happens within a (small) window of time, have you thought about using such ideas for the evaluation in the challenge and in particular how the evaluation metric could use such windows ?
Posted by: rayush7 @ May 9, 2021, 11:15 p.m.Hi, thanks for your question.
In our GEBD task, a boundary can be either a timestamp or a short range. If it is a range, we represent it by its middle timestamp during evaluation. Thus, our evaluation task is to measure the discrepancy between the detected timestamp and the ground truth timestamp, regardless of their types or semantic meanings. You can refer to our paper for more details: https://arxiv.org/pdf/2101.10511.pdf
Posted by: leiwx52 @ May 10, 2021, 5:16 a.m.