I have found some training data is confusing, such as video signer13_sample194_color, signer8_sample2369_color, signer23_sample161_color and signer28_sample161_color, both of them belong to class 42, but the action is totally different. Besides, the video signer42_sample669_color, signer41_sample79_color, signer13_sample1616_color and signer15_sample1_color, they belong to class 164, while they have a subtle difference, hand in former two videos will be opened finally while be closed in last two video
So I wonder it's just mislabeled or they belong to same action?
The labels are correct.
Since the signers are performed by different users, there may be slight differences (as you stated in those belonging to class 164). For the samples that belong to class 42, signer8 is left-handed and others are right-handed. Also, there are differences in hand positions.