What does SampleFrames means?

[nixczhou]

What does SampleFrames actually mean in the code? I see the parameters are clip_len, frame_interval, and num_clips. Can anybody explain it to me? I am a newbie here :)

[innerlee]

@dreamerlin

[innerlee]

Does the doc offer any help? https://mmaction2.readthedocs.io/en/latest/api.html#mmaction.datasets.pipelines.SampleFrames

[irvingzhang0512]

SampleFrames defines sample strategy for input frames. Sample strategy is defined as clip_len x frame_interval x num_clips. There are two sample strategy, uniform sampling and dense sampling.

Dense strategy samples continuous frames. To be more specific, samples one frame every frame_interval frames and finally get clip_len frames.
Uniform strategy(clip_len == 1) devides all frames into num_clips parts and samples one frame from each part.

If there are 96 frames for one sample, frame id in range [1, 96].
Dense strategy demo: 32x2x1 may select no.1, 3, 5, ..., 63 frames.
Uniform strategy demo:1x1x8 divides 100 frames into [1, 12], [13, 24], [25, 36], [37, 48], ..., [85, 96] and may select no.1, 13, 25, ..., 85 frames.

More details could refer to source code

[nixczhou]

Actually, I am using R2+1D for detecting whether a person successfully catches a tennis ball or not. So, do you mean if I follow the recommended strategies for sample frames is better?

[irvingzhang0512]

Actually, I am using R2+1D for detecting whether a person successfully catches a tennis ball or not. So, do you mean if I follow the recommended strategies for sample frames is better?

It's hard to say which model and which strategy is the best. You'd better try everything and find yourself. Recommended strategies have kinetics-400 pretrained mdoels, which will help you get better results. And tsm is my favourite

[nixczhou]

Thank you for the reply, will try it then, can i know the reason why do you prefer tsm?

Regarding kinetics pretrained model, is there any tutorial in this API?

[irvingzhang0512]

TSM is faster than 3d models and accuracy is good enough for me.

You could refer to model zoo and Tutorial 2: Finetuning Models

[nixczhou]

Great, thank you !

[stephen-nju]

it may like sliding window on video frame sequence,the windows size is clip_len*fram_interval,the start index of the windows can computed by num_clips.

[JasseurHadded1]

Dense strategy demo: 32x2x1 may select no.1, 3, 5, ..., 63 frames

Does that mean if I have a video with 200 frames for example and I use clip_len=32, frame_interval=2, num_clips=1 then we discard all frames from 65 to 200?