Hand Detection Optimization

[nskaiser]

Hi Vlad, are there any configs or ideas you can share on how we can make the hand detection performance on par w/ MediaPipe Hands? If you try their demo, https://codepen.io/mediapipe/pen/RwGWYJw, you'll see the hand detection is near-instant and tracks well. With my application (and your demos) it seems to not detect the hands quite often. Any ideas would be appreciated, thanks!

[vladmandic]

which configuration settings did you try so far? there is plenty that can be done via configuration to increase priority of hand processing and disable other models

for example, check out live demo at https://github.com/vladmandic/human-motion and select hand model and wireframe output and select webcam as input

but when comparing with mediapipe, can you look at their JS webgl solution instead?
the link youve posted uses their custom wasm binaries for processing which are fully proprietary
(which goes against open architecture i'm using)

[nskaiser]

Thanks Vlad. I'll try to find a demo of the WebGL hand solution. My application requires quite a few models (blazeface, facemesh, iris, faceres, emotion, handtrack, handlandmark, movenet-lightning) and we use the humangl backend without caching (each frame is important for our functionality).

I tried your human-motion demo above with just the hand model and wireframe output. The problem I run into is often when the hand is on the face (ie holding your head up w/ your hand). The hand is usually not detected in that case, likely it doesn't see enough separation in fingers?

[vladmandic]

without caching (each frame is important for our functionality).

In that case, its always going to be slower than MediaPipe as they perform extensive detection caching.
Basically, even if you don't want to cache actual results, you still want to avoid rerunning detector on each frame and use calculated bounding box from previous frame instead (as hand box likely doesn't move that much).

The hand is usually not detected in that case, likely it doesn't see enough separation in fingers?

Its the fact that detector cannot distinguish between face and hand when there is a large overlap so hand basically falls under bounding box of a face. I could force analysis of face as hand and then throw away if low confidence, but that would slow overall processing quite a lot.