Real-time camera-based 3D hand tracking

Hand tracking has applications in many fields, for example for navigation in virtual environments, virtual prototyping, gesture recognition, and motion capture. The goal of this project is to track the global position and all finger joint angles of a human hand in real-time.

Due to measurement noise, occlusion, cluttered background, inappropriate illumination, high dimensionality, and real-time constraints, hand-tracking is a scientific challenge.

We use multiple cameras to capture images of the hand from different directions. Features like skin segmentation, edge detection, skin texture, and previous hand position can be used to extract the 2D shapes of the hand in the images. We utilize dimension reduction techniques to cope with the high complexity of the tracking problem (the hand has about 21 local DOFs and 6 global DOFs).

The following figure shows the overall architecture of our system.

hand-tracking


Poster

This poster illustrates the main steps of the tracking algorithm.

Publications

Position Papers



Results

Segmentation-Free, Area-Based Articulated Object Tracking

Videos

video
DivX, WMV9

The video shows a visual comparison of a conventional skin segmentation-based similarity computation (left column) with our novel divergence-based (right row). Both similarity measures are combined with a brute force (upper row) and our new smart mode finding approach (bottom row) to detect the best matching position and hand pose. The resuls show that the divergence-based approach outperforms the conventional approach in situations where the skin segmentation does not work well.

FAST: Fast Adaptive Silhouette Area based Template Matching

Videos

video
video
video
MS-Video-1, MOV DivX, WMV9, MOV DivX, WMV9, MOV

The left most video demonstrates the rectangle covering computation algorithm. The middle and right videos demonstrate the matching approach. Of most interest is the bottom left panel that shows the rectangular representation of the templates. We would like to point out that, at matching time, we can adaptively choose the matching accuracy by simply adjusting the number of rectangles per template used to compute the similarity.

Silhouette Area Based Similarity Measure for Template Matching in Constant Time

Videos

video
DivX, WMV9, MOV

The video demonstrates the main steps of our area based similarity measure. The skin segmentation is applied to the input image. As result, we get the skin likelihood map shown on the top right panel. The templates are represented by a very compact data structure consisting of a set of rectangles (bottom left panel). We compute a similarity measure based on the per-pixel joint probability. Utilizing the integral image of the log-likelihood, we are able to compute the similarity measure more then 10 times faster compared to state-of-the-art approaches.

Continuous Edge Gradient-Based Template Matching

Videos

Original image Best matching template determined by our approach superimposed at the hand labeled position
Combined confidence map generated by the chamfer based approach Combined confidence map generated by our approach
video
DivX, WMV9
video
DivX, WMV9
video
DivX, WMV9
The videos demonstrate that our approach generates fewer and much more significant maxima (possible hand positions), which leads to considerably easier true hand position finding.
Note:

The videos demonstrate that our approach generates fewer and much more significant maxima (possible hand positions), which leads to considerably easier true hand position finding.

Skin Segmentation

Original Image
Jones and Rehg
Our Approach
result1o.png
result1o.png
result1s.png
result2o.png
result2o.png
result2s.png
result3o.png
result3o.png
result3s.png
result4o.png
result4o.png
result4s.png
result5o.png
result5o.png
result5s.png

Video

The Video below shows the segmentation algorithm. The Camera is positioned on the rear right side. On the screen the left window shows the captured image from the camera, the right window the segmentation result of our algorithm.

video
DivX, WMV9, MOV