Hand Gesture Recognition For Human-Machine Interaction: Elena Sánchez-Nielsen Luis Antón-Canalís Mario Hernández-Tejera
Hand Gesture Recognition For Human-Machine Interaction: Elena Sánchez-Nielsen Luis Antón-Canalís Mario Hernández-Tejera
Interaction
Elena Sánchez-Nielsen Luis Antón-Canalís Mario Hernández-Tejera
Department of Statistic, Department of Statistic, Institute of Intelligent Systems and
O.R. and Computer Science, O.R. and Computer Science Numerical Applications in
University of La Laguna University of La Laguna Engineering.
Edificio de Física y Matemáticas Edificio de Física y Matemáticas Campus Universitario de Tafira
38271, La Laguna, Spain 38271, La Laguna, Spain 35017, Las Palmas, Spain
[email protected] [email protected] [email protected]
ABSTRACT
Even after more than two decades of input devices development, many people still find the interaction with
computers an uncomfortable experience. Efforts should be made to adapt computers to our natural means of
communication: speech and body language. The PUI paradigm has emerged as a post-WIMP interface paradigm
in order to cover these preferences. The aim of this paper is the proposal of a real time vision system for its
application within visual interaction environments through hand gesture recognition, using general-purpose
hardware and low cost sensors, like a simple personal computer and an USB web cam, so any user could make
use of it in his office or home. The basis of our approach is a fast segmentation process to obtain the moving
hand from the whole image, which is able to deal with a large number of hand shapes against different
backgrounds and lighting conditions, and a recognition process that identifies the hand posture from the
temporal sequence of segmented hands. The most important part of the recognition process is a robust shape
comparison carried out through a Hausdorff distance approach, which operates on edge maps. The use of a
visual memory allows the system to handle variations within a gesture and speed up the recognition process
through the storage of different variables related to each gesture. This paper includes experimental evaluations
of the recognition process of 26 hand postures and it discusses the results. Experiments show that the system can
achieve a 90% recognition average rate and is suitable for real-time applications.
Keywords
Man-Machine Interaction, Perceptual user interface, Image Processing, Hand gesture recognition, Hausdorff
distance
previous tracking hand motion [San99] or they can Resize Hand Posture to Hand VMS
be generated during the recognition process. Compare Hand Posture with VMS
Execution Action Process
This paper is organized as follows: Section 2 Execute Action Resize Hand VMS to Hand Posture
introduces an overview of system components. Hand Compare VMS with Hand Posture
represents the jth user hand posture edge map fashion that hk (S (U 1j n ), Pmq
1n
) , replacing U 1jn by Pmq1n
computed from the user hand posture detection
1n denotes each
module (described in section 3) and PMq and Pmq
1n by
U 1j n . In each comparison stage, it is only
one of the M stored patterns edge map in VMS. necessary to compute the distance transform image
of U 1j n , since the one of Pmq
1n has previously been
With the aim of considering the different changes in
appearance and non-rigid distortions of user hand calculated and stored in VMS.
postures regarding the stored patterns, a resize Figure 6(a) shows a detail of the comparison process,
operation is computed. Firstly, U 1j n is scaled to the corresponding to the computation of hk (U 1j n , Pmq
1n
),
size of the qth sample, which represents a stored where the user hand posture is compared with the
pattern Pmq
1n in VMS and secondly, the specific distance transform image of one of the stored pattern
pattern in VMS, Pmq 1n is scaled to the size of in VMS. It should be noticed that, if the postures that
U 1j n .
are compared are similar, the size change does not
Figure 5 illustrates this process in the steps 1 and 4. affect gravely the shape of the posture, like observed
We denote both scaled operations respectively in 6(a). If they are different, on the other hand, an
S (U 1j n ) and S (Pmq )
1n .
image deformation can take place like observed in
6(b).
In order to accelerate the computation of the
bidirectional partial Hausdorff distance and make it
suitable for real-time recognition, a pre-processing
5. EXPERIMENTS
The hand posture detection and recognition approach
stage for each one of the stored pattern in VMS is
was implemented in Borland Delphi on an AMD K7
calculated. The pre-processing step is based on the
700Mhz PC running Windows XP with an USB
storage of the edge map, the distance transform and
Logitech Quickcam webcam. The recognition
the morphologic information in the initialization
approach has been tested with real world images as
process of the system (described in section 2).
the input mechanism of a user interface, surveillance
The distance transform [Pag92] for each edge map system, autonomous robot vision system or virtual
posture image generates an image where background environment applications. All tests were executed on
pixels have been labelled with their Manhattan 128x96 images with 24b colour depth, which were
distance to the closest object pixel. sampled at a rate of 25 frames/sec.
The system was tested under two different light The average processing time per second on an AMD
conditions: natural daylight and artificial light, in K7 700Mhz PC system is 8 frames. Being impossible
order to test its adaptability to different lighting for a human to make 25 postures in a second time, it
conditions. Also, two different users were is feasible the analysis on just 8 frames. It is noticed
considered. The first one, named “original subject” that this feature is needed to be improved; a tracking
was the one who created the visual memory. The system should be implemented in order to avoid the
second one tested the application using the same complete analysis of each frame. Anyway, the use of
visual memory created by the original subject. The a faster PC system than the currently used would
gesture vocabulary in our gesture interface is assure a real time processing, which allows the use of
composed by 26 postures, shown in Figure 7, each of the proposed approach in real-time video
which represent a gesture command mode. It can be applications.
observed that some postures are really similar, like
postures pair 1-20, 2-21, 5-25, 18-19, 21-22.
11%
7. ACKNOWLEDGMENTS
3%
This work has partially been supported by the Canary
Autonomous Government under Project PI2000/042.
8. REFERENCES
86%
[Bar98] Barnabas Takacs. Comparing Face Images
using the modified Hausdorff distance. Pattern
Right Discarded W rong Recognition, Vol 31, nº 12, pp. 1873-1881, 1998.
[Bob95] Bobick, A. And Wilson, A. A state-based
Figure 9. Recognition Rates: second subject,
technique for the summarization and recognition
artificial light.
of gesture. In Proc. IEEE Fifth Int. Conf. on
Computer Vision, Cambridge, pp. 382-388, 1995.
[Dam97]A. Van Dam. Post-WIMP user interfaces.
Comunications of the ACM, VOL. 40, nº2, pp.
63-67, February 1997.
[Dav94] Davis, J. and Shah, M. Visual gesture
recognition. IEE Proc. Vis. Image Signal Process,
141(2) :101-106, 1994.
[Her99]M. Hernández, J. Cabrera, M. Castrillón, A. Machine Intelligence, Vol.23, Nº. 12, December
Domínguez, C. Guerra, D. Hernández, J. Isern. 2001.
DESEO: An Active Vision System for Detection, [Turk98] M. Turk, (ed.). Proceedings of the
Tracking and Recognition. Volume 1542 of Workshop on Perceptual User Interfaces, San
Lecture Notes in Computer Science, 1999. Francisco, CA, November 1998.
[Hun95]. E. Hunter, J. Schlenzig, and R. Jain. [Wil95]William T. Freeman, Craig D. Weissman.
Posture Estimation in Reduced-Model Gesture Television Control by Hand gestures. IEEE Intl.
Input Systems. Proc. Int´l Workshop Automatic Workshop on Automatic Face and Gesture
Face and Gesture Recognition, pp. 296-301, 1995. Recognition, Zurich, June, 1995.
[Jai95] R. Jain, R. Kasturi, B. Schunck. Machine
Vision. McGraw-Hill Editions, 1995.
[Jon98] M. Jones and J. M. Rehg. Statistical Colour
Models with Application to Skin Detection.
Technical Report Series, Cambridge Research
Laboratory, Dec, 1998.
[Ju97] Ju, S., Black, M., Minneman, S., Kimber, D.:
Analysis of Gesture and Action in Technical Talks
for Video Indexing. IEEE Conf. on Computer
Vision and Pattern Recognition, CVPR97.
[Mag95] C. Maggioni. Gesturecomputer. New Ways
of Operating a Computer. Proc. Int´l Workshop
Automatic Face and Gesture Recognition, 1995.
[Ruc96] Ruckelidge, W. Efficient Visual
Recognition Using the Hausdorff Distance,
volume 1173 of Lecture notes in computer
science, Springer-Verlag, 1996.
[Pag92] Paglieroni, D.W. Distances Transforms.
Computer Vision, Graphics and Image Processing:
Graphical Models and Image Processing, 54 :56-
74, 1992
[Pen98]Pentland, A. Smart Rooms: Machine
Understanding of Human Behavior. Computer
Vision for Human-Machine Interaction, eds.
Cambridge University Press, pp. 3-21, 1998.
[San01] Sánchez-Nielsen Elena, Lorenzo-Navarro
Javier, Hernández-Tejera F. Mario. Increasing
Efficiency of Hausdorff Approach for Tracking
Real Scenes with Complex Enviroments. 11th
IEEE International Conference on Image analysis
and processing, pp. 131-136, 2001.
[San99] Sánchez-Nielsen Elena, Hernández-Tejera F.
Mario. Hand Tracking using the Hausdorff
Distance. Proceedings of the IV Ibero American
Symposium on Pattern Recognition, 1999.
[Tri01] J. Triesch and C. von der Malsburg. A
System for Person-Independent Hand Posture
Recognition against Complex Backgrounds.
IEEE, 2001 Transactions on Pattern Analysis and