Graphics, Vision and Image Processing Journal, ISSN 1687-398X, Volume 17, Issue 1, ICGST LLC, Delaware, USA,

June 2017

Robust Mouse Control based on Dynamic Template Matching of Hand Gestures

R. Arcangel, S. Capuso, M. Gerbuyos, C. Laviña, G. Mabuti, R. Tolentino

Electronics Engineering, Polytechnic University of the Philippines – Santa Rosa Campus
[email protected], [email protected], [email protected],
[email protected], [email protected], [email protected],
http://www.pup.edu.ph

Abstract RGB colours is used to perform mouse functions. Another

This paper introduces the application of the Template paper entitled “Simulation of Real-time Hand Gesture
Matching algorithm in recognition dynamic hand gestures. Recognition for Physically Impaired” [9] uses Principal
Images undergo thresholding to convert each pixel of Component Analysis (PCA) by combining hand gesture
image into binary which will eventually be filtered to recognition based on centroids with real-time 3D hand
reduce the noise present. To make consistent and precise tracking, which achieved a recognition rate of 87.75%. A
detection, this image can still be improved by enhancing recent research for virtual mouse using static hand gesture
the edge of the hand, which serves as the initial input for recognition entitled “Cursor Control Using Hand Gestures
the process of template matching. The saved templates and a Webcam” [10] that received a recognition rate of
serve as the basis for comparison in Template Matching 58.8%%. It used convexity defects of hand contour and
algorithm. The processed sequence of images will be analysed the fingertips and the angle of the fingers in order
compared to the templates saved in the database. When to perform the mouse functions.
the current templates have significant relation or no
significant relation to the saved database, the result will Based on the previous studies that used static gesture
turn to the recognition of the mouse function. Hence, the recognition, the tremor of the hand has become more
mouse function will be executed. noticeable when hanged in the air. Thus, due to the low
precision detection of gesture caused by the instability of
Keywords: template matching, dynamic hand gesture, the hand, misdetection and inaccuracy of recognition had
virtual mouse been the results.

Nomenclature The authors intend to provide a system that can recognize

PCA Principal Component Analysis the dynamic hand gestures as mouse control. It integrates
RGB Red, Green and Blue the use of dynamic hand gestures to eliminate the effect of
YCBCR Luminance; Chroma Blue; Chroma Red tremor of the hand in recognizing the gesture. Embedded
3D Three-dimensional in the system are the templates saved in the database.
These frames are composed of pictures which are captured
1. Introduction from the starting position of the hand until the last position
The evolution of computer mouse has become a vigorous of the hand while performing a certain mouse function,
trend as technology develops rapidly. Yet, there have been served as the pilot copy in the database. The system
many transformations of a computer mouse since the start integrates the method of template matching to increase the
of its invention, from a trackball to an optical mouse. accuracy of the hand gesture recognition. As the camera
Today’s generation has become more involved in creating detects a hand gesture, the captured input data will be
devices that can replace mechanically operated processed through thresholding.
technologies. One of the technologies that substituted the
traditional computer mouse is called virtual mouse. This Reduction of noise is also necessary by implementing
technology enables the user to navigate the window and image filtering and edge sharpening for the enhancement
execute functions of a mouse by only executing actions of the edges of the hand. Template matching of the saved
like static hand gestures. templates and the current detected templates is the next
A research study entitled “Gesture Recognition Based process of the system. From the frames of the templates,
Mouse Events” [13] that achieved 87.5% rate of the current template will be matched to the saved
recognition, involves the use of YCbCr color model. In templates. If there is relation in the matching process, the
this method, the dependency on light intensity has been gesture would be recognized. Thus, the mouse control with
eliminated. To track the gesture positions, selection of the respect to the recognized gesture is performed.

9
Graphics, Vision and Image Processing Journal, ISSN 1687-398X, Volume 17, Issue 1, ICGST LLC, Delaware, USA, June 2017

The remained of the paper is organized as follows: Section

(2) discusses the theory about Template Matching
algorithm, how it works and how it is applied in the system
to recognize dynamic hand gesture; Section (3) presents
the results and discussion; and Section (4) presents the
conclusion where the summary of the contribution of this
work and identify areas for further work is presented.

2. Theory
The system for recognizing the dynamic hand gesture is
equipped with the use of computer with 2.3 GHz 8GB
RAM, administering Windows 7. This is made possible
through the use of Microsoft Visual Studio 2012, which
has embedded Aforge and Accord framework for C# .NET
language, with MS Visual Studio form environment. The
assessment of the result depends on the performance of the
system. The set up for the distance of the hand and the
camera is approximately 1 meter, depending on the
external interferences.

A. Recognizing dynamic hand gesture using Template

Matching algorithm for controlling a virtual mouse
Figure 1 illustrates how the system works step by step. It
starts from how the camera captures the movement until Figure 1. Conceptual Framework
the recognition of the dynamic hand gesture. After the
camera detects the gesture, the captured input data would
be processed through the method of segmentation to
identify the correct contour.

For the reduction of noise, filtering is necessary to have a

clear image of the hand through smoothening, sharpening
and edge enhancement. Once the data is easily
distinguished, the process for gesture recognition
proceeds. Template matching of the saved templates and Figure 2. Hand Detection for HSV Color Model
the current detected templates is the next process of the
system. Hand Detection: This includes processes of acquiring data
from the Template Matching algorithm. The raw data of
Subsequently, this study involves the use of dynamic hand the system are the frames or postures of the hand while
gestures; the data should include sequence of frames performing a certain mouse function. The skin detection
according to the posture of the hand. The saved templates method is used to determine the hand area. The R, G and
would be used as the reference for comparison to the B pixels are used to find the skin regions that includes a
acquired data. This system integrated a program for mouse wide range of colors such as red, pink, brown and orange.
events. When there is correct matching, the system will Training samples of skin region is done by saving a sample
automatically perform the mouse function. color. Once the hand is identified through skin detection,
image segmentation is the next process to be done. It is
applied to separate the arm region from the hand region.
From the pre-processing segment of the system, the user’s When the threshold process takes place, which involves
way of moving his or her hand allows the system to either replacing each pixel of an image into black pixel where the
easily recognized the gesture or does not recognize the image intensity is lower than some set constant or a white
gesture performed. The detection of the system depends pixel if it is higher than that set constant, the next processes
on how the environment affects the capturing of the hand, are the noise reduction and sharpening of the hand to
and how the user executes certain mouse function which enhance the edges.
also determines that the hand of the user must be of the
same orientation as the saved templates. Moreover, this Template Matching Algorithm: This process involves
paper involves the application of the Template Matching comparing and classifying the acquired data of the hand
algorithm in recognizing the dynamic hand gestures to gesture to the saved templates. Since Template Matching
make this system be more accurate in recognizing moving is the involved process in performing the dynamic hand
gestures. For the result of the system, the current templates gesture, each gesture has its respective set of frames that
have significant relation or no significant relation to the are dependent on how the gesture is being done.
saved database.

10
Graphics, Vision and Image Processing Journal, ISSN 1687-398X, Volume 17, Issue 1, ICGST LLC, Delaware, USA, June 2017

The concept of Template Matching is to provide a which the system would compare the current templates of
reference image and an image to be compared to the saved each dynamic hand gesture to the saved templates.
templates during the Template Matching process. The Gesture Recognition: To compare current templates to the
designated dynamic hand gesture will have certain time for saved templates, posture and positioning of the fingers
training and saving the templates. The saved templates should be properly observed. Left click function is
after the training will have 4 images that comprise the executed by bending the index finger, just like how a
template. mechanical mouse is used. For right click, the middle
finger should be slightly bent also. For the scroll up and
down, the templates are illustrated. In drag and drop
functions, index finger is also used.

B. Evaluating the system’s performance in terms of

accuracy and identifying the significant
improvement of the current study in comparison to
the previous studies
This paper focuses on how to increase the accuracy rate of
Figure 3. Hardware of the System
gesture recognition for mouse control and its significant
improvement from the previous study. The scheme for
assessing the system’s performance is that the accuracy of
the hand based on the actual dynamic gesture and image
template is checked by performing the mouse functions so
that it can be proved that the instability of the hand will
not affect the performance in terms of accuracy of the
mouse functions since the gesture is based on dynamic
movements. The method for evaluating the accuracy is to
perform 30 trials for each mouse function.
To compute for the accuracy rate of the system,
recognition rate formula is involved. Number of
recognized gestures will be provided from the experiment
conducted by the proponents. To know the error rate of the
Figure 4. Software Applied in the System system, number of unrecognized gestures over the total
number of trials is the formula to be used.

100

(1)

100

(2)

This study provides a system that can recognize a certain

dynamic hand gesture that has corresponding mouse
function. It involves six mouse functions such as left click,
right click, double click, scroll up, scroll down and drag
and drop. The accuracy rate is the implicated parameter
which needs to be improved by the current study. The
Figure 5. Template for Clicking
result of the current study will be compared to the data of
Figure 6. Template for Scroll
Functions: Left click (left) and Functions: Scroll-up (left) the previous study provided below and it will be used to
Right Click (right) and Scroll-down (right) identify the significant improvement of the current study
from the previous studies.
The user will execute a dynamic hand gesture that has
equivalent mouse function. The data will start from the
frames or posture of the hand from the beginning up to the Table 1. Result from the Study Entitled “Gesture
final position of the hand. The saved frames for each Recognition Based Mouse Events”
template will be compared to the current set of images. Mouse Functions Detection Rate
Correct recognition of the hand can only be possible if the Right Click 92%
pre-processes for hand detection are achieved correctly. Left Click 87%
When the templates to be compared are turned into binary Double Click 87%
(white and black), the process for template matching is Drag and Drop 84%
also initialized. The user should conduct an experiment in

11
Graphics, Vision and Image Processing Journal, ISSN 1687-398X, Volume 17, Issue 1, ICGST LLC, Delaware, USA, June 2017

Table 2. Result from the Study Entitled “Simulation of Table 5 indicates the mismatched templates and
Real-time Hand Gesture Recognition for Physically unrecognized gestures from the gathered data. Due to the
Impaired” variation of finger positions in the frames extracted from
Mouse Functions Detection Rate the real-time video, it lead to erroneous detection of the
Right Click 88% hand features that caused the system to misinterpret the
Left Click 91% gesture being performed by the user. The last position of
Scroll Down 87% the gesture performed is incorrectly recognized shown in
Scroll Up 85% Table 5, since double click function is set as two
consecutive left click function like a mechanical mouse,
Table 3. Result from the Study Entitled “Cursor Control
the user is likely to perform the double click function
System Using Hand Gesture Recognition”
immediately without a pause that lead to recognition of
Mouse Functions Basis incorrect gestures. The function drag and drop is not
Right Click 13/30 properly recognized due to the improper orientation of the
Left Click 11/30 hand in the last position of the frames extracted from the
Double Click 16/30 real-time video.
Scroll Down 3/30
Scroll Up 19/30 Table 6. Accuracy Rate and Error Rate of the System
Mouse Functions Accuracy Rate Error Rate
3. Results and Discussion Left Click 100% 0%
The study aims to provide a system that can achieve a high Right Click 93.3333% 6.6667%
accuracy rate in terms of recognizing the dynamic hand Double Click 83.3333% 16.6667%
gesture for each mouse movement. The authors conducted Scroll Up 100% 0%
the gathering of data by allowing 30 trials for each mouse Scroll Down 100% 0%
function to be executed by the user. Drag and Drop 93.3333% 6.6667%

Table 4. Result of the Study Table 6 indicates the accuracy rate and error rate of the
Mouse Functions Recognized Missed system’s performance. The accuracy rate is computed
Left Click 30/30 0/30 using the formula for the recognition rate. Recognition rate
Right Click 28/30 2/30 is just the ratio of success recognitions to the total number
Double Click 25/30 5/30 of trials. The system’s performance in terms of accuracy
Scroll Up 30/30 0/30 for the left click, scroll up and scroll down is 100%. There
Scroll Down 30/30 0/30 is 93.3333% for the mouse functions right click, and drag-
Drag and Drop 28/30 2/30 and-drop. For the double click function, there is 83.3333%
of recognition rate.
The tables below show the significant improvement of
Some of the samples exhibiting recognition errors or
the current study from the previous studies.
misclassified errors are shown in Table 4. In the
experiment that was conducted by the proponents, the
mismatched mouse functions are as follows: right click,
Table 7. Comparison of Recognition Rate of the Current
double click, and drag and drop.
Study from the Study Entitled “Gesture Recognition
Based Mouse Events”
Table 5. Mismatched and Unrecognized Gestures
Mouse Functions Current Study Previous Study
Right Click
Left Click 100% 87%
Right Click 93.3333% 92%
Saved
Template Double Click 83.3333% 87%
Drag and Drop 93.3333% 84%
Overall
92.5% 87.5%
Mismatched Recognition Rate
Actual
Performance

Table 8. Comparison of Recognition Rate of the Current

Double Click
Study from the Study Entitled “Simulation of Real-time
Unrecognized
Hand Gesture Recognition for Physically Impaired”
Actual Mouse Functions Current Study Previous Study
Gesture Left Click 100% 91%
Right Click 93.3333% 88%
Drag and Drop Scroll Up 100% 85%
Scroll Down 100% 87%
Unrecognized
Overall
Actual 98.3333% 87.75%
Gesture Recognition Rate

12
Graphics, Vision and Image Processing Journal, ISSN 1687-398X, Volume 17, Issue 1, ICGST LLC, Delaware, USA, June 2017

Table 9. Comparison of Recognition Rate of the Current matching will be affected. This detection is affected by the
Study from the Study Entitled “Cursor Control System lighting condition of the place where it is performed. The
using Hand Gesture Recognition” proponents are able to arrive with a system that will
Mouse Functions Current Study Previous Study manage proper recognition of dynamic gestures that are
Left Click 100% 63% the same in controlling a commercialized mouse.
Right Click 93.3333% 57% The authors conclude that the current system surpassed the
Double Click 83.3333% 47% system’s performance of the previous studies. This study
Scroll Up 100% 37% also proves that the problem of inaccuracy due to tremor
of the hand has been eliminated by integrating Template
Scroll Down 100% 90%
Matching algorithm in dynamic hand gestures.
Overall
95.33332% 58.8%
Recognition Rate
5. Acknowledgements
In identifying the system’s performance in terms of its The students involved in making this project would like to
accuracy, the authors executed 30 trials for each mouse express their sincere gratitude and appreciation to all
function which is based on the reference journal. The persons who had extended their help and support and who,
current system is able to identify six (6) mouse functions, in one way or another, had enabled them to bring this work
while the previous system has 5 mouse functions. This to successful completion.
study has recognition rate of 83 to 100 percent for each To their family and friends who willingly give their love,
mouse function. This numerical value signifies that there encouragement, patience, consideration, inspiration and
is only low number of missed recognition for each mouse truthful understanding for them to finish the said project;
function with error rate of 5.0002%. and for motivating them to give nothing but their best.
The authors assess the result of the experiments by To Engr. Roselito E. Tolentino, their subject adviser, for
observing the possible reasons of the missed recognition. his guidance and supervision throughout all the phases of
From the gathered data, it can be seen that there are several the work, for reviewing their work and give some advises
reasons for the misclassification of the gestures. The that greatly helped the group to come up with the finished
authors evaluate that some misclassified gestures are work.
caused by disorientation because of the tilting or slanting And most of all, the authors would like to thank our
of the user’s hand while executing a function (drag and Almighty God for all the blessings and for everything that
drop). It is also show that when the gesture is not He outspreaded all throughout the difficulties and
performed well from the start of the execution, obstacles of this research.
misclassification of the hand gesture occurs which affects
the recognition of the system. This happens when the 6. References
system (camera) is not able to determine that the user [1] Suresh, D. S. and Bhavana V. Virtual Mouse
already moved her finger, in case of the right click Implementation using Color Pointer Detection.
function. International Journal of Research Studies in Science,
This study aims to compare the result with previous Engineering and Technology, 2014.
studies. As to compare the results from the previous study, [2] Zhong , W., Zhang, Q., et. al. The Simulated Mouse
the overall system’s performance with regards to accuracy Method Based on Dynamic Gesture Recognition. 8th
is computed based on the five mouse functions. The International Congress on Image and Signal
previous study achieved a rate of recognition of about Processing, 2015.
58.8% while the current study has overall recognition rate [3] Patil, A. M., Dudhane S. U., et. al. Cursor Control
of 95.33332%. System Using Hand Gesture Recognition.
The authors also compare the result of the experiment International Journal of Advanced Research in
from other two papers. The results of the past studies are Computer and Communication Engineering, 2013.
87.75% and 87.5%. The result of the current study implies [4] Zhang, K., Huang, C. and Tsai, T. Embedded virtual
that there is significant improvement in terms of accuracy mouse system by using hand gesture recognition.
of the system. IEEE International Conference on Consumer
Electronics, 2015.
4. Conclusion [5] Adjeisah, M., et. al. Joint Filtering: Enhancing
Recognition of the dynamic hand gesture is carried upon Gesture and Mouse Movement in Microsoft Kinect
the implementation of Template Matching in executing Application. 12th International Conference on Fuzzy
mouse functions. The designated dynamic hand gestures Systems and Knowledge Discovery, 2015.
for the mouse functions are unique, so that it will not cause [6] Carrera, K. P., et. al. Application of Template
confusion to the system. The saved template is used as a Matching Algorithm for Dynamic Gesture
basis for comparing the actual template executed by the Recognition of American Sign Language Finger
user and the existing template. Spelling and Hand Gesture. Asia Pacific Journal of
The authors find that hand detection process of the system Multidisciplinary Research, 2014.
highly affects the recognition process of the dynamic hand [7] Pham, I., Jalovecky, R., and Polasek, M. Using
gesture because the pre-processes of the system should be template matching for object recognition in infrared
completed properly to make the actual templates have less video sequences. IEEE/AIAA 34th Digital Avionics
noise and be ready for recognition process. This means Systems Conference, 2015.
that when the detection is not properly working, process of

13
Graphics, Vision and Image Processing Journal, ISSN 1687-398X, Volume 17, Issue 1, ICGST LLC, Delaware, USA, June 2017

[8] Jarial, P., and Kaur, A. Object Recognition using LAVIÑA, COLIN B. He
Template Matching with the Help of Features studied at Polytechnic
Extraction Method. International Research Journal of University of the Philippines-
Engineering and Technology, 2015. Santa Rosa Campus. He
[9] Hemantha, M. and Srikanth, M. Simulation of Real received the degree in Bachelor
Time Hand Gesture Recognition for Physically of Science in Electronics and
Impaired. International Journal of Advanced Communications Engineering in
Research in Computer and Communication 2017. He was a member of
Engineering, 2013. IECEP (Institute of Electronics Engineers of the
[10] Williamson, A. Cursor Control Using Hand Gestures Philippines.
and a Webcam, 2014. http://www.academia.edu/
[11] Ahuja, K., and Tuli, P. Object Recognition by MABUTI, GERALIE S. She
Template Matching using Correlations and Phase studied at Polytechnic
Angle Method, 2013. http://ijarcce.com/ University of the Philippines-
[12] Wang, X., Xia, M., et. al. Hidden-Markov-Models- Santa Rosa Campus. She
Based Dynamic Hand Gesture Recognition, 2015. received the degree in Bachelor
http://www.kurims.kyotou.ac.jp/ of Science in Electronics and
[13] Puri, R. Gesture Recognition Based Mouse Communications Engineering in
Events, 2014. https://arxiv.org/ 2017. She was a member of
IECEP (Institute of Electronics Engineers of the
Biographies Philippines.
ARCANGEL, ROSE ANN D.
She studied at Polytechnic ROSELITO E. TOLENTINO
is a registered Electronics
University of the Philippines-
Engineer and IECEP Member.
Santa Rosa Campus. She He is a graduate of B.S.
received the degree in Bachelor Electronics and
of Science in Electronics and Communication Engineering at
Communications Engineering in Adamson University in 2004
2017. She was a member of IECEP (Institute of under the scholarship of DOST.
Electronics Engineers of the Philippines. He finished his Master of Science in Electronics
Engineering Major in Control System at Mapua
Institute of Technology under the scholarship of DOST-
CAPUSO, SHALAINE T. She
ERDT. He currently takes up Doctor of Philosophy in
studied at Polytechnic
Electronics Engineering at the same Institute. He is
University of the Philippines-
currently working as a part time instructor at
Santa Rosa Campus. She
Polytechnic University of the Philippines Santa Rosa
received the degree in Bachelor
Campus and De La Salle University - Dasmarinas. His
of Science in Electronics and
research interests are more on Robotics and Machine
Communications Engineering in
Vision.
2017. She was a member of
IECEP (Institute of Electronics Engineers of the
Philippines.

GERBUYOS, MARQUEE P.
She studied at Polytechnic
University of the Philippines-
Santa Rosa Campus. She
received the degree in Bachelor
of Science in Electronics and
Communications Engineering in
2017. She was a member of
IECEP (Institute of Electronics Engineers of the
Philippines.

14