International Conference on Communication and Signal Processing, April 6-8, 2016, India

Finger Cursor Using Gesture Recognition

Shiva Gupta, Suvarna Pansambal, Swati Shirke

Abstract—Since the computer is developed we are using mouse Our system is based on webcam. We have developed this on
for cursor movement. In this era of touch screen devices human Windows 7 operating system in MATLAB R2008a software.
machine interaction is very important factor nowadays. Many We have used two small objects of green color for detection.
devices are developed which can be operated without touching Our project is not so expensive to develop as only one
the system. So there should be any system in computer and
hardware (webcam) and one software (Matlab) is used. We
laptop also to operate them without touching physically. So in
this paper we have described new concept named ‘Finger have used two objects of same color for cursor movements
Cursor’. It means operating computer’s cursor using our and mouse events. Webcam will first capture the live video
finger’s gestures. In this paper we have described methods and and then we divide it into frames. Then processing on that
algorithms we have used for this system. We have described frames is done. Processing includes extracting required color,
methods for color detection, image filtering, finding centroid and removing noise from frames, converting frames to binary,
bounding a box to it, mouse movements and executing mouse finding centroid of object, making bounding box to object,
events. tracking centroid, moving mouse and performing mouse
events.
Keywords—median filter, color detection, centroid.
The rest of the paper is organized as follows. Section II
I. INTRODUCTION summarizes the related works. Section III gives proposed
system and mouse events. Section IV presents the conclusion.
As we know today human–machine interaction is very
important aspect for user friendliness and ease access. Many
touch screen devices like mobile phones, tablets and even
touch screen TVs are developed .Most of the systems today
works on touch screen technology. Computers and laptop will
also start using this touch screen technology. But using touch
screen system is too often nowadays. So we have studied and
developed new system i.e. ‘Finger Cursor Using Gesture
Recognition’. Till now everybody is using mouse to operate
computers. But survey has found that there are many health
problems causes using mouse. It has found that due to
incorrect position of mouse from users sitting position and
very acute movements of hand required to operate mouse
causes many health related problems. The placement of mouse
is awkward to reach. Mouse is placed away from the user’s
sitting position and also mouse wire is kept short which make Fig. 1. Problems due to excessive usage of mouse
it difficult to use. Medical problems like carpal tunnel
syndrome, repeated motion injuries, gorilla arm problem,
mouse shoulder, computer back occurs due to excessive use of
mouse [1].

Shiva Gupta, System Enginerr, Infosys Limited, Pune, Maharashtra. (e.mail:

[email protected])

Suvarna Pansambal (Shirke), Dept. Of Computer Engineering, Atharva

college of Engineering, Mumbai, Maharashtra. (e.mail: Fig. 2. Wrong Mouse Movement
[email protected])

Swati Shirke, Dept.of Computer Engineering, NBN Sinhgad, School of

Engineering, Pune, Maharashtra. (e.mail: [email protected])

978-1-5090-0396-9/16/$31.00 ©2016 IEEE

orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 06,2025 at 05:32:07 UTC from IEEE Xplore. Restrictions ap

1747
 II. RELATED WORK figure shape offers on the shading channels autonomously, the
utilization of shading traits as an express shading
Abhik Banerjee et al [1] attempted to control mouse representation is proposed. The subsequent picture
developments utilizing webcam and shading tapes.A constant representations are smaller and computationally proficient
feature has been caught utilizing a Web- Camera. The client while giving great location execution on difficult datasets. CH.
wears shaded tapes to give data to the framework. Sravana Lakshmi et al [5] utilized Modified Decision Based
Transformingof individual edges of the feature happen Unsymmetric Trimmed Median Filter (MDBUTMF) strategy
independently. The handling systems include a picture to expel commotion from pictures. M.Sankari, C. Meena [6]
subtraction calculation to identify hues. When the hues are has utilized versatile average channel to evacuate clamor.
recognized the framework performs different operations to SandeepKatta [7] portrayed system for changing over
track the cursor and performs control activities. No extra grayscale picture to binary picture. A. SaiSuneel [9] put
equipment is needed by the framework other than the standard systems for object tracking, picture filtering. Bala Subburaman
webcam which is given in every smart phone or laptop Venkatesh [11] utilized sliding window strategy for boundary
computer. RachitPuri [2] has utilized YCbCr shading model box estimation. In this paper introduces a technique to reduce
for motion distinguishment. A webcam has been utilized to the number of miss detections while increasing the grid
perceive the signals. The above exploration has likewise been spacing when using the sliding window approach for object
tried on ease 1.3 megapixel portable computer webcam. There detection. Utilizing a little fix the bounding box of an article
is limit esteem for separation (in meters) between the client inside a neighborhood look region is anticipated. It is vital that
and cam which can further be shifted as per cam's the bouncing box forecast is practically identical or better than
determination. It means subject needs to verge on certain the time it takes in normal for the item classifier to reject a
settled separation to the cam to be perceived with his hand subwindow to accomplish speed. A choice tree as is
motions in some environment. This examination was carried effectively utilized for the application. KazimSekeroglu [12]
out on 1.3 megapixel webcam with edge estimation of 2m. described various MATLAB functions for colour detection,
Kamran Niyazi et al [3] utilized two shading tapes and utilized binary image conversion, noise detection, flipping etc.
separation in the middle of thumb and finger to recognize Radhika Bhatt et al [13] studied various health problems
mouse cursor developments. To start with, the potential hand causes due to mouse use.
pixels from the non-hand pixels are differentiated utilizing
foundation subtraction plan which portions any potential
closer view hand Information from the non-changing III. PROPOSED SYSTEM
foundation scene. At the framework startup, a couple of
foundation pictures are caught to speak to the static workspace
from cam view. Utilizing fitting foundation picture the
resulting casings are sectioned out, consequently moving
closer view information. The procedure of skin division is
carried out after foundation subtraction where a histogram-
based skin classifier appoints each of the RGB pixels in the
preparation set to either a 3D skin histogram or non-skin
histogram. The skin division procedure yields a picture which
is prepared for discovery of shading tapes in the finger. The
likelihood is figured that given RGB shading has a place with
the skin or non-skin classes taking into account these
histograms. For this a calculation in light of HSV shading
space is utilized which is exceptionally compelling to choose a
certain shading out of a picture. On the off chance that the
RGB pixels are changed over into the HSV shading plane, it
will be less influenced to varieties in shades of comparative
shading. At that point, a resilience cover is utilized over the
changed over picture as a part of the immersion and shade
plane, coming about into a double picture which is then go
through a convolution stage to decrease the commotion
presented. FahadShahbaz Khan et al [4] utilized blend of
shading and shape to distinguish object. In this examination
the focal points of joining shading with shape on the two most
prevalently utilized identification systems, in particular part-
based recognition with deformable part models and Efficient
Subwindow Search (ESS) for article confinement is
Fig. 3. Workflow of System
illustrated. As opposed to routine combination approaches that

orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 06,2025 at 05:32:07 UTC from IEEE Xplore. Restrictions ap

1748
 Algorithm B. Frame Grabbing
Algorithm Finger Cursor Then we have acquired frames from this live video. We set 20
x Take video input from camera frames per trigger with no limit on repetition of trigger.
x Get snapshot of each frame and perform following
C. Flipping Frames
operations
Live video captured by webcam is always horizontally flipped.
x Flip the input Vertically it is as it is. Horizontally flipped means if we move
x Extract the grey scale of green colour our hand from left to right, the video will show it moving right
x Apply median filter to eliminate noise to left and vice versa. So for better performance and ease use
x Convert to binary image and to avoid confusion of movements we need to flip the
acquired image. So after grabbing frames we have flipped
x Find and plot centroid of the object detected in binary
each frame horizontally. MATLAB inbuilt function flipdim is
image used for flipping process.
x Divide the input image into 9 equal parts
x In case only one object is detected:-
x If object is in 1st region mouse moves in upper left
direction
x If object is in 2nd region mouse moves in upper
direction
x If object is in 3rd region mouse moves in upper right
direction
x If object is in 4th region mouse moves in left
direction
x If object is in 5th region mouse performs single click
after 2 seconds
x If object is in 6th region mouse moves in right
direction
x If object is in 7th region mouse moves in lower left
direction
x If object is in 8th region mouse moves in down
direction
x If object is in 9th region mouse moves in lower left
direction
x If two objects are detected :-
x If 1st object is in 5th region and 2nd object is in 3rd
region right click is performed
x If 1st object is in 5th region and 2nd object is in 9rd
region double click is performed
x If 1st object is in 2th region and 2nd object is in 6rd Fig. 4. Flipping the image

region scroll up is performed D. Detecting required colour

x If 1st object is in 8th region and 2nd object is in 6rd Now next step is to acquirered, blue or green color. We have
region scroll down is performed selected green color over red and blue because both red and
x In case if no object is detected then no operations will blue colors are very often in background, so selecting red or
blue color object was causing problems in detection because
be performed
we needed to adjust much with surrounding and even
sometime skin color also appears as red. So we selected green
A. Acquiring video from webcam colored object for detection. It is not necessary that you use
This is first step of our project. We have used 16 megapixel only color band, you can use any small object like pencil
webcam to acquire live video. We set default resolution of sharpner, pen cap etc. but of that particular color. To detect
640x480. We acquired video in RGB color space. green object we have subtracted green color component from
RGB frames. Then we converted that RGB image into gray
scale image. After that we have filtered this image with

orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 06,2025 at 05:32:07 UTC from IEEE Xplore. Restrictions ap

1749
 median filter. MATLAB’s imsubtract function is used to In whatever region the green object is the mouse will move in
extract required color from frames. We have used rgb2gray that direction from its original position.If object is in region-1
for converting rgb image to gray. then cursor will start moving upper-left direction at constant
speed. If in region-3 then upper-right direction, if in region-7
then bottom-left direction and if in region-9 then bottom-right
E. Median Filter
There are many filters available for filtering noise in image.
Median filter is one of them. We have used median filter to
filter noise in image. The median filter puts an empty window
on image matrix. It then considers each pixel in the image one
by one and then looks at its nearby neighbor pixels to decide
whether or not it is representative of its surroundings. Then it
replaces center pixel value in that window with the median of
those values. The median of those values is calculated by first
sorting all the pixel values from the surrounding neighborhood
into numerical order and then replacing the pixel being
considered with the middle pixel value. If the neighbors are an
even number of pixels, then average of the two middle pixel
values is used to replace.MATLAB’s medfilt2 function is
used for filtering image.

F. Coverting Into Binary Fig. 5. Dividing Screen into Region

The grayscale image is converted into a binary image in the .
next step. The output image replaces all pixels in the input
image into black and white color. A threshold is set which is
used to decide which pixels are black and which are white.
Pixels below threshold value are converted to white and
other remaining pixels are converted to black. im2bw is used
for binary conversion.bwareaopen is used to clean binary
image by removing small objects.

G. Detecting Centroid
Next we need to track the object we required i.e. green color
object. To track object we have to track centroid of that object
so we can get its exact location on screen. That’s why
detection of centroid is required. Centroid gives x and y Fig. 6. Subdivision of the region 2,4,6,8
coordinates of object’s center which shows where exactly the
object is on screen.

H. Putting Bounding Box

As centroid shows where object is, bounding box is required
to bind the object area. So we have drawn rectangular box to
the object which indicates exact object’s area.

I. Mouse Movement
Mouse movement is main part of our project. For all above
steps we have used MATLAB’s built-in functions but
MATLAB has no built-in function for mouse movements and
mouse events. For this process we have used Robotic class of
java i.e. java.Robot.awt. Also we have implemented our own
algorithm for mouse movements and events. We have divided
output screen (640x480) into 9 equal regions as shown in Fig.
5.

orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 06,2025 at 05:32:07 UTC from IEEE Xplore. Restrictions ap

1750
 of objects or with this gesture left double click will occur.
Here first object is leftmost object, so for right handers first
object is object in index finger and second object is object in
middle finger.

Fig. 7. Cursor movement direction and speed

direction. Mouse movement will stop if the object is in center

region i.e. region-5. If the object is in region 2, 4, 6 and 8 then
cursor will move in up, left right and down respectively. But
we have divided region 2, 4, 6 and 8 in two sub-regions,
asshown in Fig. 6. As shown in Fig. 6, all this four regions are
divided equally in two parts, region 2 and 8 horizontally and
region 4 and 6 vertically according to mid-point. This is for
separating cursor movement speed wise. As shown in figure, Fig. 9. Object position for left double click.
if object is in slow region cursor will move with less speed
and if object is in fast region cursor will move with more
speed. 3) Right click:
Right click is bit similar to left double click. For right click
Mouse events first object’s position will be in center region but second
After cursor movement mouse events are required. We have object must be in region-3 as shown in figure.
added mouse events like left single click, left double click,
right click, scroll up and scroll down. For these mouse events
we have kept different gestures for different mouse events.
Here also we have make use of regions. For mouse events we
used two objects of same color, one to be wear in index finger
and second in middle finger.
The positions for two objects to perform different events are
as follow:

1) Left single click:

For left single click only one object is required. For left single
click object must be in center region i.e. region 5 (means
mouse movement has stopped) and it must be hold there for 2 Fig. 10. Object position for right click
second to occur left single click event.
4) Scroll up and down:

For scroll up and down second object’s position is static. For

both scroll-up and down second object must be in region-6.
While second object i.e. middle finger is in region-6 and if we
keep first object in region-2 the scroll up will occur. Similarly
if we keep first object in region-8 scroll-down will occur.

Fig. 8. Object position for left single click.

2) Left double click:

For left double click we required two fingers i.e. two objects.
For this click position of first object will be in center region
(region-5) and second object is in region-9. With this position Fig. 11. Object position for scroll up.

orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 06,2025 at 05:32:07 UTC from IEEE Xplore. Restrictions ap

1751
 algorithm required for cursor movement and mouse events.
Also we have compared our project with other similar
projects; we have found that no one has implemented scroll up
and scroll down event in there project.

REFERENCES

[1] Abhik Banerjee, Abhirup Ghosh, Koustuvmoni Bharadwaj, Hemanta

Saikia, “Mouse Control using a Web Camera based on Colour
Detection”, [International Journal of Computer Trends and
Technology(IJCTT) – volume 9 number 1– Mar 2014 ISSN: 2231-2803]
Fig. 12. Object position for scroll down. [2] Rachit Puri, “Gesture Recognition Based Mouse Events”, [International
Journal of Computer Science & Information Technology (IJCSIT) Vol
5, No 6, December 2013]
[3] Kamran Niyazi, Vikram Kumar, Swapnil Mahe, Swapnil Vyawahare,
TABLE I “Mouse Simulation Using Two Coloured Tapes”, [International Journal
COMPARISON TABLE of Information Sciences and Techniques (IJIST)Vol.2, No.2, March
Paper Mouse Events 2012]
Name Move Left Left Scroll Right [4] Fahad Shahbaz Khan, Rao Muhammad Anwer, Joost van de Weijer,
Click Double Click Andrew D. Bagdanov, Maria Vanrell, Antonio M. Lopez, “Colour
Attributes for Object Detection”
Click
[5] CH.Sravana Lakshmi1, V.Ambika2, K.Suri “Impulse Noise Removal
Mouse In images Using Modified Trimmed Median Filter:Matlab
Control using Implementation And Comparitive Study ”, [International Journal of
a Web
Camera — — — Engineering Research and Applications (IJERA) ISSN: 2248-9622
Vol. 2, Issue 5, September- October 2012]
based on [6] M.Sankari,C. Meena,“Estimation of Dynamic Background and Object
Detection in Noisy Visual Surveillance” [(IJACSA) International
Color Journal of Advanced Computer Science and Applications, Vol. 2, No.
Detection [1] 6, 2011]
Mouse [7] Sandeep Katta, “Visual Secret Sharing Scheme using Grayscale
Simulation Images” [Department of Computer Science, Oklahoma State
University Stillwater, OK 74078]
Using Two
Colored — — — — [8] Rupali Patharkar, Sonali Bodkhe, Dr. A.R Mahajan “Background
Detection by Two Way Technique” [(IJCSIT) International Journal of
Tapes [3] computer Science and Information Technologies, Vol. 4 (3), 2013,
Gesture 289 – 291]
Recognition [9] A.Sai Suneel “Person or object tracking and velocity estimation in

— — — —
Based Mouse real time videos” [Vol 04, Special Issue01; 2013 Publications of
problems & application in engineering research - Paper
Events [2] http://ijpaper.com/CSEA2012 ISSN: 2230-8547; e-ISSN: 2230-8555]

— — — —
Virtual [10] Shubham Srivastava, Pratibha Singh “Real-Time Object Tracking
Mouse Using Using Colour Feature” [International Journal of Innovative
a Webcam Technology and Exploring Engineering (IJITEE) ISSN: 2278-3075,
Volume-3, Issue-8, January 2014]
[12]
[11] Bala Subburaman Venkatesh1,2 and S´ebastien Marcel “Fast
Finger Bounding Box Estimation based Face Detection” [Idiap Research
Cursor Institute, 1920, Martigny, Switzerland Ecole PolytechniqueF´ed´erale
Using
Gesture
— — — — — de Lausanne (EPFL), 1015, Lausanne,Switzerland]
[12] Kazim Sekeroglu, Semister report on “Virtual Mouse Using a
Webcam”
Recognition
[13] Radhika Bhatt , Nikita Fernandes , Archana Dhage “Vision Based
[Our Hand Gesture Recognition for Human Computer Interaction”
Paper/Syste [International Journal of Engineering Science and Innovative
m] Technology (IJESIT) Volume 2, Issue 3, May 2013]

IV. CONCLUSION
This is how we developed a concept of ‘Finger Cursor Using
Gesture Recognition’. In this paper we have stated our own

orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 06,2025 at 05:32:07 UTC from IEEE Xplore. Restrictions ap

1752