VIRTUAL MOUSE USING HAND GESTURES

RECOGNITION
A Project Report

Submitted By

M.RAJESH
200303124373

in Partial Fulfilment For the Award of

the Degree of

BACHELOR OF TECHNOLOGY

COMPUTER SCIENCE AND ENGINEERING

Under the Guidance of

Prof.MUSKAN KUMARI

Assistant Professor

PARUL UNIVERSITY

VADODARA

October - 2023
 PARUL UNIVERSITY
C ERTIFICATE
This is to Certify that Project - 2 -Subject code 203105400 of 7th Semester entitled “VIRTUAL
MOUSE USING HAND GESTURES RECOGNITION” of Group No. PUCSE 135 has been
successfully completed by

• M.RAJESH - 200303124373

under my guidance in partial fulfillment of the Bachelor of Technology (B.TECH) in Computer

Science and Engineering of Parul University in Academic Year 2023- 2024.
Date of Submission :-

Project Guide Head of Department,

Prof. Muskan Kumari Dr. Amit Barve

Project Coordinator CSE, PIET

Prof.Yatin Shukla Parul University

External Examiner

ii
 Acknowledgements

”The single greatest cause of happiness is gratitude.”

-Auliq-Ice

Any significant project that a person undertakes has the support of the people who assisted
him in overcoming obstacles and achieving his objective. It brings me great pleasure to offer my
profound gratitude to my renowned Prof. Muskan Kumari for her steadfast, exceptional, and
extremely helpful collaboration and mentoring. Being led by her is a success for me. She provides
regular motivation and encouragement, which makes any complexity simple. She provided me with
a great deal of insightful advice and timely suggestions during the course of the assignment. I will
always be grateful to her, and I’m proud to have worked for her.

I also want to convey my gratitude and respect to Professor and Department Head of Computer
Science and Engineering, Dr. Amit Barve, I consider myself extremely fortunate to have received
his invaluable counsel, direction, and leadership. Last but not least, a sincere thank you to the
All-Powerful God.

Place : Vadodara
Date :

M.RAJESH- 200303124373
 Abstract

This project presents an approach to develop a real-time hand gesture recognition based in
—Vision Based|| that uses only built-in-camera and Computer Vision technology, such as image
processing that can acknowledge many gestures for use in computer human interaction.

The application of real time hand-gesture recognition in the real world are countless, due to the
fact that it can be used almost anywhere where we interact with computers. The Principle
application of this project is to imitate the mouse as a visual inputting device with all of its tasks
such as left click, selecting, curser moving and scroll.
Table of Contents

Acknowledgements iii

Abstract iv

Table of Contents v

List of Figures viii

1 Introduction 1

1.1 PROBLEM STATEMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 PROJECT OVERVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 APPLICATIONS FOR PROJECT . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3.1 Augmented reality (AR) and virtual reality (VR) . . . . . . . . . . . . . . 2

1.3.2 Home automation and smart TVs: . . . . . . . . . . . . . . . . . . . . . . 3

1.3.3 Applications in Education: . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3.4 Report Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.4 System Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.5 Software Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.5.1 Open CV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.5.2 Numpy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.5.3 Media pipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.5.4 Autopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

v
 TABLE OF CONTENTS

1.5.5 PyAutoGUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.6 System Work flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.7 DETECTION PROCESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.7.1 Image acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.7.2 Hand detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.7.3 Hand Recognition Images . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.8 Gesture Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.8.1 Gesture Recognition Images . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.8.2 Implemented work outputs . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1.9 Virtual Mouse Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

1.9.1 Applications using Autopy . . . . . . . . . . . . . . . . . . . . . . . . . . 15

1.9.2 Applications using Pyautogui . . . . . . . . . . . . . . . . . . . . . . . . 15

1.10 ALGORITHM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.11 TOOLS AND TECHNOLOGY . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.11.1 Software Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.11.2 Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2 Literature Survey 17

2.1 Non geometric characteristics are used in a multivariate Gaussian distribution to

identify hand movements Hasan, et aI 2020 . . . . . . . . . . . . . . . . . . . . . 17

2.2 Computational Gaussian distribution applied on the segmented image Hasan; et al

2020. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.3 Recognizing mouse click and movement eventss.shetty et al 2015. . . . . . . . . . 18

2.4 Expansion of mouse-free cursor control technique P.C.Shindhe et al . . . . . . . . 18

2.5 Controlling mouse pointer using webcam G. Sahu et al.; D. Gupta et al. . . . . . . 18

2.6 Create and implement a hand-based virtual mouse. . . . . . . . . . . . . . . . . . 18

vi
 TABLE OF CONTENTS

2.7 Interaction between human and machine using hand gestures recognition

(cheng,zhang,wang,2004) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.8 Recents improvements in hand gestures recognition for human computer interaction

(rajesith, li, zhang, niu,2008) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.9 Hand Posture and Gesture Recognition Techniques and

Technology(Czuszynski,Murad,2009) . . . . . . . . . . . . . . . . . . . . . . . . 19

2.10 Techniques for Hand Gesture Recognition in Virtual Reality(Mitra, Acharya,2012) 19

2.11 Recognition of Hand Gestures Based on Vision(Rautaray, Agrawal,2007) . . . . . 20

2.12 Some gestures are unimanual some gestures are bimanual symmetric Bourdot et al.

(2010) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.13 Gestures are categorized in time as either static or dynamic.(D.Gupta, G.Sahu, 2006) 22

2.14 Contextual classification(Quek, 1995) . . . . . . . . . . . . . . . . . . . . . . . . 22

2.15 Use a neural network method to identify American Sign Language’s static posture

Kulkarni . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.16 Normalization based on brightness factor matching for gesture recognition . . . . . 23

2.17 Boundary histogram-based postures that are rotation-invariant Wysoki el at . . . . 23

3 Conclusion 24

4 Future Work 25

vii
List of Figures

1.1 System work flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.2 Hand LandMarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Left Hand. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4 Right Hand. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Both Hand. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6 Index finger for moving. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.7 Thumb finger for Left Click. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.8 Index and Middle fingers for scroll up. . . . . . . . . . . . . . . . . . . . . . . . . 11
1.9 Index and pinky fingers for scroll down. . . . . . . . . . . . . . . . . . . . . . . . 12
1.10 Checking the model whether recognizing or not. . . . . . . . . . . . . . . . . . . . 13
1.11 Thumb finger for Left click. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.12 Index and pinky fingers for scroll down. . . . . . . . . . . . . . . . . . . . . . . . 14
1.13 Index finger. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

viii
Chapter 1

Introduction

Human-computer interaction has been transformed by the development of hand gesture detection
technology, which gives consumers a more natural and engaging experience when using digital
gadgets. The development of a virtual mouse that can be operated with hand gestures is an
intriguing application of this technology. With this advancement, people can traverse digital worlds
naturally and without using their hands. It goes beyond conventional input techniques. We will
discuss the idea of a hand gesture-driven virtual mouse in this introduction, as well as its possible
advantages and effects on the field of interactive computing.

The physical mouse is replaced by a digital one that reacts in real-time to the user’s hand
movements thanks to hand gesture recognition technology. Users can point, click, and browse
through computer interfaces utilizing gestures made in the air rather than interacting with a physical
device. This idea not only adds a layer of futuristic interactivity but also takes accessibility and
hygienic issues into account.

1.1 PROBLEM STATEMENT

The challenge here is to find a means for people to communicate with a computer without being
physically connected to it. Many suggestions were made, but each one called for a specific piece of
hardware. Utilizing the photoelectric effect concept is another technique. However, a hardware is
required for it, and it is not financially viable. The fourth alternative is to create a virtual mouse that
leverages straightforward and affordable image processing methods.

1
 CHAPTER 1. INTRODUCTION

1.2 PROJECT OVERVIEW

By developing a simple and hands-free approach for users to manipulate digital interfaces, the
Virtual Mouse Using Hand Gesture Recognition project seeks to revolutionize human-computer
interaction. The project suggests creating a virtual mouse system that can recognize and react
to users’ hand motions in real-time using cutting-edge computer vision and machine learning
algorithms. This advancement caters to a variety of applications, including gaming, presentations,
healthcare, and more, and is consistent with the general trend towards natural and immersive
interfaces.
Realize excellent accuracy when identifying different hand movements. Implement powerful
computer vision techniques and, if possible, deep learning models to reliably and precisely interpret
gestures. In order to provide a seamless and natural contact experience, real-time responsiveness
should be prioritized. Algorithms should be improved for low latency processing, especially in
high-performance applications like virtual reality environments.
Create a virtual mouse system that is adaptable and works with a variety of apps. Ensure seamless
integration with presentation tools, gaming platforms, healthcare interfaces, and other sectors to
enable wide adoption.
OpenCV is a well-known computer vision library that offers a variety of tools for processing images
and videos. It can be used to build hand gesture recognition because it has modules for object
identification, image processing, and feature extraction.

1.3 APPLICATIONS FOR PROJECT

A hand-gesture-controlled virtual mouse in the gaming sector provides a more interactive and
immersive experience. Players can control characters, traverse menus, and carry out in-game tasks
with simple hand motions.

1.3.1 Augmented reality (AR) and virtual reality (VR)

For the creation of engaging VR and AR experiences, hand gesture recognition is essential. Users
can use their natural hand movements to engage with virtual environments, manipulate objects, and
operate programs within these realities.

2
 CHAPTER 1. INTRODUCTION

1.3.2 Home automation and smart TVs:

An easy method of interacting with smart TVs and home automation systems uses a virtual mouse
that can be controlled by hand gestures. Without the need of remote controllers, users can change
the volume, manage smart home devices, and switch between channels.As an example of this, we
integrated it with online interactive 2 player game so that two players can compete in the game each
with his hands.

1.3.3 Applications in Education:

A virtual mouse that can be operated with hand gestures in educational settings can improve hands-
on learning opportunities. Students can handle 3D models, engage with educational applications,
and take part in virtual experiments.

1.3.4 Report Summary

• The representation of the system environment, software system, and system restrictions
employed in this project will be protected.

• The detection step, which covers the acquisition and transformation of images throughout the
image processing life cycle until the required images are needed for recognition.

• Using the result of the detection procedure, or recognition, to identify the gesture will be
discussed.

• For this project, each of the events that will be produced in response to gestures will be defined
independently.

• Following that, the result will be shown.

• We’ll discuss the references we utilized.

1.4 System Environment

The user must place their hand in an area without strong light so that the system camera can
concentrate on it. The monitor will display the output while the camera’s input is being captured.
The operator will interact with the system by gesturing in front of the camera without wearing a
colored wristband. Using opencv, the hand’s shape and location will be acquired and detected.

3
 CHAPTER 1. INTRODUCTION

1.5 Software Information

We are using machine learning technologies to create this project. Machine learning is defined as
an application of artificial intelligence where data is utilized to process or support the processing
of statistical data using algorithms. While automation elements are included in machine learning,
human interaction is still necessary. A group of data analysis methods are offered by the relatively
young field of computer science known as application of man-made intelligence. A select few of
these approaches (such as principal component analysis and logistic regression) and many more are
based on well-proven statistical techniques.

1.5.1 Open CV

In our experiment, we used the OpenCV 2.0 library to process video frames. Intel or Progress’
Open Source Computer Vision Library is known as OpenCV. It is a collection of C result and a
few C++ classes that execute a few well-liked image processing and computer vision techniques.
OpenCV is a middle to high level cross-platform API made up of several hundred C components.
While some of them may be used at compile time, it is independent of external numerical libraries.
We must import this library as import cv2 in order to use it.

1.5.2 Numpy

A library made up of multidimensional array objects and a collection of patterns for processing
arrays is called the NumPy Library, which stands for Numerical Python. It provides a vast library of
high-level mathematical functions that work on these arrays as well as sophisticated data structures
to Python that provide properly organized computations using arrays and matrices. Additionally, it
will provide a concise explanation of the basic mathematical operations in a simple manner. Numpy
is becoming more well-known and is utilized in several production systems. We must import numpy
in order to use this library.

1.5.3 Media pipe

The MediaPipe framework is utilized for hand tracking and gesture detection, and the OpenCV
library is used for computer vision. For the purpose of tracking and identifying hand movements
and hand tips, the set of instructions uses machine learning ideas. A framework called MediaPipe
is a Google open source framework that is applied in a machine learning pipeline. Given that
the framework was created utilizing time series data, the Mediapipe structure is advantageous
for cross-platform development. The MediaPipe structure has multiple modalities, allowing this

4
 CHAPTER 1. INTRODUCTION

architecture to be used with a variety of audio and video files.Developers create and analyze systems
through plot using the MediaPipe structure. They also utilize it to evolve systems for application
purposes. The pipeline configuration is where the actions in the MediaPipe-using system are carried
out. We must import this library as import mediapipe in order to use it.

1.5.4 Autopy

An easy-to-use GUI automation toolkit for Python is called Autopy. It includes cross-platform,
effective, and straightforward controls for the keyboard and mouse, locating colors and bitmaps
on the screen, and showing alarms. Several mouse control functions are included in AutoPy. The
autopy module has many functions for simulating mouse movements and button clicks of the mouse
wheel. Mainly it works in our project as a functional controlling of a mouse. To use this library we
have to import it as: import autopy.

1.5.5 PyAutoGUI

PyAutoGUI is a python automation library. It includes functions for controlling the mouse in a
simple manner. This package is works on the windows, linux and macOS x which provides the
ability to simulate mouse curser moves and button clicks. Mainly it works in our project as a
functional controlling like scrolling of the mouse. To use this library we have to import pyautogui.

5
 CHAPTER 1. INTRODUCTION

1.6 System Work flow

Figure 1.1: System work flow.

6
 CHAPTER 1. INTRODUCTION

1.7 DETECTION PROCESS

1.7.1 Image acquisition

Using the camera, the video is captured and making the each frame more clear. For accessing the
system camera and capture video, we used cv.2VideoCapture().

1.7.2 Hand detection

Step 1: From the video, we get to differ the hand from other objects by using the library source
named opencv.
Step 2 : Human hand gets a landmarks after getting detected, by using the package called mediapipe.

Figure 1.2: Hand LandMarks.

Step 3 : Marked hand gets the mathematical representation of hand tips by using the numpy library.

7
 CHAPTER 1. INTRODUCTION

1.7.3 Hand Recognition Images

Figure 1.3: Left Hand.

Figure 1.4: Right Hand.

8
 CHAPTER 1. INTRODUCTION

Figure 1.5: Both Hand.

1.8 Gesture Recognition

We are making the system to recognize the gestures of human hand using the representation of hand
landmarks.
Step 1: We are representing the each finger as finger() and tips as fingertip(). And giving the indexes
for fingers and tips as follows

1. Thumb finger as finger(0) and tip as fingertip(4).

2. Index finger represented as finger(1) and fingertip(8).

3. fingertip (12) and fingertip (2) on the middle finger.

4. Ring finger as finger(3) and tip as fingertip(16).

5. Pinky finger as finger(4) and tip as fingertip(20).

Step 2: We are giving the indication whether the finger is up or down.

1. If the finger is up then it indicates it as 1.

2. If the finger is down then it indicates it as 0.

9
 CHAPTER 1. INTRODUCTION

Step 3: We are deciding the gestures depending on user hand indications.

1. Firstly, user has to raise all fingers, then after the gesture denotes gets start.

2. If the only index finger is up, where finger(1) is 1 and remaining are 0 then the curser has to
move from current position to next position.

3. If only the middle finger is up then the curser has to be hold at the current location or position.

4. If the thumb finger is up then the lift click operation has to take place.

5. The scroll up function must be used if both the index and middle fingers are up.

6. The scroll down feature must be used if both the index and pinky fingers are up.

7. If the all fingers are down then the no function is going to happens. Now gesture is determined.

1.8.1 Gesture Recognition Images

Figure 1.6: Index finger for moving.

10
 CHAPTER 1. INTRODUCTION

Figure 1.7: Thumb finger for Left Click.

Figure 1.8: Index and Middle fingers for scroll up.

11
 CHAPTER 1. INTRODUCTION

Figure 1.9: Index and pinky fingers for scroll down.

12
 CHAPTER 1. INTRODUCTION

1.8.2 Implemented work outputs

Figure 1.10: Checking the model whether recognizing or not.

Figure 1.11: Thumb finger for Left click.

13
 CHAPTER 1. INTRODUCTION

Figure 1.12: Index and pinky fingers for scroll down.

Figure 1.13: Index finger.

14
 CHAPTER 1. INTRODUCTION

1.9 Virtual Mouse Applications

1.9.1 Applications using Autopy

Dragging of the mouse

• Measure the finger position on the screen and add it to the current cursor position.

• Verify that the index finger is up before updating the cursor’s location to the next one and
leaving the previous one empty.

• Upgrade the current position of curser frequently as the curser is always moving as the index
finger is up.

• For this we used the autopy.mouse.move().

Left Click on

• Get the finger location, scale it to the screen position, and the motion that is
being used right now.

• Make that the left click function is executed at the current cursor position if only the thumb
finger is raised.

• Keep modernize the curser position and left click function is to be operated.

• For this used the autopy.mouse.click().

1.9.2 Applications using Pyautogui

Scroll up

• We update the cursor position when we see that the index and middle fingers are up.

• Scroll function is kept to positive number.

• Then the monitor screen is performs the scroll up.

• For this we used the pyautogui.scroll().

Scroll down

• Find the index and pinky fingers are up then the update the current curser position.

• Scroll function is kept to negative number.

15
 CHAPTER 1. INTRODUCTION

• Then the monitor screen is performs the scroll down.

• For this we used the pyautogui.scroll().

1.10 ALGORITHM
1. Taking image or video as input through the webcam.

2. Now the human hand is detected.

3. Giving the fingers tip id’s and skeleton or mathematical representation of hand

4. Recognizing the hand gesture from the fingers positions (up and down).

5. According to the position of fingers we found, it will be linked to the cursor position.

6. Performs the operations left click, moving and scrolling in the system.

1.11 TOOLS AND TECHNOLOGY

The tools and technology that were used in the implementation of this model are

1.11.1 Software Tools

1. Open CV

2. Numpy

3. Media pipe

4. Autopy

5. Pyautogui

6. Python

7. pycham / Jupiter

1.11.2 Technology

• Machine Learning

16
Chapter 2

Literature Survey

2.1 Non geometric characteristics are used in a multivariate Gaussian

distribution to identify hand movements Hasan, et aI 2020

The acquired hand image is segmented using two separate techniques: clustering-based thresholding
techniques and skin color-based segmentation using the HSV color model. In order to extract hand
features, a few functions are used to capture the shape of the hand. The alter Direction Analysis
Algorithm is then adopted to establish a relationship between statistical parameters (variance and
covariance) from the data, and it is then used to compute the slope and trend of an object (the
hand), by determining the direction of the hand gesture. The segmented image is given a Gaussian
distribution, and it follows the hand’s movement.

2.2 Computational Gaussian distribution applied on the segmented image

Hasan; et al 2020.
Form the result Function of Gauss To eliminate the rotating impact, the image has been divided into
circular areas, or domains that are formed in a terrace shape. The shape is divided into 11 terraces,
each of which has a width of 0.1. The 0.1 width division results in 9 terraces, numbered 1 through 9,
0.9 to 0.8, 0.7 to 0.6, 0.6 to 0.5, 0.5 to 0.4, 0.4 to 0.3, 0.3 to 0.2, and 0.2 to 0.1, plus one terrace with
a value of less than 0.1 and the last terrace for the outside region that stretched beyond the outer
terrace. This division’s explanation is illustrated. Terraces split with a 0.1 probability. Every terrace
is divided into 8 sections known as the feature regions, and empirical research reveals that number
8 is the largest.

17
 CHAPTER 2. LITERATURE SURVEY

2.3 Recognizing mouse click and movement eventss.shetty et al 2015.

A virtual mouse system was created by S. Shetty and colleagues utilizing color detection. They made
use of the built-in OpenCV functions to leverage the webcam to detect mouse cursor movement and
click events. In addition, a java-written mouse driver is needed. This system doesn’t function well
in challenging environments.

2.4 Expansion of mouse-free cursor control technique P.C.Shindhe et al

P. C. Shindhe et al. improved a technique for mouse cursor free control in which hand fingers are
used to direct the mouse cursor. Through camera 8, they have gathered various hand movements
utilizing color detection techniques. This method makes use of the built-in functionality of Image
Processing Toolbox and a Java mouse driver. The cursor was quite sensitive to motion, so the
pointer was not particularly effective in the air.

2.5 Controlling mouse pointer using webcam G. Sahu et al.; D. Gupta et al.
Gupta, D., et al.A system for manipulating the mouse pointer with a webcam was created by G. Sahu
et al. It can make or finish phone calls and control the volume of a media player and powerpoint
slides. The user’s finger was recognized using RGB color tapes.

2.6 Create and implement a hand-based virtual mouse.

presented a method for creating a hand-gesture-based virtual mouse. They executed mouse activities
in accordance with the various gestures they saw on the webcam. This method produced 77.

2.7 Interaction between human and machine using hand gestures recognition

(cheng,zhang,wang,2004)
Hand gesture recognition techniques for a human machine interaction uses sensors like depth
cameras or even gloves equipped with sensors to capture hand movements depth information is
praticularly useful for capturing 3D hand gesture’s and some for feature extraction like hand contour
and hand shape etc...and also neural networks like convolutional neural network,recurrent networks
are also used for identifying hand motions.

18
 CHAPTER 2. LITERATURE SURVEY

2.8 Recents improvements in hand gestures recognition for human computer

interaction (rajesith, li, zhang, niu,2008)

By using multiple sensors like gyroscopes and accelerometers which enhances robustness of hand
gestures recognition machines and by making variations in lighting conditions and different shapes
and sizes increase improvements in gestures recognition machines.

2.9 Hand Posture and Gesture Recognition Techniques and

Technology(Czuszynski,Murad,2009)
a wide range of hand posture and gesture identification methods and tools that shed light on both
their uses and limitations.Hand gesture detection relies heavily on image-based methods. Traditional
computer vision algorithms were used in early techniques, but more recent improvements have made
use of deep learning models, particularly Convolutional Neural Networks . With the use of these
models, diverse postures and movements may be accurately recognized by automatically extracting
hierarchical information from hand photos.A smooth interface between humans and computers
requires real-time gesture recognition. A lot of work is still being done to optimize algorithms and
models for low-latency processing, which will allow for useful applications in interactive systems
like gaming and virtual reality.

2.10 Techniques for Hand Gesture Recognition in Virtual Reality(Mitra,

Acharya,2012)
In order to improve engagement in virtual reality (VR) environments, hand gesture detection is
essential. This technology gives users a more immersive and organic manner to interact with digital
material. Effective hand gesture recognition in VR is made possible by a variety of methods and
technology.For realistic interactions in virtual reality, it is essential to estimate the hand’s
three-dimensional posture. The spatial arrangement of the hand joints is predicted using methods
like direct regression approaches or model-based pose estimation.A flawless VR experience
requires real-time processing. To accomplish low-latency hand gesture recognition and reaction,
optimization approaches including model simplification and parallel processing are used.Enhancing
the impression of presence and interaction through visual feedback in VR includes showing virtual
hands that replicate the user’s actual hand movements.Users can sense tactile sensations in reaction
to their gestures by integrating haptic feedback devices, which further improves the immersive

19
 CHAPTER 2. LITERATURE SURVEY

experience.Convolutional neural networks and recurrent neural networks, in particular, have

demonstrated notable effectiveness in hand gesture detection using deep learning approaches.
While RNNs effectively capture the temporal connections in gesture sequences, CNNs are effective
at extracting spatial characteristics from hand images.Data about hand movements and orientation
can be obtained using wearable sensors such as inertial measurement units and other devices
attached to the user’s hands. Using this information for gesture recognition provides a portable and
adaptable solution.

2.11 Recognition of Hand Gestures Based on Vision(Rautaray,

Agrawal,2007)
Computer vision techniques are used to interpret and comprehend hand movements and
configurations from visual input, which is commonly gathered by cameras, in vision-based hand
gesture identification. This method is frequently utilized in a variety of applications, including
virtual reality and human-computer interaction.Recognizing dynamic motions requires constant
hand tracking between frames. The position and motion of the hand are estimated by tracking
algorithms. For reliable tracking, methods like Kalman filtering or particle filtering are frequently
utilized.Once trained, the model may be used to recognize hand movements in real time. Based on
the recognized patterns, the system divides the input motions into predetermined categories.3D
hand pose estimation is used in applications that need three-dimensional data. This entails figuring
out how the hand joints are arranged spatially in three dimensions. Accurate pose estimation is
made possible by depth data from cameras or specialized sensors.For responsive interactions,
real-time processing that is accomplished through parallelism and optimization is essential. The
system associates recognized motions with certain application activities and provides feedback via
visual, aural, or haptic means. A seamless and immersive user experience depends on robustness to
external elements, adaptability to various settings, and intuitive user mappings. Current study
examines developments in sensor fusion, deep learning, and practical applications in several
fields.The system’s primary function, gesture recognition, categorizes input motions into
predetermined groups. The seamless integration of these detected gestures into apps is ensured by
mapping them to certain actions. The user experience is improved when feedback is given, such as
visual representation or haptic responses. For real-world applications, resilience to environmental
changes, a variety of hand forms, and various illumination situations are essential. The limits of
vision-based hand gesture identification are being pushed by ongoing research into cutting-edge

20
 CHAPTER 2. LITERATURE SURVEY

methodologies like deep learning and multi-modal sensor fusion in industries ranging from gaming
to healthcare.

2.12 Some gestures are unimanual some gestures are bimanual symmetric

Bourdot et al. (2010)

Unimanual gestures, in which a single hand is moved or positioned, are frequently employed to
accentuate or supplement verbal communication.These motions draw attention to a topic or assertion.
To underline the finality of a choice, one individual might, for instance, execute a chopping motion
in the air with just one hand.Unimanual gestures can be used to indicate an object’s form, size, or
motion. For example, pointing is a basic unimanual gesture used to draw attention.Single-handed
gestures can control the conversation’s pace by signaling when it’s one’s turn to talk or when
someone else should continue.These hand motions serve to visually express an item or action.
Using the thumb and fingers to make a circle, for instance, to denote that something is finished or
ideal.Coordinated movements of both hands are utilized in bimanual symmetric gestures, which are
frequently used to indicate more complicated meanings or activities. The communication might
have additional layers of significance by using gestures that are symmetrical or mirror images
of one another.To send a cohesive message, both hands move synchronously and simultaneously.
Clapping your hands together, for instance, can convey approval or admiration.Both hands move
synchronously and simultaneously to convey a single message. For example, clapping your hands
together can express admiration or approval.Bimanual symmetric gestures are also employed to
represent equilibrium or balance. As an illustration, keeping both palms parallel to one another can
convey steadiness or impartiality.These symmetrical motions feature a series of movements from
both hands. This could serve as an example of a narrative or step-by-step approach.It’s important
to remember that gestures, whether unimanual or bimanual, can be interpreted very differently
depending on the culture. What one culture views as a kind deed could imply something very
different to another.

21
 CHAPTER 2. LITERATURE SURVEY

2.13 Gestures are categorized in time as either static or dynamic.(D.Gupta,

G.Sahu, 2006)
A crucial component of gesture recognition is temporal classification, which distinguishes between
static and dynamic gestures based on the speed and style of the hand movements. Understanding the
temporal properties of gestures is crucial for the development of more precise and context-sensitive
recognition systems.Static gestures involve a consistent hand arrangement that is maintained over
time with little to no movement. The hand conveys information without the need for dynamic
changes by remaining in a specific posture or shape.The thumbs-up sign, pointing, and making
specific hand shapes that correspond to letters or symbols in sign language are examples of common
static gestures.In human-computer interaction scenarios, commands are frequently conveyed through
static gestures. For example, a hand held in a particular position in front of a camera can be
connected to particular orders or activities.In contrast, dynamic gestures feature constant, changing
hand movements. The ability to recognize these gestures depends on being able to record the
temporal evolution of hand configurations.Dynamic gestures include waving, swiping, and making
circular motions with the hand. Gestures that shift from one position to another are referred to be
dynamic gestures in sign language.In order to develop more flexible and expressive gesture detection
systems, researchers frequently investigate hybrid techniques that blend static and dynamic aspects.
To fully comprehend the gesture, one must take into account both the initial hand configuration and
the subsequent dynamic movements.

2.14 Contextual classification(Quek, 1995)

In order to improve the precision and utility of gesture interpretation, contextual categorization in
hand gesture recognition takes the surrounding situations and environmental aspects into account.
Systems can better grasp user intent and adjust to different settings by taking into consideration the
context in which gestures occur.Systems for recognizing hand gestures benefit from an
understanding of the surroundings. The interpretation of motions can be impacted by elements
including illumination, background clutter, and the presence of other objects. Context-aware
systems modify their recognition algorithms to take into account these external factors, ensuring
consistent performance in a variety of environments.It is critical to understand that various users
may utilize different gestures for the same command or phrase. Contextual classification takes into
consideration user-specific variances, enabling customized gesture models that take into account
unique preferences and quirks.

22
 CHAPTER 2. LITERATURE SURVEY

2.15 Use a neural network method to identify American Sign Language’s

static posture Kulkarni

In order to separate the hand from a uniform background, the input image is transformed into the
HSV color model, shrunk to 80x64, and then various image preprocessing operations are conducted.
The features are retrieved using the Hough method and histogram technique. Feed more. To classify
gestures, neural networks with three layers are used. The system achieved 92.78 using the MATLAB
language when 8 samples were used for each set of 26 sign language characters, 5 samples were
used for training, and 3 samples were used for testing.

2.16 Normalization based on brightness factor matching for gesture

recognition
With the background being black, the input image is segmented using a thresholding technique. In
order to adjust the coordinates to match the centroid of the hand object at the origin of the X and Y
axes, any segmented picture is normalized and its center mass is computed.A scaled normalization
process is used to address this issue and maintain image dimensions because this method, which is
based on the central mass of the object, generates images with varying dimensions. Two methods
are utilized to extract the features: first, edge matrices; and second, normalized features, which
measure only the brightness rates of pixels and exclude other black pixels in order to shorten the
feature vector. There are six different gestures in the database, each with ten examples (5 for training
and 5 for assessment).

2.17 Boundary histogram-based postures that are rotation-invariant Wysoki

el at
After the input image is captured by the camera and a skin color detection filter is applied, the
clustering procedure is utilized to locate the boundaries of each group in the clustered image using
a standard contour-tracking technique. Neural Networks MLP and Dynamic Programming DP
matching were employed in the grouping process. A color segmentation method based on the skin
color filter was used to identify the hand region, and the hand form morphology was found using
the (SGONG) network. The number of raised fingers and other aspects of the hand’s shape were
determined by the finger identification procedure, which was utilized to extract the features.

23
Chapter 3

Conclusion

Processing rates have significantly grown in the modern, digital world, and current computers are
now capable of assisting people in challenging jobs. However, coding technologies appear to
significantly impede the completion of a small number of activities, utilizing the resources at hand
inefficiently, and limiting the expressiveness of program usage. Here, gesture recognition can be
helpful. To attain interactivity and usability, computer vision techniques for human gesture
association must outperform present performance in terms of robustness and speed.

This project’s goal was to develop a system that could identify a variety of hand gestures in real
time and use that knowledge to prosecute the movements in the right situations for our application.

24
Chapter 4

Future Work

Two-handed 3D: Using several cameras, it would be feasible to track the gestures made by both
hands while they are both in the frame. To recognize a gesture indicated by a partially obscured
hand, a process would need to be created. It would be significantly more difficult to carry out this
process.

Because in order to recognize the gestures, we must process multiple frames at once from
various cameras. These motions could be used with sign language if we put them to use.

25
References

1. J. Katona, —A review of human–computer interaction and virtual reality research fields in

cognitive InfoCommunica- tions,|| Applied Sciences, vol. 11, no. 6, P. 26462, 2021.

2. D. L. Quam, —Gesture recognition with a DataGlove,|| IEEE Conference on Aerospace and

Electronics, vol. 2, pp. 755–760, 1990.

3. D.-H. Liou, D.Lee, and C.-C.Hsieh, —A real time gesture control virtual mouse using motion
history image,|| in Proceedings of the 2010 2nd International Conference on Signal Processing
System, July 2010.

4. S.U.Dudhane, —Cursor control system using hand gesture recognition,|| IJARCCE, vol. 2, no.
5, 2013.

5. K.P.Vinay, —Cursor control using gesture control virtual mouse,|| International Journal of
Critical Accounting, vol. 0975–8887, 2016.

6. L.Thomas, —Gesture control virtual mouse,|| International Research Journal of Engineering

and Technology (IRJET, vol. 5, no. 4, 2018.

7. P. Nandhini, J. Jaya, and J. George, —Computer vision system for food quality evaluation—a
review,|| in Proceedings of the 2013 International Conference on Current Trends in Engineering
and Technology (ICCTET), PP. 85–87, C

8. J. Jaya and K.Thanushkodi, —Implementation of certain system for medical image diagnosis,||
European Journal of Scientific Research, vol. 53, no. 4, pp. 561–567, 2011.

9. P.Nandhini and J.Jaya, —Image segmentation for food quality evaluation using computer
vision system,|| International Journal of Engineering Research and Applications, vol. 4, no. 2,
pp. 1– 3, 2014.

26
 CHAPTER 4. FUTURE WORK

10. J.Jaya and K.Thanushkodi, —Implementation of classification system for medical images,||
European Journal of Scientific Research, vol. 53, no. 4, pp. 561–569, 2011.

11. J. T. Camillo Lugaresi, —MediaPipe: A Framework for Building Perception Pipelines,|| 2019
https://arxiv.org/abs/1906.08172.

12. Google, MP, https://ai.googleblog.com/2019/08/on-device real time hand tracking with html.

13. V. Bazarevsky and G. R. Fan Zhang. On-Device, MediaPipe for Real-Time Hand Tracking.

14. K.Pulli, A.Baksheev, K.Kornyakov, and V.Eruhimov, —Realtime computer vision with
openCV,|| Queue, vol. 10, no. 4, pp. 40–56, 2012.

15. https://www.tutorialspoint.com〉 opencv.

16. D.-S. Tran, N.-H. Ho, H.-J. Yang, S.-H. Kim, and G. S. Lee, —Real-time virtual mouse
system using RGB-D images and fingertip detection,|| Multimedia Tools and Applications
Multimedia Tools and Applications, vol. 80, no. 7, pp. 10473–10490, 2021.

17. A. Haria, A. Subramanian, N. Asokkumar, S. Poddar, and

18. J. S.Nayak, —Hand gesture recognition for human computer interaction,|| Procedia Computer
Science, vol. 115, pp. 367– 374, 2017.

19. K. H. Shibly, S. Kumar Dey, M.A.Islam, and S.Iftekhar Showrav, —Design and development
of hand gesture based virtual mouse,|| in Proceedings of the 2019 1st International Conference
on Advances in Science, Engineering and Robotics Technology (ICASERT), pp. 1–5, Dhaka,
Bangladesh, May 2019.

27