Mini Project Presentation

On

Cursor movement by hand Gesture.

Chatrapati Shivaji Maharaj Institute of Technology

Date:-24/3/25

Presenter Guide
Vedika Telge prof. Priyanka
Vedant Patil
Gaurav Jagdhane
Gaurav Parker
 Outline
 Introduction

 Scope of the proposed study

 Literature Review

 Research Gaps

 Research Objective

 Experimental Setup

 Result

 Future Work

 References
 Introduction
With the development technologies in the areas of augmented reality and devices that we usein our daily life, these
devices are becoming compact in the form of Bluetooth or wireless technologies. an AI virtual mouse system that
makes use of the hand gestures and hand tip detection for performing mouse functions in the computer using
computer vision. The main objective of the proposed system is to perform computer mouse cursor functions and scroll
function using a web camera or a built-in camera in the computer instead of using a traditional mouse device. Hand
gesture and hand tip detection by using computer vision is used as a HCI with the computer. With the use of the AI
virtual mouse system, we can track the fingertip of the hand gesture by using a built-in camera or web camera and
perform the mouse cursor operations and scrolling function and also move the cursor with it.
While using a wireless or a Bluetooth mouse, some devices such as the mouse, the dongle to connect to the PC, and
also, a battery to power the mouse to operate are used, but in this paper,the user uses his/her built-in camera or a
webcam and uses his/her hand gestures to control the computer mouse operations. In the proposed system, the web
camera captures and then processes the frames that have been captured and then recognizes the various hand gestures
and hand tip gestures and then performs the particular mouse function.
Python programming language is used for developing the AI virtual mouse system, and also, OpenCV which is the
library for computer vision is used in the AI virtual mouse system. In the proposed AI virtual mouse system, the
model makes use of the MediaPipe package for the tracking of the hands and for tracking of the tip of the hands, and
also, Pynput, Autopy, and PyAutoGUI packages were used for moving around the window screen of the computer for
performing functions such as left click, right click, and scrolling functions. $e results of the proposed model showed
very high accuracy level, and the proposed model can work very well in real-world application with the use of a CPU
without the use of a GPU.
Scope of the proposed study
The scope of this project is to develop a virtual mouse that will be operated without touching any
device or screen. In today’s world where we are adjusting our living while being in a pandemic, a
touch less mouse controller will be useful to eliminate the risk of spreading infection through
touch on public service devices. A virtual mouse will be introduced soon to replace the physical
computer mouse to promote convenience while still allowing accurate interaction and control of
the computer system. The virtual mouse can be used without touching the screen. This project
can improve the scope of Human Computer Interaction technology to be explored more. To do
that, the software requires to be fast enough to capture and process every image, in order to
successfully track the user's gesture. Therefore, this project will develop a software application
with the aid of the latest software coding technique and the open-source computer vision library
also known as the OpenCV. The scope of the project is as below:
▪ Real time application.
▪ User friendly application.
▪ Removes the requirement of having a physical mouse.
Literature Review
There are numerous ways to control the cursor with a hand gesture, but for the longest period of
time, Data Glove must be worn. It lessens the effectiveness of the user's and the system's
performance. The system's complexity in this method is a major problem.
Adaptive skin color models and a motion history image-based hand moving direction detection
technique are implemented in a paper published by Dung-Hua Liou and Chen-Chiung Hsieh. The
average accuracy of this project was 94.1%, and processing takes 3.81 milliseconds per frame. The
primary problem with paper is that it has trouble recognizing more complex hand gestures when
used in a working environment.
This paper mainly applied visional hand gesture identification to the HCI interface, holding
control usage, written by Chang-Yi Kao and Chin-Shyurng Fahn. According to experimental
findings, the face tracking rate is over 97% under typical circumstances and over 94% when the
face has temporal occlusion. The system's execution efficiency is excellent, and we are inspired to
market the robot soon. High configuration computers are required for accurate results.
A Machine-user interface that performs hand gesture recognition using multimedia techniques and
basic computer vision. A paper was published on this topic by Ashwini M. Patil et al. Before
utilizing the gesture comparison algorithms, they discovered a significant limitation. From the -
stored frames, hand segmentation and skin pixels must be completed.
The primary goal of this research was to create a real-time hand gesture detection system based on
the skin color model, which was published by Angel et al. Since hand gestures may readily
communicate thoughts and activities, employing these different hand forms, when spot by the
gesture recognition system and processed to create related events, have the potential to give a
more natural interface to the computer vision system. However, it was unable to function in a
complex environment and was only calculable in proper lighting.
Object detection (OD), salient object detection (SOD), and category-specific object detection are
just a few of the object detection techniques that Ahmed, Muhammad, et al., explain in great detail
(COD). It examines various currently used deep learning methods and evaluates how well they
perform in difficult environments. By comparing their performance in terms of output, time
required, and comparison to more conventional methods, it illustrates recent research and
advancements in this field. Additionally, it discusses the publicly accessible datasets for this task
and explains the comparison evaluation metrics. It also provides a brief overview of the drawbacks
and the direction that more advancements might be made.
A detailed explanation of the algorithms and methodologies for the color detection of a virtual
mouse was given in this project by Kollipara Sai Varun et al. In this paper, Open CV (Open Source
Computer Version Library) is primarily used for video capture. The highlight color provided by
the user for mouse movement is used in this paper for color detection and mouse movement.
Models that are based on color detection and mouse movement based on highlighted colors
provided by the user were developed by K.S. Varun et al. It is possible to see a two-figure input
that creates two rectangles and an average point from both figures. It will function like the mouse
pointer. The mouse pointer in the runtime follows the moving point as it moves. Therefore, using
this, mouse movement can be implemented. The position of the predetermined colored caps in the
mask that is created for system comprehension determines how the mouse pointers are updated. In
order to detect the predetermined colored objects that will aid in mouse movement, the created
mask is converted from an RGB background to a black and white image and provided 84%
accuracy. If the predetermined colored caps blend in with the background, they won't be seen, and
no mouse movement will be possible.
A system that uses head and facial movements to control the mouse was proposed by T. Palleja et
al. It computes four motion matrices using an algorithm for image processing. The region of
interest is used to analyze the ten-frame cumulative image and find the movement. The process
takes some time, which slows down how quickly the mouse reacts.
Research gaps
• Accuracy and Precision Issues:- Existing models struggle with
detecting small, precise movements, leading to unintended
cursor movements.
• Latency and Real-Time Performance:-Many gesture recognition
systems have a delay between hand movement and cursor
response, affecting usability.
• Environmental Limitations:-Most systems perform well under
controlled conditions but may fail in poor lighting or cluttered
backgrounds.
• User Fatigue Due to Continuous Hand Movements:-Gesture-
based controls require users to keep their hands in the air for
extended periods, which can cause discomfort and fatigue. This
makes the system less practical for long working hours
compared to traditional input devices like a mouse or trackpad.
 Research Objective

• Intuitive Interaction: It provides a more intuitive and natural way to

interact with a computer, as it mimics real-world hand movements.
• Accessibility: It can be beneficial for individuals with disabilities who
may have difficulty using traditional input devices like mice or keyboards.
• Gesture Customization: Users can potentially customize gestures to
perform specific actions, which can improve efficiency.
• Gesture Recognition: With advanced technology, gesture recognition can
be quite accurate, reducing the need for physical hardware like a mouse or
touchpad.
• Gesture-Based Control: It can be used for controlling various applications,
from gaming todesign software, adding a new dimension of control.
FLOW DIAGRAM
 Algorithm
Step 1: Start.
Step 2: Webcam video is captured using c# video capture libraries.
Step 3: It is broken up into continuous image frames.
Step 4: Each frame is converted to grayscale.
Step 5: Each frame is now scanned to check five finger design patterns.
Step 6: Once detected it is flagged as an object.
Step 7: System now tracks its movement within frame.
Step 8: This is tracked in terms of x, y coordinates.
Step 9: These x, coordinates are now mapped onto mouse cursor positioning.
Step 10: End.
 Experimental Setup
• Hardware Requirement:
• System: Intel corei5
• Processor: Intel® Core ™ [email protected] i5 10300H
• System type: 64-bit OS, x64-based processor
Webcam: 720p HD Webcam Resolution: 1920 X 108
Installed RAM: 8.00 GB

• Software Requirement:
Front End: - Python.
Back End: - OpenCV, MediaPipe, Pynput,Autopy, and PyAutoGUI

• Testing Conditions:-
Lighting Conditions: Bright, dim, and natural light.
Different User Hands: Tested on different hand sizes and skin tones.
Testing Distance: Close-range vs. far-range hand detection.
Angle Variations: Different hand angles to test tracking accuracy.
RESULT
 Future work
There are still many improvements that can be made to our system like improving the performance of
the current system and adding features such as enlarging and shrinking windows, closing window, etc.
by using the palm and multiple fingers. The current system is variant to reflection and scale changes
and requires proper hand gestures, good illumination technology and powerful camera for the
performance of mouse functions. Precision can always be increased at the cost of recall by adding
more stages, but each successive stage takes twice as much time to find harder negative samples and
the applications which benefit from this technology. We present an image viewing application as an
example of where this technology could lead to a more natural User interface. Introduce gesture
recognition with both hands, enabling the creation of more natural interaction environments. Build
systems that are able to recognize continuous gestures, i.e., without the need to introduce pauses for
gesture or command construction.
The proposed AI virtual mouse has some limitations such as small decrease in accuracy of the right
click mouse function and also the model has some difficulties in executing clicking and dragging to
select the text. $ese are some of the limitations of the proposed AI virtual mouse system, and these
limitations will be overcome in our future work. Furthermore, the proposed method can be developed
to handle the keyboard functionalities along with the mouse functionalities virtually which is another
future scope of Human-Computer Interaction (HCI)
 REFERENCES
[1] Vijay kumar sharma “virtual mouse using hand gesture Gis science journal on Dec 29 2020

[2] Amardip Ghodichor, Binitha Chirakattu “Virtual Mouse using Hand Gesture and Color
Detection ”, Volume 128 – No.11, October 2015.

[3] AhemadSiddique, Abhishek Kommera, DivyaVarma, ” Simulation of Mouse using Image

Processing Via Convex Hull Method ”, Vol. 4, Issue 3, March 2016.

[4] Student, Department of Information Technology, PSG College of Technology,

Coimbatore, Tamilnadu, India,”Virtual Mouse Using Hand Gesture Recognition ”,Volume 5
Issue VII, July 2017.

[5] OpenCV Website – www.opencv.org

Thank You