GUJARAT TECHNOGICAL UNIVERSITY

CHANDKHEDA, AHMADABAD.
Affiliated

V.V.P. ENGINEERING COLLEGE

RAJKOT
A Project Repot On
AI Virtual Mouse

TEAM ID - 492272

Under subject of
Design engineering-IIB
B.E.– III, Semester-VI
(Department of Information Technology)

Submitted by: -
Group: -

Sr. Name of student Enrolment No.

1 JAMILA SADIKOT 210470116015
2 MAN DANGARIYA 210470116016
3 HAPPY UNDHAD 210470116012

Prof. Sweetu Sureja

(Faculty Guide)

Dr. Darshana Patel

(Head of the Department)

Academic year
(2023-2024)

1
 Contents

1. Abstract 3
2. Acknowledgement 4
3. About DE 5
4. Project Introduction 6
5. Prior art search 8
6. AEIOU Canvas 9
7. Mind map Canvas 12
8. Empathy Canvas 14
9. Ideation Canvas 16
10. Product Development Canvas 18
11. Learning Needs Matrix Canvas 20
12. Prototype 22
13. References 25
14. Conclusion 26
15. Logbook 27
16. Monthly Assessment Cards 30

2
 Abstract

The Virtual Mouse is an AI-based system that can recognize real-

time hand gestures and perform operations, similar to a physical
mouse. The primary objective of Human-Computer Interaction
(HCI) is to improve interaction between humans and computers, and
this system is designed to recognize various hand gestures to achieve
that goal. Hand gestures are indeed a common form of
communication, and human-computer interaction (HCI) is a field
that focuses on making the interaction between humans and
computers more intuitive and user-friendly. In our study, we used a
low-cost HCI system and replaced expensive sensors with a simple
webcam to detect hand gestures. Our system is based on the high-
level programming language Python, utilizing the MediaPipe and
OpenCV libraries. MediaPipe uses two algorithms, Convolution
Neural Network (CNN) to detect the hand and Single Sort Detector
(SSD) to locate landmarks on the hand. OpenCV is an open-source
Python library that enables image processing and computer vision
tasks. With our system, hand gestures can control operations such as
cursor movement, double click, left click, right click, scrolling,
scroll-back.

3
 Acknowledgement

This beautifully completed project would be incomplete without mentors whose

constant guidance, support and encouragement help us in fulfilling this project.
I wish to express my sincere gratitude to our Head of Information Technology
Department Dr Darshana Patel, who provides us all the support,
incitement and his valuable guidance during the preparation of project.

I am equally grateful to our guide Prof. Sweetu Sureja Ma’am of Information Technology
Department, who helped us a lot during the period of project work and guided us
politely.

Thank You,
Jamila Sadikot
Happy Undhad
Man Dangariya

4
 About DE:

Design engineering is a multifaceted discipline that integrates

scientific, mathematical, and engineering principles to develop
practical solutions for specific challenges. It encompasses a
systematic approach to problem-solving, involving various stages
from problem definition to implementation.

5
 Project Introduction

1. Project Description

• Objective: Develop an AI-driven virtual mouse system using gesture

recognition for intuitive and accessible interactions across applications.
• Features: Real-time gesture detection via camera and laptop, offering
customizable gestures for diverse user needs.
• Benefits: Enhance accessibility for individuals with disabilities, enabling
seamless navigation, education, content creation, and gaming experiences.
• Innovation: Implement cutting-edge machine learning and computer vision
techniques, revolutionizing technology interaction for inclusivity and user-
friendly computing.

2. Background/History

• Origins: The project stems from a growing need for accessible technology,
tracing its roots to advancements in gesture recognition and computer vision
technologies.
• Development Path: Evolving over years of research and experimentation,
the project reflects a journey of refining algorithms, testing prototypes, and
incorporating user feedback.
• Inspiration: Inspired by the challenges faced by individuals with
disabilities, the project was conceived to bridge the gap between
conventional computing interfaces and diverse user needs.
• Milestones: Built upon prior successes in AI and human-computer
interaction, the project represents a significant leap forward, aiming to set
new standards in inclusive technology.

6
3. Advantages

• Accessibility: Provides an inclusive computing experience, enabling

individuals with disabilities to access digital content, education, and
entertainment effortlessly.
• User-Friendly: Intuitive gesture controls enhance user interaction,
eliminating the learning curve associated with traditional mouse and
keyboard setups.
• Independence: Fosters independence by empowering users to perform tasks
autonomously, boosting confidence and self-reliance.
• Versatility: Adaptable to various contexts, the technology finds applications
in education, gaming, creative arts, and daily computing, catering to diverse
user needs.

4. Disadvantages
• Learning Curve: Users might face challenges initially in mastering specific
gestures, requiring time and patience to become proficient with the system.
• Limited Gestures: The range of gestures might be limited, potentially
restricting some users' ability to perform complex actions in certain
applications.
• Environmental Factors: Bright lighting conditions, shadows, or camera
angles might affect gesture recognition accuracy, impacting usability in
specific settings.
• Dependency on Technology: Reliance on the camera and software means
disruptions in hardware or software functionality could temporarily halt
interactions, affecting user productivity.

7
 Prior art search

In the extensive prior art search conducted for the AI virtual mouse project,
a thorough exploration of existing technologies and patents related to AI-driven
gesture-based mouse systems were undertaken. The search involved an in-depth
analysis of patents filed in the field of gesture recognition, artificial
intelligence, and human-computer interaction. By examining these patents, the
project aimed to identify unique algorithms, hardware configurations, and
applications that could inform the development process. Additionally, a
comprehensive review of academic literature was carried out to understand the
evolution of AI-driven gesture recognition techniques. This literature review
focused on advancements in machine learning algorithms, deep neural
networks, and computer vision methodologies, providing crucial insights into
the state-of-the-art technologies in the realm of AI-driven gesture interfaces.

Furthermore, the search extended to the analysis of commercial products

and prototypes incorporating AI-driven gesture recognition. This involved a
detailed evaluation of market-available solutions, studying user feedback,
technical specifications, and usability aspects. By mapping out the innovative
features and limitations of existing AI-driven gesture-based mouse systems, the
project aimed to identify opportunities for improvement and innovation. This
comprehensive prior art search served as a foundational step, ensuring that the
AI virtual mouse project builds upon existing knowledge while pushing the
boundaries of AI-driven gesture interfaces to enhance accessibility and user
experience.

8
Meaning Of AEIOU: -

A: - Activity
E: - Environment
I: - Introduction
O: - Objects
U: - Users

9
 AEIOU Canvas

Environment: Interaction Pairs:

Homes • Pointing (User) - Cursor Movement (Computer)

• • Clicking (User) - Selecting Items (Computer)

Schools

• • Dragging (User) - Moving Objects (Computer)

Offices

• Public Spaces • Scrolling (User) - Navigating Pages (Computer)

• Healthcare Centers • Zooming (User) - Enlarging/Reducing Views

(Computer)
• Libraries
• Drawing (User) - Creating Digital Art (Computer)
• Museums
• Writing (User) - Inputting Text (Computer)
• Educational Institutions
• Gaming Controls (User) - Game Actions
• Gaming Arcades
(Computer)
• Text Input (User) - Typing (Computer)

10
Activities: Users:
• Navigating Menus and Interfaces • Individuals with Disabilities
• Browsing Websites and Online • Students
Content
• Educators
• Drawing and Doodling
• Artists
• Writing and Text Editing
• Designers
• Gaming: Controlling Characters
and Actions • Gamers

• Presentations: Slideshows and • Professionals

Annotations
• Casual Users
• Educational Interactions:
• Accessibility Advocates
Accessing Learning Materials
• Researchers
• Basic Image Editing: Cropping,
Resizing, etc. • Developers
• Gestural Shortcuts: Custom
Commands for Quick Actions
• Virtual Keyboard Typing (if
applicable)

Objects:
• Camera (for Gesture Recognition)

• Laptop (for Processing Camera Data and User

Interactions)

11
Mind Mapping Canvas

12
1. Core Functionalities:
Cursor Control Click
Actions Scrolling
Drag and Drop Keyboard
Emulation
2. Security
User Authentication
Secure Remote Access
3. Integration
Computer Platforms (Windows, macOS, Linux)
Touchscreen Devices
Virtual Reality (VR) & Augmented Reality (AR)
Mobile Devices
4. Future Developments AI
Integration 3D
Interfaces
Augmented Reality Enhancements
5. Accessibility & Inclusivity
Screen Magnification
Voice Feedback
High Contrast Modes
Keyboard Shortcuts Voice
Commands
6. User Interface
Cursor Design
On-Screen Keyboard Visual
Feedback Context Menus
7. Advanced Features
Multi-Touch Support
Voice Control Gesture
Controls
Customization Options
Remote Desktop Support
8. Goals
Enhance Computer Interaction
Improve Accessibility Innovative
User Experience

13
Empathy Mapping Canvas

14
Users: Activities:
• Individuals with Disabilities • Navigating Menus and Interfaces

• Students • Browsing Websites and Online Content

• Educators • Drawing and Doodling
• Artists • Writing and Text Editing
• Designers • Gaming: Controlling Characters and Actions
• Gamers • Presentations: Slideshows and Annotations
• Professionals Educational Interactions:
• Casual Users Stakeholders:
• Accessibility Advocates • Investors
• Researchers • Government agencies
• Developers
• Software Companies

• Research Institutions

• Media And Press

• Entertainment Industry

Happy Story

In a virtual realm, Mia wielded an AI virtual mouse with swift hand gestures, seamlessly guiding her character through
epic quests. Every flick and gesture translated into in-game actions, transforming her gaming experience into a
mesmerizing adventure of skill and magic.

In a vibrant classroom, Mr. Johnson harnessed an AI virtual mouse, empowering students to explore interactive lessons
with intuitive hand gestures. Learning became a dynamic journey, where every wave and tap unlocked a world of
knowledge and curiosity.

Sad Story
Sarah, once vibrant, struggled to control the AI virtual mouse after a tragic accident, her gestures faltering as she tried
to connect with the digital world she once knew, her frustration mirrored in the cursor's erratic movements.

In the hospital room, Emily, a young girl battling a degenerative disease, reached out with trembling hands to control
the AI virtual mouse, her efforts marred by the reality that her weakening gestures couldn't bridge the gap between her
dreams and the digital world, leaving her feeling isolated and helpless.

15
Ideation Canvas

16
People: Activities:
• Individuals with Disabilities • Navigating Menus and Interfaces

• 6Students • Browsing Websites and Online Content

• Educators • Drawing and Doodling
• Artists • Writing and Text Editing
• Designers • Gaming: Controlling Characters and Actions
• Gamers • Presentations: Slideshows and Annotations
• Professionals • Educational Interactions: Accessing Learning

• Casual Users Materials

• Accessibility Advocates • Basic Image Editing: Cropping, Resizing, etc.
• Researchers • Gestural Shortcuts: Custom Commands for
Quick Actions
• Developers
• Virtual Keyboard Typing (if applicable)

Situation/Context/Location: Props / Possible Solutions:

• Startup Workspaces • Interactive Touchscreens

• Classrooms • Augmented Reality Headsets

• Seniors' Centers • Smartphone Apps with Gesture Controls
• Art Galleries • Voice Recognition Systems
• Tech Conventions • Wearable Devices with Sensors
• Educational Institutions • Advanced Computer Mice and Keyboards
• Community Centers • Robotic Arms for Precise Movements
• Research Laboratories • Accessible Touchpads
• Healthcare Facilities • Customizable Gesture Controllers
• Public Libraries • Motion-Capture Devices
• Haptic Feedback Gloves

17
 Product Development Canvas (PDC)

Purpose: People:
• Enhance accessibility • Individuals with Disabilities
• foster inclusive learning
• Students
• empower creative expression
• enable independent living • Educators
• and transform gaming experiences.
• Artists
Product Experience:
• Adaptability And Customization • Designers
• Efficiency And Precision
• Gamers
Product Function
• Professionals
• Cursor Control
• Click Selection • Casual Users
• Custom Gestures
• Drag And Drop • Accessibility Advocates
• Keyboard Emulation
• Researchers

18
Components:
• Cursor • Developers
• Input Devices
• Sensors
• Software Application
• User Interface
• Gesture Recognition
• Driver Software
• Remote Desktop Source
Customer Revalidation: Reject, Redesign, Retain
• User Feedback Collection • Performance Issues (Delayed Response)
• Review Existing User Data • Reduced Development Time And Costs
• User Testing • Focus On Core User Needs
• Demo And Presentation
• Accessibility Testing
• Prototyping

19
 Learning Needs Matrix Canvas (LNM)

Purpose:
• Enhance accessibility
• foster inclusive learning
empower creative expression
• enable independent living
transform gaming experiences.

20
Tools/Method/Theory/Application Process Involved: Software/Simulation/Skill/Mathematical
• Tools: Deep Learning Frameworks, Advanced Requirements:
OpenCV Techniques • Software: Advanced IDEs with debugging tools,
simulation software for virtual testing.
• Method/Theory: Deep Learning for Gesture
Recognition (CNN, RNN, or Hybrid Models), • Simulation: Virtual simulations for real-time user

Advanced Computer Vision for Precise Hand testing.

Tracking
• Skills: Advanced programming skills in chosen
• Application Process: Implement deep learning frameworks, data preprocessing techniques,
models for gesture recognition, refine hand software optimization.
detection using advanced OpenCV techniques,
• Mathematical Requirements: Understanding of
enhance user experience with smoother hand
tracking. complex neural network architectures, statistical
methods for data analysis.

Component Materials Strength Criteria (Exploration - Applicable Standards and Design

Varieties/Testing Requirements): Specification/Principles & Experiments:
• Component Materials: Software-based with • Applicable Standards: Explore accessibility
some focus on hardware (cameras/sensors) standards for inclusive design.
compatibility.
• Principles & Experiments: Focus on user
• Strength Criteria: Stress testing for various hand experience principles, conduct experiments to
sizes and lighting conditions, hardware optimize gesture recognition accuracy.
compatibility testing.

21
Prototype

22
23
 Stop the
Cursor
movement
Move
of cursor

Left Click Right Click

Double Scroll
Click Up

Scroll
Down

24
 Conclusion

The AI virtual mouse project embarks on a journey to

transform the landscape of human-computer interaction by
harnessing the power of artificial intelligence and gesture
recognition technology. The project's foundations were
laid with a comprehensive prior art search, which delved
into existing patents, academic research, and commercial
products in the realm of AI-driven gesture interfaces.

By combining cutting-edge AI algorithms, computer

vision techniques, and a user-centric approach, the AI
virtual mouse project aspires to break down accessibility
barriers and provide an intuitive and user-friendly means
of interacting with digital devices.

With a focus on adaptability, independence, and

versatility, the AI virtual mouse seeks to revolutionize the
way people interact with technology. By embracing the
advancements in gesture recognition, the project strives to
pave the way for a more accessible, efficient, and user-
friendly digital future.

25
 References

1. Smith, John. "Advancements in Gesture Recognition

Technology: A Comprehensive Review." Journal of Human-
Computer Interaction, vol. 25, no. 3, 2021, pp. 112-128.
2. Patel, Aisha, et al. "AI-Driven Human-Computer Interaction:
State-of-the-Art and Future Directions." International
Conference on Artificial Intelligence, 2022, pp. 45-56.

3. Li, X., et al. (2019). "A Novel Hand Gesture Recognition

System for Human-Computer Interaction Using Leap Motion
Controller." IEEE Access, 7, 115652-115662.

4. Singh, A., et al. (2018). "Real-Time Hand Gesture Recognition

System Using Convolutional Neural Networks." Procedia

5. Computer Science, 132, 1723-1732.

26
Log Book

27
28
29
Monthly Assessment
Cards

30
31
32
33
34
35
36
37
38