Assistive Device Based on Machine Learning Approach for Communication of Visually Challenged and Muted Community
Assistive Device Based on Machine Learning Approach for Communication of Visually Challenged and Muted Community
Department of Electronics and Department of Electronics and Department of Medical and Electronics
Telecommunication Engineering Telecommunication Engineering Dayananda Sagar College of
Dayananda Sagar College of Dayananda Sagar College of Engineering, Bangalore, India
Engineering, Bangalore, India Engineering, Bangalore, India [email protected]
2023 4th International Conference on Smart Electronics and Communication (ICOSEC) | 979-8-3503-0088-8/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICOSEC58147.2023.10275924
[email protected] [email protected]
Abstract— It is a great challenge to find a means of built-in camera takes a picture of the printed text, which
communication for people who are suffering from visual or is then subsequently transformed into digital text by
hearing impairment and also for those who are speechless. This
research study aims to develop a device from Raspberry Pi
Google Vision API. The text is then translated to audio
which can communicate with the visually challenged by using the TTS (Text-to-Speech) module to produce output
converting the messages to audio. The proposed device also that sounds exactly like the original book or paper.
helps the people with hearing loss by converting audios to text
and display the same. It also helps the speech impaired by The vocally handicapped can benefit from recording voice
converting the sign language into text or audio by using image or audio, converting that data into text, and display for
to text conversions. them to read. The message may be typed on a monitor
Keywords— Raspberry P, Assistive Device, Deaf, KNN using a custom keyboard, and for those with vocal
Classifier, Sign Language, Braille impairments, the gadget will read it out. With the help of
the TTS (Text to Speech) library, this text is rendered as
I. INTRODUCTION speech, and the user's audio input is recovered in a
Globally, 1.5 billion people have some degree of vision synthesized voice [2]
impairment, of which 200 million have a mild condition,
According to the World Health Organization, there are 285
220 million have a moderate to severe condition, and 35
million blind people, 300 million people who are hard of
million are visually impaired. The most of blind individuals
hearing, and 1 million mute people in the world.
in the world are thought to reside in India. The number of
Communication is a common challenge for those who are
mute and deaf people worldwide is 9 billion. Almost 5% of
dumb, deaf, or blind in daily life. The primary subject of
the global population, or 466 million people, suffer from a
this essay will be the aforementioned fact. It aims to
hearing loss that is incapacitating [1]. Technology is always
develop a new technology that will make it easier for
evolving, and over the past few decades, it has improved
people who are blind, deaf, or dumb to interact with
our quality of life and ease. Yet, the group of people with
normal people in social situations
physical disabilities in our culture has not received enough
attention. They are not aware of scientific breakthroughs, II. OBJECTIVES
yet they nevertheless face a range of challenges every day.
Human existence is impossible without communication.
➢ To provide protracted communication methods
There is a gap, though. They dislike Braille and the sign
employing wireless technologies for people who
language they use to communicate. They are frequently are blind, deaf, or hard of hearing.
compelled to enhance their communication abilities or to
➢ To create a more durable and lightweight
rely on outside help, like another person. This paper's main
wearable device in the form of a glove by
goal is to close that gap by giving them the self-assurance attaching sensors that would capture and evaluate
and communication skills to interact with regular people. different hand movements made by a deaf-mute
This gadget is precise, efficient, and robust thanks to the person.
Raspberry Pi and Google API, which make up its two main ➢ To provide means of braille communication for
parts. The system consists of three main parts, a single one the visually impaired.
for each of the three impairments: visual, auditory, and
➢ To develop distinctive testing strategies
verbal. It makes use of a Raspberry Pi, which Google API
specifically for speaker dependent testing using
supports, together with a camera, microphone, speaker, and Deep Learning Techniques for Feature
screen. For people who are blind or visually challenged, the Extraction
➢ To develop an effective way of text to speech speech conversion, this method enables blind people to listen
conversion for information delivery by fusing it to audio. Speech inputted through the microphone for a person
with Artificial Intelligence. who is deaf is translated into text and shown in a pop-up
window on the user's screen. The screen-visible keyboard is
A. Rationale for taking up the project: used by dumb people to input text, which is then translated
• Key objective of the prototype is to enable efficient into voice and spoken to them through the speaker[8].
interaction among all three types of disabilities viz. This study aims to develop a clever method that will allow
visual, hearing and speech impaired as well as blind people to read braille and hear audio messages that are
normal individuals. converted from the text that has been delivered in both braille
and text forms. Reading text on a screen rather than listening
• 24 different types of input output combinations are to sounds is more comfortable for deaf people. The technology
made possible in this prototype by transmission and that recognizes sign language and converts it into text that is
shown on the screen and sounds through the speaker allows
reception of messages suiting to the disabled the dumb to communicate with non-dumb people. A small
individual’s state(s). instrument and the Raspberry Pi are both used in the
• The prototype is wireless which makes it highly development of this system. This device has a QWERTY
suitable for long distance communication. keyboard that is connected to an LCD display on one side and
• This project mainly focuses on aiding social cause a 3-cell braille display on the other. On a QWERTY keyboard,
a person with normal vision types something. A blind person
by enabling visual, hearing and speech impaired can now read the displayed text by placing their fingertips on
individuals a way to bridge the communication gap the Braille display. To reply, use the Braille input keys. The
among themselves and the society. sighted individual can read the message on the LCD[9].
Because to the advancements in science and innovation,
human existence has gotten better and easier. The World
III. LITERATURE SURVEY
Health Organization (WHO) estimates that there are 285
In order for impaired persons to communicate easily with million blind people, 300 million deaf mute people, and 1
other normal people or others of their type, the focus of this million dumb people in the globe. This prototype aims to offer
study will be on developing new technology. The goal is to these types of disabled people a communication channel. This
create technology that can assist people who are struggling gadget accepts input from flex sensors from a deaf-mute
with blindness, deafness, or speechlessness. Therefore, the person using a sensor glove that recognizes hand motions, text
Sharojan Bridge is created and is based on wearable input from a blind person using a Braille keypad, and input
technology, which allows the user to wear the device while from a normal person using a web application. All three of the
moving the system with ease. Here, communications between aforementioned disabled people are now able to interact with
disabled persons were transmitted using an Arduino circuit other people and one another effectively. The message result
board and Texas Instruments circuitry[5]. will be displayed on an LCD screen, the speech output is
There are three modules in this project: one for the blind, produced using a speaker, and the Braille output is produced
one for the deaf, and one for the dumb. For communication using four solenoid motors arranged in a manner that
with other people, blind persons will use the microphone in resembles braille characters.
the blind module. They also have an app for this where a blind IV. METHODOLOGY
person can communicate with a specific contact by using his
own gestures. Anyone can use the terminal to communicate This research focus on how the technology recognizes sign
with a deaf person in the deaf module. Anything they type is language and translates it into voice and text. The main layout
visible to everyone else on the terminal page[6]. of the device is shown in the accompanying block diagram.
The LCD display, speaker, SD card, and camera are all housed
People who are dumb, blind, or deaf cannot communicate on the Raspberry-Pi, which is the prototype's main
effectively with others. These impaired people's component. The camera records sign language for those who
communication is improved because of this method. The have vocal impairments, and this gadget converts it to voice
Bluetooth-enabled Arduino Board and flex sensor are and text. For those who are visually blind, the audio output
supposed to be used to communicate. Flex detectors are put through the speaker is useful, and for those who are auditory
on the gloves and then connected to the Arduino board so that deaf, the message is shown on the LCD module [20]. Figure1
they may flex in response to finger motion. The LCD and explains the block diagram of this research study. This work
Speaker modules are connected to the Arduino, which is concentrate on how sign language is recognized and translated
configured with the code to display a certain message on the into speech and text in this system. The following block
LCD if the sensor is off-centre and it outputs sound on the diagram depicts the general configuration of the gadget. The
Speaker module. The Arduino IDE is used to create the LCD display, speaker, SD card, and camera are all associated
application. By attaching a Bluetooth module to Arduino, the with the Raspberry-Pi, which serves as the prototype’s main
message is also communicated via the app on a smartphone component. The system tends to work for both vocal and
and is shown on an LCD. The current location is tracked by a visual challenged people since the camera captures the sign
GPS module and shown on an LCD screen[6] language used by the vocally impaired, the audio output is
The goal of this study is to offer a straightforward, speedy, made through the speaker, which is useful for the visually
accurate, and cost-effective solution. For people who are impaired, and the message shows up on the LCD module for
blind, deaf, and unable to speak, the project uses a Google API the audibly impaired individual.
and Raspberry Pi-based solution. Due to image- to-text and
This method allows for the identification of numerous Step 2: Based on the quantity and positioning of the hand's
hand gesture patterns. Thus, the message can be heard over the edges, the image is compared to the pre-trained code, and the
speaker by the blind, and it can be seen over the LCD by the message is shown on the LCD.
deaf. Additionally, the microphone is used to record any user's Step 3: The collected text is then converted to speech using
voice, which is subsequently converted into text using an the gates API.
algorithm and shown on an LCD to help deaf persons
understand the intended message. Step 4: The Raspberry Pi is linked to a high-quality
speaker, which outputs the audio, allowing a blind person to
decipher the message by listening to it.
For deaf:
Step 1: The USB microphone attached to the Raspberry Pi
in the device captures the sound or words being spoken and
saves them as an mp3 file for the user, who in this case may
not have hearing.
Step 2: The Google Speech API takes this audio file and
converts it into text that the user understands.
Step 3: The translated text is shown on the device's LCD
screen using a pop-up window created particularly for this
module using Python. As a result, the client comprehends
everything given to him quickly and efficiently.
Fig 2: Methodology
Fig 4: Alphabet Sign language
KNN-CLASSIFIER
One of the most fundamental supervised learning-based
machine learning algorithms is K-Nearest Neighbor. On the
premise that the new iteration and the previous cases are
Fig 5: Hand detection outcome
comparable, the K-NN technique classifies the new
Hand detection technique is shown in Fig. 5. This figure occurrence in the group that is the nearest equivalent to the
shows how the hand area was photographed using a camera. existing categories. This algorithm stores all the information
that is previously available and categorizes additional
130 ≤ 𝐶𝑟 ≤ 180 &130 ≤ 𝐶𝑏 ≤ 180&0.01 ≤ 𝐻 information based on similarities. This indicates that new data
. A binary image created from Fig. 5 is shown. Hand can be reliably and quickly categorized using the K-NN
segmentation using skin colour detection is shown in above approach [10]. The majority of its employment and research
figure. After that, a binary image of the segmented image is is in classification issues, even though it may be utilized for
created. The steps that follow are used to create the binary both classification and regression.
image: Use the Otsu method to produce a binary image after K-NN is a quasi-method that makes no assumptions about
first converting the segmented image from the skin detection the underlying data. KNN is also known as the "lazy learner
into a grayscale image using the luminance algorithm. Also, algorithm" due to the fact that it keeps the training dataset
the bounding box method based on the top-end x and y rather than learning from it right away. After using the dataset
coordinates was used to crop the binary image [17]. to finish a procedure, it sorts the input. This approach
effectively keeps the input constant throughout the training
phase and, after acquiring new data, classifies it into a
A. Feature Extraction Algorithms category that is relatively similar to the old data. [11]
● Decide the number of neighbors first, therefore, The camera takes a photo of the hand and recognizes the
k=5. letter in it based on the number and positioning of edge points
● The Euclidean distance between the data points will
then be computed after that. The distance between
any two locations that have already been
investigated in geometry is known as the Euclidean
distance.
Author Contributions: Funding: This research received Fund from Karnataka State Science
Conceptualization, S.S.; Methodology, S.S. and S.B.V.; software, S.S. and Technology Faculty Proposal Scheme.
and ARA.; validation, S.S., S.B.V., S.S..; formal analysis, S.S and Data Availability Statement: This research mainly focuses on data
S.B.V.; investigation, S.S. and S.B.V.; resources, S.S.; data curation, communication text and image data. The availability of data and
A.R.A and S.S writing—original draft preparation, S.S.; writing— materials is dummy data.
review and editing, S.B.V., S.S. and A.R.A visualization, S.S., and Acknowledgements: We would like to thank the management of
S.B.V supervision. Dayananda Sagar College of Engineering for the support rendered.
Conflicts of Interest: The authors declare no conflict of interest.