IRJET- Vision Based Sign Language by using Matlab

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 05 Issue: 03 | Mar-2018 www.irjet.net p-ISSN: 2395-0072
© 2018, IRJET | Impact Factor value: 6.171 | ISO 9001:2008 Certified Journal | Page 3241
VISION BASED SIGN LANGUAGE BY USING MATLAB
R. Sruthi1, B. Venkateswara Rao2, P. Nagapravallika3, G. Harikrishna4, K. Narendra Babu5
1 B.Tech IV Year, Department of ECE, Chalapathi Institute of Engineering & Technology, AP.
2 Assistant Professor, Department of ECE, Chalapathi Institute of Engineering & Technology, AP.
3,4,5 B.Tech IV ECE, Chalapathi Institute of Engineering & Technology, AP.
---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract - Human Computer Interaction moves forward in
the field of sign language interpretation. Indian Sign
Language (ISL) Interpretation system is a good way to help
the Indian hearing impaired people to interact with normal
people with the help of computer. As compared to other sign
languages, ISL interpretation has got less attention by the
researcher. In this paper, some historical background, need,
scope and concern of ISL are given. Vision based hand gesture
recognition system have been discussed as hand plays vital
communication mode. Considering earlier reported work,
various techniques available for hand tracking, segmentation,
feature extraction and classification are listed. Vision based
system have challenges over traditional hardware based
approach; by efficient use of computer vision and pattern
recognition, it is possible to work on such system which willbe
natural and accepted, in general.
Key Words: Sign Language, Hearing impaired,Computer,
Hand gestures, Hardware and Software based,
communication mode.
1. INTRODUCTION
This paper presents SIGN LANGUAGE TRANSLATION BY
USING MATLAB software for automatic translationofIndian
sign language into spoken English to assist the
communication between speech and/or hearing impaired
people and blind people. It could be used by deaf, dumb and
blind community as a translator to people that do not
understand sign language, avoiding by this way the
intervention of an intermediate person for interpretation
and allow communication using their natural way of
speaking.
The proposed software is standalone executable interactive
application program developed usingMATLABsoftwarethat
can be implemented in any standard windows to developed
operating laptop, desktop to operate with the camera,
processor and audio device. For sign to speech translation,
the one handed Sign gestures of the user are captured using
camera; vision analysis functions are performed in the
operating system and provide corresponding speech output
through audio device. For blind people special switches are
incorporated so that the corresponding speech output
through speaker. This system is trained to translate one
handed sign representations of predefined sign gestures to
voice. The software does not require the user to use any
special hand gloves.
The results are found to be highly consistent, reproducible,
with high precision and accuracy.
1.1 Related work
The authors Y.Madhuri , G.Anitha,M. Anburajan This report
presents a mobile VISION-BASED SIGN LANGUAGE
TRANSLATION DEVICE for automatic translation of Indian
sign language into speech in English to assist the hearing
and/or speech impaired people to communicate with
hearing people. It could be used as a translator for people
that do not understand sign language, avoiding by this way
the intervention of an intermediate person and allow
communication using their natural way of speaking. The
proposed system is an interactive application program
developed using LABVIEW software and incorporatedintoa
mobile phone. The sign language gesture images are
acquired using the inbuilt camera of the mobile phone;
vision analysis functions are performed in the operating
system and provide speech output through the inbuilt audio
device thereby minimizing hardware requirements and
expense[1].
The authors Dawod A.Y., Abdullah . and Alam MJ. Hand
segmentation is often the first step in applications such as
gesture recognition, hand tracking and recognition. We
propose a new technique for hand segmentation of color
images using adaptive skin color model. Our method
captures pixel values of a person's hand and converts them
into YCbCr color space. The technique will then map the
CbCr color space to CbCr plane to construct a clustered
region of skin color for the person. Edge detection isapplied
to the cluster in order to create an adaptive skin color
boundaries for classification. Experimental results
demonstrate successful detection over a variety of hand
variations in color, position, scale, rotation and pose[2].
The authors Dhruva N., Rupanagudi S.R., Sachin S.K.,Sthuthi
8., Pavithra R.and Raghavendra, Eigen values and Eigen
vectors are a part of linear transformations. Eigen vectors
are the directions along which the linear transformationacts
by stretching, compressing or flipping andEigenvaluesgives
the factor by which the compression or stretching occurs.
For recognition ofhand gestures, only hand portiontillwrist
is required, thus the unnecessary part is clipped off using
this hand cropping technique. After the desired portion of
the image is being cropped, feature extraction phase is
carried out. Here, Eigen values and Eigen vectors are found
out from the cropped image. Here is designed a new
classification technique that is Eigen value weighted

Euclidean distance between Eigen vectors which involved
two levels of classification[3].
The authors Bhame. V, Sreemathy. R, Dhumal. H. Hand
gesture is one of the methods used in sign language which is
most commonly used by deaf and dumb people to
communicate with each other or with normal people. The
proposed algorithm aims at developing real time image
processing based system for hand gesture recognition on
personal computer with an USB web cam. This paper
proposes a method to detect and recognize the static image
of Indian Sign Language numbering system from zero to
nine. The method is based on counting the open fingers in
the static images. The proposed algorithm for gesture
recognition is based on boundary tracing and finger tip
detection and also deals with images of bare hands, which
allows the signer to interact with the system in a natural
way. The proposed algorithm isfirst detectandsegmentsthe
hand region from the real time captured images. Then using
the proposed methodology, it locate the fingersandclassifies
the gesture. Further the system convertIndiansignsintotext
and then speech using an audio file stored on PC. The
algorithm is size invariant but it is orientation dependent.
The proposed system is implemented using OpenCV[4].
The authors Zhou, Qiangqiang; Zhao, Zhenbing, Video
surveillance system in the substation will enable operation
and maintenance staffs to monitor on-site power
equipments. But there's a variety of on-site equipments, it is
necessary to add image recognition technology to the
system. In this paper, a substation equipment image
recognition method based on SIFT feature matching is
proposed. To perform SIFT feature matching between the
template image and the monitoring image of the substation
equipment, mismatches are eliminated by meansofRANSAC
algorithm and the relative distanceratiobeingequalmethod.
The edge of the template image is gotten through OTSU
segmentation. At last, the location of the template image is
marked in monitoring image. Comparison of experimental
results show that the proposed methodcanmorecorrectand
effective than the original SIFT method, and it can
accomplish well substation equipmentimagerecognition[5].
The authors Archana S. Ghotkar, Dr. Gajanan K. Kharate, In
this paper, we introduce a hand gesture recognition system
to recognize the alphabets of Indian Sign Language. In our
proposed system there are 4 modules: real time hand
tracking, hand segmentation, feature extraction and gesture
recognition. Camshift method and Hue, Saturation, Intensity
(HSV) color model are used for hand tracking and
segmentation. For gesture recognition, Genetic Algorithm is
used. We propose an easy-to-use and inexpensive approach
to recognize single handed as well as double handed
gestures accurately. This system can definitely help millions
of deaf people to communicate with other normal people[6].
The authors Dominikus Willy, Ary Noviyanto, Aniati Murni
Arymurthy, The songket recognition is a challenging task.
The SIFT and SURF, which are feature descriptors, are
considered as potential features for pattern matching. The
Songket is a special pattern originally from Indonesia The
Songket Palembang isused in this research. One motif in the
Songket Palembang may has several differentbasicpatterns.
The matching scores, i.e., distance measure and number of
keypoint, are evaluated corresponding with the SIFT and
SURF method. SIFT method has been better than SURF
method, but SURF has been extremely faster than SIFT[7].
The authors Chenglong Yu, Member, IEEE, Xuan Wang,
Member, IEEE, Hejiao Huang, Member, IEEE, Jianping Shen,
Kun Wu,. This paper presents a featureextractionmethodor
hand gesture based on multi layer perception. Median and
smoothing ﬁlters are integrated to remove the noise.
Combinational parameters of Hu invariant moment, hand
gesture region, and Fourier descriptor are created to form a
new feature vector which can recognize hand gesture [8].
The authors Penghui Jiang, Shengjie Zhao, Samuel Cheng,
Speeded Up Robust Features (SURF) is one of the most
robust and widely used image matching algorithmsbasedon
local features. However, the performance for rotation image
is poor when one image is a rotated version of the other. To
improve the matching accuracy of rotation image, we
present an modified image matching algorithm combining
Haar wavelet and the rotation invariant Local Binary
Patterns (LBP). Firstly, keypoints are extracted from the
images for matching by applying the Hessian matrix and
integral images. Secondly, each keypoint is described by the
Rotation Invariant LBP patterns and Haarwavelet,whichare
computed from the image patch centered at the keypoint.
Finally, the matching pairs between thetwosetsofkeypoints
are determined by using the nearestneighbordistancebased
on matching strategy. The experimented resultsshowthatin
comparison with prior works the proposed algorithm is
efficient when tested on the images of scaling, rotation,
blurring and brightening [9].
2. DESCRIPTION
In general, deaf people have difficultyincommunicatingwith
others who do not understand sign language. Even those
who do speak aloud typically have a “deaf voice” of which
they are self-conscious and that can make them reticent.The
dumb people cannot speak and they will have difficulty in
communicating with others. The blind people cannot
communicate with others since they cannot see the world.
To overcome this sign language should be interfaced with
some other means for easy communication. So, we came up
with a solution called “VISION BASED SIGN LANGUAGE
TRANSLATOR BY USING MATLAB”.A coloured tape is fitted
to the fingers which acts as a input. we have to show this
fingers to pc with installed MATLAB.
The output from pc is converted to digital and processed by
using microcontroller and then it responds as voice by using
speaker. This voice output can be used for both blind and
dumb. Here, we provide four buttons for blind with
predefined voices and the last buttonforemergencycall.GSM
module supports for receiving the call.

In this project we have used a microcontroller, a speech IC
and also speaker to produce the output. Hardware
components used are power supply, and voice
IC(aPR33a3).Software used is
MATLAB. It is used in medical applications and used as a
speaking aid for dumb, deaf and blind patients who are in
ICUs MATLAB helps in grasping the gestures and transmits
the data pre-recorded accordingly, reliability in output etc
A video is captured at around 30 frames per second. This
number of frames per second is enough for computation.
More number of frames will lead to more time for
computation asmore data needs to be processed. The image
acquisition process is subjected to many environmental
factors like the exposure of light, the background and
foreground objects. In order to make feature extraction
easier later on, a plain background (white) is easier to work
on. This part consists of hand segmentation followed by
morphological operations. One method for this is proposed
an adaptive skin color model for hand segmentation by
mapping YCbCr color space into YCbCr color plane. Another
method for hand segmentation and tracking isbased onHSV
histogram. These methods can segment the hand in simple
as well ascomplex background. For simplicity purposes, the
hand segmentation isperformed by transforming the image
acquired into black and white image, where the background
will be white and the foreground i.e. the hand will be black.
Another way to preprocess the images acquired is the color
thresholding method. Using this method,thecolorregioncan
be segmented and the position of the region of interest can
be determined. Then, the color segmented images are
analyzed to obtain the unique features of each sign in the
sign language. But this method issuitable only for alphabets’
(A – Z) and numbers’ (0 – 9) recognition, because it uses the
unique combination of fingertip position for feature
extraction of each sign. The Gauss-Laplace edge detection
method can also be used to get the hand edge. The
experimental result shows the Gauss-Laplace algorithm is
used to effectively implement the hand edge detection.
Figure -1: Gauss Laplace Translator used in template
Figure-2: Gauss Laplace Algarithm
The technique to be chosen to recognize the sign depends
on the method used for preprocessing the image. If color
thresholding followed by fingertip position extraction is
used, then the recognition will be based on the position of
finger in the bounding box. A threshold value is set for the
difference between the database image and the input image.
If the difference obtained is above this threshold value, a
match is said to have been found.
3. CONCLUSION
In this paper, various algorithms and methods by which the
process of sign language translation can be performed are
discussed. The advantages of some methods over others are
also discussed. The vision of an efficient system to translate
sign language to text is quite achievable, but the challenges
lie in optimization.
4. RESULT
All input images were captured by webcam or an android
device. The captured image is then read in MATLAB and
converted in binary form of size defined so that it takes less
time and memory space during pattern recognition. Now by
using PCA it calculates the Eigen vectors and shows the
equivalent image to the input gesture with minimum
equivalent distance. At last it gives out the matched hand
gesture and alphabet corresponding tothegiveninputimage
from the database. It also produces an audio sound
indicating the output sign gesture.
REFERENCES
[1] Madhuri, Y.; Anitha, G.; Anburajan, M., "Vision-based sign
language translation device," in InformationCommunication
and Embedded Systems.
[2] Dawod A.Y., Abdullah 1. and Alam MJ., "Adaptive skin
color model for hand segmentation," Computer
Applications and Industrial Electronics.

[3] Dhruva N., Rupanagudi S.R., Sachin S.K., Sthuthi 8.,
Pavithra R. and Raghavendra, "Novel segmentation
algorithm for hand gesture recognition," Automation,
Computing, Communication, Control and Compressed
Sensing (iMac4s), 2013.
[4] Bhame, V.; Sreemathy, R.; Dhumal, H., "Vision basedhand
gesture recognition using eccentric approach for human
computer interaction," in Advances in Computing,
Communications.
[5] Zhou, Qiangqiang; Zhao, Zhenbing, "Substation
equipment image recognition based on SIFT feature
matching," in Image and Signal Processing.
[6] Archana S. Ghotkar, Dr. Gajanan K. Kharate, ”Study of
vision based hand gesture recognition using Indian sign
language “, International journal on smart sensing and
intelligent systems .
[7] Dominikus Willy, Ary Noviyanto, Aniati Murni
Arymurthy, “Evaluation of SIFT and SURF Features in the
Songket Recognition”.
[8] Chenglong Yu, Member, IEEE, Xuan Wang, Member,IEEE,
Hejiao Huang, Member, IEEE, Jianping Shen, Kun Wu,
“Vision-Based Hand Gesture Recognition Using
Combinational Features”, 2010 Sixth International
Conference on Intelligent Information Hiding and
Multimedia Signal Processing
[9] Penghui Jiang, Shengjie Zhao, Samuel Cheng, “Rotational
Invariant LBP-SURF for Fast and Robust Image Matching”.

IRJET- Vision Based Sign Language by using Matlab

More Related Content

What's hot (20)

Similar to IRJET- Vision Based Sign Language by using Matlab (20)

More from IRJET Journal (20)

Recently uploaded (20)

IRJET- Vision Based Sign Language by using Matlab