International Journal of Engineering & Technology, 7 (4.

6) (2018) 133-137

International Journal of Engineering & Technology

Website: www.sciencepubco.com/index.php/IJET

Research paper

Speech Recognition Using Convolutional Neural Networks

D. Nagajyothi 1 , P. Siddaiah 2
1
Associate Professor of Electronics and Communication Engineering, Vardhaman College of Engineering, Shamshabad, Telangana, India.e-
mail: [email protected]
2
Dean of Department of Electronics and Communication Engineering, University College of Engineering and Technology, Acharya
Nagarjuna University, Guntur, India.
*Corresponding author E-mail: [email protected]

Abstract

Automatic speech recognition (ASR) is the process of converting the vocal speech signals into text using transcripts. In the present era of
computer revolution, the ASR plays a major role in enhancing the user experience, in a natural way, while communicating with the machines.
It rules out the use of traditional devices like keyboard and mouse, and the user can perform an endless array of application s like controlling
of devices and interaction with customer care. In this paper, an ASR based Airport enquiry system is presented. The system has been
developed natively for telugu language. The database is created based on the most frequently asked questions in an airport en quiry. Because
of its high performance, Convolutional Neural Network (CNN) has been used for training and testing of the database. The salient feature of
weight connectivity, local connectivity and polling result is a through training of the system, thus resulting in a superior testing performance.
Experiments performed on wideband speech signals results in significant improvement in the performance of the system in comparison to the
traditional techniques.

Keywords: Neural Networks (NN),Convolutional Neural Networks(CNN).

1. Introduction of an ASR system is quite difficult as it faces several sources of

variability. One of them is the style of speech. Just the style of
speech can have a lot to do with the performance of an ASR
Speech recognition is a system that translates spoken utterances
system. Rather than continuous speech individually, isolated
into text. Text can be either in terms of words or word sequences
words are much easier for the ASR systems. This is because in
or it could be in syllables, or it can be any sub-word units or
continuous speech words are flowing freely into one another.
phones, or even characters, but you're translating speech into its
There is a phenomenon called coarticulation, where the preceding
corresponding text form. Some of the well known examples are
words affect the words that are coming and so on. This
YouTube’s closed captioning it has an ASR engine running
phenomenon becomes challenging for the ASR systems to handle
producing the corresponding transcripts for the speech, the audio
itv[2].The other source of variability is environment. If a person
and the video clips. The voice mail transcription also has an ASR
is talking under noisy conditions or if the acoustics of the room
engine running. The older prototypes of ASR systems are the
such as the echo produced in the room becomes challenging.
dictation systems. The dictation systems are the words are spoken
Background noise could be of two types, if the noise is of
out and then the corresponding transcripts are produced. Siri,
vehicles then it can be isolated but if the noise is made by people
Cortana, Google Voice, all of their front ends are ASR
talking behind then that would be hard for the ASR system to
engines.ASR is strictly just translating the spoken utterances into
pick the foreground voice. Various characteristics of speakers are
text. The development of a good ASR system is very desirable as
also a challenging problem, as the main constraints are grammar,
it fetches a lot of advantages [1]. A person can save his time
vocabulary and the language which might not have a written
which is mostly used in typing, so rather than typing he can speak
form, age also changes the characteristics of the speech.
to the devices. There is another kind of socially desirable aspect
of building a good ASR system. As we know that the present
technology has interfaces, if an ASR system is built into the
interfacing device, then it can it can be used both by literate and
the illiterate users. So even the users who cannot read or write in
a particular language, can interact with the technology, if it is
voice driven. Lots of languages are close to extinction, so if the
technologies are built for such languages then it could be
contributed towards the preservation of such languages. Building

Copyright © 2018 Authors. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted
use, distribution, and reproduction in any medium, provided the original work is properly cited.
International Journal of Engineering & Technology 134

2. Convolutional Neural Network in this if a pixel value is negative then the negative values are
replaced with zeros. This is done to all the filtered signals. This
becomes another type of layer which is known as a rectified
linear unit, a stack of signals which becomes a stack of signals
with no negative values. Now the three layers are stacked up so
that one output will become the input for the next. The final layer
is the fully connected layer.

Fig.2: CNN simplified workflow model.

The standard feed-forward fully connected Neural network (NN)

Fig.1: CNN structure is a computational model composed of several layers. An input to
a particular unit is outputs of all the units in the previous layer (or
Convolutional Neural Network can do a lot of good things if they input data for the first layer). The unit output is a single linear
are fed with a bunch of signals for instance to learn some basic regression, to which output value a specific activation function is
signals such as frequency changes, amplitude changes. Since, applied. Convolutional neural network (CNN) is a type of NN
they are multi neural networks, the first layer is fed with this where the input variables are related spatially to each other [11].
information. The second layer is fed with some recognizable To take into account very important spatial positions, CNNs were
features. To illustrate this, a signal of two-dimensional array of developed. Not only they are able to detect general spatial
pixels is considered. It is a check board with each square on the dependencies, but also are capable of specific patterns
board is either light or dark colour. By observing the pattern CNN recognition. Shared weights, representing different patterns,
decides whether it is a signal with frequency change or amplitude improve the convergence by reducing significantly the number of
change. parameters. CNN recognize small patterns at each layer,
generalizing them (detecting higher order, more complex
The convolutional neural network match the parts of the signal patterns) in subsequent layers. This allows detection of various
instead of considering the whole signal of pixels as it becomes patterns and keeps the number of weights to be learnt very low
difficult for a computer to identify the signal when the whole set [12][13].
of pixels are considered [8][9]. The mathematics behind matching
these is filtering. The way this is done is by considering the 3. Two dimensional representation
feature that is lined up with this patch signal and then one by one
pixels are compared and multiplied by each other and then add it
up and divide it with the total number of pixels. This step is The first step is to providing fragments in a two dimensional
repeated for all the pixels that is considered. The act of matrix. The filters of the first layer start doing recording for two
convolving signals with a bunch of filters, a bunch of features dimensional values simultaneously [14]. The recorded values are
which creates a stack of filtered images is called as convolutional then combined in the next, higher level layers. The CNN filter has
layer. It is a layer because it is operating based on stack that is in different network behavior depending on its activity level. CNN
convolution one signal becomes a stack of filtered signals. We get was able to learn simple patterns and then predict more accurately
a lot of filtered signals because of the presence of the filters. by learning different combinations.
Convolution layer is one part.
4. Selection of activation functions
The next big part is called as pooling that is how a signal stack
can be compressed. This is done by considering a small window One of the important decisions related to CNN was setting proper
pixel which might be a 2 by 2 window pixel or 3 by 3. On activation functions in the units. The most common activation
considering a 2 by 2 window pixel and pass it in strides across the function tanh is used in first Convolutional layer, while Rectified
filtered signals, from each window the maximum value is Linear Unit in the next two Convolutional layers. We do not
considered. This passed through the whole signal. At the end it is want to allow an input from the previous layer [15].Because some
found that by considering only the maximum values the size of patterns indicating a correlation.
the filtered signal is reduced [10]. The third part is normalization,
135 International Journal of Engineering & Technology

5. Improvement of the CNN Filter E : Can I make a group booking?

T :Nenu ekumandiki ela book cheskovali / Nenu ekumanidki ela

The first step is the input data was normalized to increase the
book cheskovachu
learning speed. The second step is improve the network
structure..The last step we used max pooling in the last
Convolutional layer,. Since the errors are only propagated to the నేను ఏకు మందికి ఎలా బుకచేసుకోవాలి / నేను ఏకు మందికి ఎలా
position of the maximally activated unit, it is highly probable that బుకచేస్క ోవచ్ుే
in a wider span there will exist a strong indication of a
communication between cells [16].
E : Can I book and hold a reservation and pay later?
6. Experimental Results
T :Nenu ticket mundu book chesi dabbulu taravata katacha
Input Data:
The input data for model is based on the airport enquiry system. నేను టికెట్మందు బుకచేసి డబుులు తరవాత కటట చ్చే
The following questions are used as transcript.

Questions: Etc

E :How do I book my flight? Total words trained for the process of recognition are 583. The
total phone count resulted is 1765. The distinct phones are BU,
T :Nenu vimana ticket ela book cheskogalanu / Nenu vimana CHE, E, GA, K, KE, KO, LA, LI, MA, NA, NE, NU, S, SKO,
ticket ela book cheskovali T, TI, VA, VI.

నేను విమాన టికెట్ ఎలా బుకచేసుకోగలను /నేను విమాన టికెట్ ఎలా

బుకచేసుకోవాలి

(a) (b)

(c) (d)
International Journal of Engineering & Technology 136

(e) (f)

(h) (g)

(i) (j)
137 International Journal of Engineering & Technology

(k) (l)
Figure. (a) – (l) – Similarity matrices of phones 3 phones per sample.

7. Conclusion transcription,”in Proc. IEEE Workshop Autom. Speech

Recognition Understand.(ASRU), 2011, pp. 24–29.
[9] N. Morgan, “Deep and wide: Multiple layers in automatic
In this paper, we investigated the Performance of an ASR based speechrecognition,” IEEE Trans. Audio, Speech, Lang. Process.,
on CNNs, which takes raw speech signal, as input to large vol. 20, no.1, pp. 7–13, Jan. 2012.
vocabulary task. Our analysis on wideband signals proved that [10] A. Mohamed, G. Dahl, and G. Hinton, “Deep belief networks for
the CNN based system is able to achieve good performance than phonerecognition,” in Proc. NIPS Workshop Deep Learn. Speech
the conventional Neural Network Techniques based system. Recognition Related Applicat., 2009.
[11] A. Mohamed, D. Yu, and L. Deng, “Investigation of full-
sequencetraining of deep belief networks for speech recognition,”
8. Acknowledgement in Proc.Interspeech, 2010, pp. 2846–2849.
[12] L. Deng, D. Yu, and J. Platt, “Scalable stacking and learning
forbuilding deep architectures,” in Proc. IEEE Int. Conf.
We sincerely thank Acharya Nagarjuna University (ANU) and Acoustics,Speech, Signal Process., 2012, pp. 2133–2136.
Vardhaman College of Engineering (VCE) for allowing us to use [13] G. Dahl,D.Yu, L.Deng, and A. Acero, “Context-dependent pre-
resources to carry out this work. traineddeep neural networks for large-vocabulary speech
recognition,” IEEETrans. Audio, Speech, Lang. Process., vol. 20,
no. 1, pp. 30–42, Jan.2012.
9. References [14] F. Seide, G. Li, and D. Yu, “Conversational speech transcription
using context-dependent deep neural networks,” in Proc.
Interspeech, 2011,pp. 437–440.
[1] H. Jiang, “Discriminative training for automatic speech [15] T. N. Sainath, B. Kingsbury, B. Ramabhadran, P. Fousek, P.
recognition: A survey,” Comput. Speech, Lang., vol. 24, no. 4, pp. Novak,and A. Mohamed, “Making deep belief networks effective
589–608, 2010. for largevocabulary continuous speech recognition,” in IEEE
[2] X. He, L. Deng, and W. Chou, “Discriminative learning in Workshop Autom.Speech Recogn. Understand. (ASRU), 2011, pp.
sequential pattern recognition—A unifying review for 30–35.
optimization-oriented speech recognition,” IEEE Signal Process. [16] J. Pan, C. Liu, Z. Wang, Y. Hu, and H. Jiang, “Investigation of
Mag., vol. 25, no. 5, pp. 14–36, Sep. 2008. deepneural networks (DNN) for large vocabulary continuous
[3] L. Deng and X. Li, “Machine learning paradigms for speech speech recognition:Why DNN surpasses GMMs in acoustic
recognition: An overview,” IEEE Trans. Audio, Speech, Lang. modeling,” in Proc.ISCSLP, 2012.
Process., vol. 21, no. 5, pp. 1060–1089, May 2013.
[4] G. E. Dahl, M. Ranzato, A. Mohamed, and G. E. Hinton, “Phone
recognitionwith the mean-covariance restricted Boltzmann
machine,” Adv.Neural Inf. Process. Syst., no. 23, 2010.
[5] A. Mohamed, T. Sainath, G. Dahl, B. Ramabhadran, G. Hinton,
andM. Picheny, “Deep belief networks using discriminative
features forphone recognition,” in Proc. IEEE Int. Conf. Acoust.,
Speech, SignalProcess. (ICASSP), May 2011, pp. 5060–5063.
[6] D. Yu, L. Deng, and G. Dahl, “Roles of pre-training and fine-
tuningin context-dependent DBN-HMMs for real-world speech
recognition,”in Proc. NIPS Workshop Deep Learn. Unsupervised
Feature Learn.,2010.
[7] G. Dahl, D. Yu, L. Deng, and A. Acero, “Large vocabulary
continuousspeech recognition with context-dependent DBN-
HMMs,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process.,
2011, pp.4688–4691.
[8] F. Seide, G. Li, X. Chen, and D. Yu, “Feature engineering in
context-dependent deep neural networks for conversational speech