Menu

Long Short-Term
Memory Networks
(LSTM)- simply
explained!
4. June 2022 / Machine Learning

The Long Short-Term Memory (short: LSTM) model is a subtype of Recurrent Neural Networks (RNN). It is
used to recognize patterns in data sequences, such as those that appear in sensor data, stock prices, or
natural language. RNNs can do this because, in addition to the actual value, they also include its position
in the sequence in the prediction.

What are Recurrent Neural Networks?

To understand how Recurrent Neural Networks work, we have to take another look at how regular
feedforward neural networks are structured. In these, a neuron of the hidden layer is connected with the
neurons from the previous layer and the neurons from the following layer. In such a network, the output of
a neuron can only be passed forward, but never to a neuron on the same layer or even the previous layer,
hence the name “feedforward”.
 Structure Feedforward Neural Network | Source: Author

This is different for recurrent neural networks. The output of a neuron can very well be used as input for a
previous layer or the current layer. This is much closer to how our brain works than how feedforward neural
networks are constructed. In many applications, we also need to understand the steps computed
immediately before improving the overall result.

What Problems do RNNs face?

Recurrent Neural Networks were a real breakthrough in the field of Deep Learning, as for the first time, the
computations from the recent past were also included in the current computation, significantly improving
the results in language processing. Nevertheless, during training, they also bring some problems that need
to be taken into account.

As we have already explained in our article on the gradient method, when training neural networks with
the gradient method, it can happen that the gradient either takes on very small values close to 0 or very
large values close to infinity. In both cases, we cannot change the weights of the neurons during
backpropagation, because the weight either does not change at all or we cannot multiply the number
with such a large value. Because of the many interconnections in the recurrent neural network and the
slightly modified form of the backpropagation algorithm used for it, the probability that these problems
will occur is much higher than in normal feedforward networks.

Regular RNNs are very good at remembering contexts and incorporating them into predictions. For
example, this allows the RNN to recognize that in the sentence “The clouds are at the ___” the word “sky”
is needed to correctly complete the sentence in that context. In a longer sentence, on the other hand, it
becomes much more difficult to maintain context. In the slightly modified sentence “The clouds, which
partly flow into each other and hang low, are at the ___ “, it becomes much more difficult for a Recurrent
Neural Network to infer the word “sky”.
How do Long Short-Term Memory Models work?

The problem with Recurrent Neural Networks is that they have a short-term memory to retain previous
information in the current neuron. However, this ability decreases very quickly for longer sequences. As a
remedy for this, the LSTM models were introduced to be able to retain past information even longer.

The problem with Recurrent Neural Networks is that they simply store the previous data in their “short-term
memory”. Once the memory in it runs out, it simply deletes the longest retained information and replaces
it with new data. The LSTM model attempts to escape this problem by retaining selected information in
long-term memory. This long-term memory is stored in the so-called Cell State. In addition, there is also
the hidden state, which we already know from normal neural networks and in which short-term
information from the previous calculation steps is stored. The hidden state is the short-term memory of
the model. This also explains the name Long Short-Term Networks.

In each computational step, the current input x(t) is used, the previous state of short-term memory c(t-1),
and the previous state of hidden state h(t-1).

LSTM Architecture | Photo based on Towards Data Science

These three values pass through the following gates on their way to a new Cell State and Hidden State:
 1. In the so-called Forget Gate, it is decided which current and previous information is kept and which is
thrown out. This includes the hidden status from the previous pass and the current input. These values
are passed into a sigmoid function, which can only output values between 0 and 1. The value 0 means
that previous information can be forgotten because there is possibly new, more important information.
The number one means accordingly that the previous information is preserved. The results from this
are multiplied by the current Cell State so that knowledge that is no longer needed is forgotten since it
is multiplied by 0 and thus dropped out.

2. In the Input Gate, it is decided how valuable the current input is to solve the task. For this, the current
input is multiplied by the hidden state and the weight matrix of the last run. All information that
appears important in the Input Gate is then added to the Cell State and forms the new Cell State c(t).
This new Cell State is now the current state of the long-term memory and will be used in the next run.

3. In the Output Gate, the output of the LSTM model is then calculated in the Hidden State. Depending on
the application, it can be, for example, a word that complements the meaning of the sentence. To do
this, the sigmoid function decides what information can come through the output gate and then the
cell state is multiplied after it is activated with the tanh function.

Using our previous example, the whole thing becomes a bit more understandable. The goal of the LSTM is
to fill the gap. So the model goes through word by word until it reaches the gap. In the Recurrent Neural
Network, the problem here was that the model had already forgotten that the text was about clouds by
the time it arrived at the gap.

As a result, no correct prediction could be made. Let us, therefore, consider how an LSTM would have
behaved. The information “cloud” would very likely have simply ended up in the cell state, and thus would
have been preserved throughout the entire computations. Arriving at the gap, the model would have
recognized that the word “cloud” is essential to fill the gap correctly.

Which applications rely on LSTM?

For many years, LSTM Networks were the best tool in natural language processing because they could
hold the context of a sentence “in memory” for a relatively long time. The following concrete programs rely
on this type of neural network:

Apple’s keyboard completion is based on an LSTM network. In addition, Siri, Apple’s voice assistant, was
also based on this type of neural network.

Google’s AlphaGo software also relied on long short-term memory and was thus able to beat real
humans in the game of Go.

Google’s translation service was based on LSTMs, among other things.

Nowadays, however, the importance of LSTMs in applications is declining somewhat, as so-called