8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

Images haven’t loaded yet. Please exit printing, wait for images to load, and try to
Vindula Jayawardana Follow
print again.
Postgraduate Researcher at UOM | Research Engineer | Writer | Entrepreneur | www.vindulaj.com
Aug 4, 2017 · 5 min read

Autoencoders — Bits and Bytes of

Deep Learning

One way to think of what deep learning does is as “A to B mappings,”

says Andrew Ng, chief scientist at Baidu Research. “You can input an
audio clip and output the transcript. That’s speech recognition.” As
long as you have data to train the software, the possibilities are endless,
he maintains. “You can input email, and the output could be: Is this
spam or not?” Input loan applications, he says, and the output might be
the likelihood a customer will repay it. Input usage patterns on a eet of
cars, and the output could advise where to send a car next.

Rather making the facts complicated by having complex de nitions,

think of deep learning as a subset of a subset. Arti cial Intelligence
encircles wide range of technologies and techniques that enable
computers systems to unravel problems in ways that at least
super cially resemble thinking. Within that sphere, there is that whole
toolbox of enigmatic but important mathematical techniques which
drives the motive of learn by experience. That subset is known to be
machine learning. Finally, within machine learning is the smaller
subcategory called deep learning (also known as deep structured
learning or hierarchical learning)which is the application of arti cial

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 1/10
8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

neural networks (ANNs) to learning tasks that contain more than one
hidden layer.

. . .

What’s an Autoencoder?
Despite its somewhat initially-sounding cryptic name, autoencoders are
a fairly basic machine learning model. Autoencoders (AE) are a family
of neural networks for which the input is the same as the output. They
work by compressing the input into a latent-space representation, and
then reconstructing the output from this representation.

Autoencoder architecture

In more terms, autoencoding is a data compression algorithm where

the compression and decompression functions are,

1. Data-speci c : Autoencoders are only able to compress data

similar to what they have been trained on. An autoencoder which
has been trained on human faces would not be performing well
with images of modern buildings. This improvises the di erence
between autoencoders and MP3 kind of compression algorithms
which only hold assumptions about sound in general, but not
about speci c types of sounds.

2. Lossy : This means that the decompressed outputs will be

degraded compared to the original inputs. Just like what you see
in JPEG or MP3.

3. Learned automatically from examples : If you have appropriate

training data, it is easy to train specialized instances of the
algorithm that will perform well on a speci c type of input. It
doesn’t require any new engineering.

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 2/10
8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

Additionally, in almost all contexts where the term “autoencoder” is

used, the compression and decompression functions are implemented
with neural networks.

Why Autoencoders?
Despite the fact, the practical applications of autoencoders were pretty
rare some time back, today data denoising and dimensionality
reduction for data visualization are considered as two main
interesting practical applications of autoencoders. With appropriate
dimensionality and sparsity constraints, autoencoders can learn data
projections that are more interesting than PCA or other basic
techniques.

Performance comparison of Autoencoders and PCA

Convolutional Autoencoders
In traditional architecture of autoencoders, it is not taken into account
the fact that a signal can be seen as a sum of other signals.
Convolutional Autoencoders (CAE),on the other way, use the
convolution operator to accommodate this observation. Convolution
operator allows ltering an input signal in order to extract some part of
its content. They learn to encode the input in a set of simple signals and
then try to reconstruct the input from them.

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 3/10
8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

A convolution between a 4x4x1 input and a 3x3x1 convolutional lter.

The result is a 2x2x1 activation map. (Source)

Refer this for the use cases of convolution autoencoders with pretty
good explanations using examples. We will see a practical example of
CAE later in this post.

Building Autoencoders
We will start with the most simple autoencoder that we can build. In
the later part, we will be looking into more complex use cases of the
autoencoders in real examples.

Following is the code for a simple autoencoder using keras as the

platform. keras provided MNIST digits are used in the example.

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 4/10
8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

1 import numpy as np
2 from keras.layers import Input, Dense
3 from keras.models import Model
4 from keras.datasets import mnist
5 import matplotlib.pyplot as plt
6
7 # this is the size of our encoded representations
8 encoding_dim = 32 # 32 floats -> compression of factor 24
9
10 # this is our input placeholder
11 input_img = Input(shape=(784,))
12 # "encoded" is the encoded representation of the input
13 encoded = Dense(encoding_dim, activation='relu')(input_img
14 # "decoded" is the lossy reconstruction of the input
15 decoded = Dense(784, activation='sigmoid')(encoded)
16 # this model maps an input to its reconstruction
17 autoencoder = Model(input_img, decoded)
18 # this model maps an input to its encoded representation
19 encoder = Model(input_img, encoded)
20 # create a placeholder for an encoded (32-dimensional) inp
21 encoded_input = Input(shape=(encoding_dim,))
22 # retrieve the last layer of the autoencoder model
23 decoder_layer = autoencoder.layers[-1]
24 # create the decoder model
25 decoder = Model(encoded_input, decoder_layer(encoded_input
26 # configure our model to use a per-pixel binary crossentro
27 autoencoder.compile(optimizer='adadelta', loss='binary_cro
28
29 # prepare our input data. We're using MNIST digits, and we
30 (x_train, _), (x_test, _) = mnist.load_data()
31 # normalize all values between 0 and 1 and we will flatten
32 x_train = x_train.astype('float32') / 255.
33 x_test = x_test.astype('float32') / 255.
34 x_train = x_train.reshape((len(x_train), np.prod(x_train.s
35 x_test = x_test.reshape((len(x_test), np.prod(x_test.shape
36 print x_train.shape
37 print x_test.shape
38
39 # train our autoencoder for 50 epochs
40 autoencoder.fit(x_train, x_train,
41 epochs=50,
42 b t h i 256

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 5/10
8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

42 batch_size=256,
43 shuffle=True,

With this code snippet, we will get the following output.

In the above image, the top row is the original digits, and the bottom
row is the reconstructed digits. As you can see, we have lost some
important details in this basic example.

Since our inputs are images, it makes sense to use convolutional neural
networks as encoders and decoders. In practical settings, autoencoders
applied to images are always convolutional autoencoders as they
simply perform much better.

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 6/10
8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

1 from keras.layers import Input, Dense, Conv2D, MaxPooling2

2 from keras.models import Model
3 from keras import backend as K
4 from keras.datasets import mnist
5 import numpy as np
6 from keras.callbacks import TensorBoard
7 import matplotlib.pyplot as plt
8
9 input_img = Input(shape=(28, 28, 1)) # adapt this if usin
10
11 x = Conv2D(16, (3, 3), activation='relu', padding='same')(
12 x = MaxPooling2D((2, 2), padding='same')(x)
13 x = Conv2D(8, (3, 3), activation='relu', padding='same')(x
14 x = MaxPooling2D((2, 2), padding='same')(x)
15 x = Conv2D(8, (3, 3), activation='relu', padding='same')(x
16 encoded = MaxPooling2D((2, 2), padding='same')(x)
17
18 # at this point the representation is (4, 4, 8) i.e. 128-d
19
20 x = Conv2D(8, (3, 3), activation='relu', padding='same')(e
21 x = UpSampling2D((2, 2))(x)
22 x = Conv2D(8, (3, 3), activation='relu', padding='same')(x
23 x = UpSampling2D((2, 2))(x)
24 x = Conv2D(16, (3, 3), activation='relu')(x)
25 x = UpSampling2D((2, 2))(x)
26 decoded = Conv2D(1, (3, 3), activation='sigmoid', padding=
27
28 autoencoder = Model(input_img, decoded)
29 autoencoder.compile(optimizer='adadelta', loss='binary_cro
30
31 (x_train, _), (x_test, _) = mnist.load_data()
32
33 x_train = x_train.astype('float32') / 255.
34 x_test = x_test.astype('float32') / 255.
35 x_train = np.reshape(x_train, (len(x_train), 28, 28, 1))
36 x_test = np.reshape(x_test, (len(x_test), 28, 28, 1)) # a
37
38 autoencoder.fit(x_train, x_train,
39 epochs=50,
40 batch_size=128,
41 shuffle=True,

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 7/10
8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

With the convolution autoencoder, we will get the following input and
reconstructed output.

Autoencoder Practical Application : Image Denoising

In this section, we will be looking into the use of autoencoders in its
real world usage, for image denoising. We will train the convolution
autoencoder to map noisy digits images to clean digits images.

We will generate synthetic noisy digits by applying a gaussian noise

matrix and clip the images between 0 and 1.

When autoencoder is trained, we can use it to remove the noises added

to images we have never seen! And here is how the input and
reconstructed output will look like.

Conclusion

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 8/10
8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

An autoencoder is an arti cial neural network used for unsupervised

learning of e cient codings. In the modern era, autoencoders have
become an emerging eld of research in numerous aspects such as in
anomaly detection. In this post, it was expected to provide a basic
understanding on the aspects of what, why and how of autoencoders.

References

1. https://hackernoon.com/autoencoders-deep-learning-bits-1-
11731e200694

2. https://blog.keras.io/building-autoencoders-in-keras.html

3. https://www.technologyreview.com/s/513696/deep-learning/

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 9/10
8/18/2018 Autoencoders — Bits and Bytes of Deep Learning – Towards Data Science

https://towardsdatascience.com/autoencoders-bits-and-bytes-of-deep-learning-eaba376f23ad 10/10