Autoencoders

An autoencoder is a type of artificial neural network that learns to represent

data
in a compressed form and then reconstructs it as closely as possible to the original
input.
Autoencoders consist of two components:
• Encoder: This compresses the input into a compact representation and captures
the most relevant features.
• Decoder: It reconstructs the input data from this compressed form to make it as
similar as possible to the original input.
• For example, if the input is a noisy image of handwritten digits, the autoencoder
can learn to remove noise by compressing the image into a smaller feature set
and reconstructing a cleaner version of the original image.
Auto-Encoders
• Autoencoders aim to minimise reconstruction error, which is the difference
between the input and the reconstructed output.
• They use loss functions such as Mean Squared Error (MSE) or Binary Cross-
Entropy (BCE) and optimise through backpropagation and gradient descent.
• They are used in applications like image processing, anomaly detection, noise
removal and feature extraction
Architecture of Autoencoder in Deep Learning

• The architecture of an
autoencoder consists of three
main components: the
Encoder, the Bottleneck
(Latent Space) and the
Decoder. Let’s deep dive into
each part to understand how
they work together.
Autoencoder that can compress the image file for
the MNIST dataset, by using the following
mechanism:
Encoder
• The encoder is the part of the network that takes the input data and
compresses it into a smaller, lower-dimensional representation.
• Input Layer: This is where the original data enters the network, e.g., an
image or a set of features.

• Hidden Layers: These layers apply transformations to the input data. The
encoder’s goal is to extract essential features and reduce the data’s
dimensionality.

• Output of Encoder (Latent Space): The encoder outputs a compressed

version of the data, often called the latent representation or encoding. This
is a condensed version of the input, retaining only the important features.
Bottleneck (Latent Space)

The bottleneck is the smallest layer of the network where the data is represented in
its most compressed form. It’s often referred to as the latent space or code.
• This layer contains a reduced set of features representing the most important
information from the input.

• The idea is that through this compression, the network learns the key patterns and
structures of the input data.
Decoder

The decoder is responsible for taking the compressed representation from the
latent space and reconstructing it back into the original data form.
• Hidden Layers: The decoder uses a series of layers to gradually expand the
compressed data back to the original input’s dimensions.

• Output Layer: This layer produces the reconstructed data and aims to closely
resemble the input data.
Loss Function in Autoencoder Training
During training an autoencoder aims to minimize the reconstruction loss which
measures the difference between the original input and the reconstructed
output. The choice of loss function depends on the type of data:
• Mean Squared Error (MSE): This is commonly used for continuous data. It
measures the average squared differences between the input and the
reconstructed data.

• Binary Cross-Entropy: Used for binary data (0 or 1 values). It calculates the

difference in probability between the original and reconstructed output.

• The network adjusts its weights to minimize this reconstruction

loss learning to extract and retain only the most important
features of the input data which are then stored in the latent
space (bottleneck layer).
Types of Autoencoders
Autoencoders come in several types, each suited for different tasks and with unique features.

1. Denoising Autoencoder: Denoising Autoencoder is trained to work with corrupted or noisy

input and learns to remove the noise and reconstruct the original, clean data.

2. Variational Autoencoder: A Variational autoencoder (VAE) makes assumptions about the

probability distribution of the data and tries to learn a better approximation of it. It uses
stochastic gradient descent to optimise and learn the distribution of latent variables. VAEs
are often used for generating new data, such as creating realistic images or text.

3. Convolutional Autoencoder: Convolutional autoencoder uses convolutional neural

networks (CNNS) which are specifically designed for processing images. In this type of
autoencoder, the encoder uses convolutional layers to extract features from an image, and
the decoder applies deconvolution, also called upsampling, to reconstruct the image.
• Variational Autoencoders (VAEs) are generative models in machine
learning (ML) that create new data similar to the input they are
trained on. Along with data generation they also perform common
autoencoder tasks like denoising.
• Encoder: Learns important patterns (latent variables) from input
data.

• Decoder: It uses those latent variables to reconstruct the input.

• Unlike traditional autoencoders that encode a fixed representation

VAEs learn a continuous probabilistic representation of latent
space. This allows them to reconstruct input data accurately and
generate new data samples that resemble the original input.
• A Variational Autoencoder is a generative model, it’s like
a regular autoencoder, but with a twist: It learns not just
to compress data into a point (latent vector), but into a
probability distribution over the latent space.
• So, instead of saying "This image maps to this exact
latent vector," a VAE says:"This image maps to a region
in the latent space (a mean and variance), and I can
sample from it.“
• It’s still encoder → bottleneck → decoder, but with
probabilistic reasoning in the bottleneck.
Architecture
1. Encoder (Inference Network)
• Instead of outputting a latent vector zzz, it outputs parameters of a distribution:
• Mean μ and standard deviation σ
2. Reparameterization Trick
• Sampling is non-differentiable, but we need to backpropagate.
• So we rewrite:
This allows gradients to flow through the network during training.
3. Decoder (Generative Network)
• Takes sampled 𝑧 and reconstructs the input.
• Output=
Loss Function (ELBO – Evidence
Lower Bound)
The VAE optimizes two terms:
1.Reconstruction Loss
1. Makes sure is close to x
2. Usually MSE or binary cross-entropy
2.KL Divergence Loss
1. Keeps the latent distribution close to a standard normal
2. Encourages smoothness and structure in latent space

3. KL divergence measures and minimises the "distance" between two probability

distributions
Why Do We Need VAEs?
Data Generation
• Regular autoencoders don't generate new data well; they just reconstruct existing
data.
• VAEs can generate realistic new examples by sampling from the latent space.
Structured Latent Space
• VAEs learn a smooth, continuous, and meaningful latent space.
• Similar inputs are mapped to nearby regions, making interpolation and sampling
intuitive.
Control + Interpretability
• Latent dimensions can capture high-level concepts (e.g., digit thickness, orientation,
facial expressions). You can manipulate latent codes to control output characteristics.
Autoencoder vs Variational Autoencoder
Feature Autoencoder Variational Autoencoder
Distribution (mean,
Output of encoder Single latent vector
variance)
Smooth, continuous,
Latent space Arbitrary, unstructured
probabilistic
Built for generative
Generation Not designed for it
modeling
Reconstruction + KL
Loss function Reconstruction only
divergence
Feature Autoencoder Variational Autoencoder