DEEP LEARNING

UNIT - VI

By,
Dr. Himani Deshpande 1
UNIT – VI
RECENT TRENDS AND APPLICTAIONS

6.1
Generative Adversarial Network(GAN) : Architecture

6.2
Applications: Image Generation, DeepFake
2
Generative Adversarial Network(GAN)

3
GENERATIVE AI

¡ Generative AI models can take inputs such as text, image, audio, video, and code and
generate new content into any of the modalities mentioned.
¡ For example, it can turn text inputs into an image, turn an image into a song, or turn
video into text.

4
GAN

¡ A generative adversarial network (GAN) is a machine learning (ML) model in which

two neural networks compete with each other by using deep learning methods to
become more accurate in their predictions.
¡ GANs typically run unsupervised and use a cooperative zero-sum game framework
to learn, where one person's gain equals another person's loss.

A generative adversarial network (GAN) has two parts: The generator learns to generate
plausible data. The generated instances become negative training examples for the
discriminator. The discriminator learns to distinguish the generator's fake data from real
5

data.
GAN

¡ The two neural networks that make up a GAN are referred to as the generator and
the discriminator.
¡ The generator is a convolutional neural network and the discriminator is a
deconvolutional neural network. The goal of the generator is to artificially
manufacture outputs that could easily be mistaken for real data. The goal of the
discriminator is to identify which of the outputs it receives have been artificially
created.

6
GAN
¡ Essentially, generative models create their own training data.
¡ While the generator is trained to produce false data, the discriminator network is
taught to distinguish between the generator's manufactured data and true examples.
¡ If the discriminator rapidly recognizes the fake data that the generator produces --
such as an image that isn't a human face -- the generator suffers a penalty.
¡ As the feedback loop between the adversarial networks continues, the generator will
begin to produce higher-quality and more believable output and the discriminator will
become better at flagging data that has been artificially created.
¡ For instance, a generative adversarial network can be trained to create realistic-
looking images of human faces that don't belong to any real person. 7
 GANS

• Generative
• Learn a generative model

• Adversarial
• Trained in an adversarial setting

• Networks
• Use Deep Neural Networks
GAN

10
GAN WORKING

• The generator import indiscriminate number of data samples and return an example of data.
• This generated example of data is feeded to the discriminator with actual data.
• Now the concept of probability comes into the play here. The discriminator has the task to
provide a probability score (in between 0 and 1 where 0 signifies fake and 1 signifies valid
instance).

• So broadly speaking there are two continuous vice-versa procedure run inside the GAN under
which discriminator shares a responsive loop with the real images to explore the ground truth
(by determining which data instance is fake and which is rea). On the other hand generator
shares a responsive loop with discriminator to determine the results of discriminator without
giving any concern to the truthness. 11
GAN

12
GAN

13
GAN

14
16
17
18
19
 MAGIC OF GANS…

Lotter, William, Gabriel Kreiman, and David Cox. "Unsupervised learning of visual structure using predictive generative networks." arXiv preprint arXiv:1511.06380 (2015).
MAGIC OF GANS…
Which one is Computer generated?

Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." arXiv preprint arXiv:1609.04802 (2016).
APPLICATIONS OF GANS…

http://people.eecs.berkeley.edu/~junyanz/projects/gvm/
23
 ADVERSARIAL TRAINING

• We can generate adversarial samples to fool a discriminative model

• We can use those adversarial samples to make models robust
• We then require more effort to generate adversarial samples
• Repeat this and we get the better discriminative model

• GANs extend that idea to generative models:

• Generator: generate fake samples, tries to fool the Discriminator
• Discriminator: tries to distinguish between real and fake samples
• Train them against each other
• Repeat this and we get better Generator and Discriminator
CHALLENGES FACED BY GANS

25
PROBLEMS OF AUTOENCODER

¡ It does not really try to simulate real images

NN As close as
code Decoder
Output
possible

One pixel difference to Also one pixel

the target difference to the target

Realistic Fake
GRADUAL AND STEP-WISE GENERATION

NN NN NN
Generator Generator Generator
v1 v2 v3

Generated Discri- Discri- Discri-

images minator minator minator
v1 v2 v3

These are Real images:

Binary classifiers
GAN – LEARN A DISCRIMINATOR
Randomly NN
Generator
sample a v1
vector 0 0 0 0

Something like Real images

Decoder in VAE Sampled from
DB: 1 1 1 1

Discri-
image minator 1/0 (real or fake)
v1
 Randomly sample
a vector

GAN – LEARN A GENERATOR

Train NN
Updating the parameters of this Generator
generator v2 v1

The output be classified They have

as “real” (as close to 1 Opposite
objectives
as possible)
Generator + Discriminator =
a network Do not Discri-
Train
minator
Using gradient descent to This
v1
update the parameters in the
generator, but fix the
discriminator 1.0 0.13
GAN’S ARCHITECTURE
x

D D(x)

G
z
G(z)
D(G(z))

• Z is some random noise (Gaussian/Uniform).

• Z can be thought as the latent representation of the image.
TRAINING DISCRIMINATOR
TRAINING GENERATOR
ADVANTAGES OF
GANS
• Plenty of existing work on Deep Generative Models
• Boltzmann Machine
• Deep Belief Nets
• Variational AutoEncoders (VAE)

• Why GANs?
• Sampling (or generation) is straightforward.
• Training doesn't involve Maximum Likelihood estimation.
• Robust to Overfitting since Generator never sees the training data.
• Empirically, GANs are good at capturing the modes of the distribution.
Goodfellow, Ian. "NIPS 2016 Tutorial: Generative Adversarial Networks." arXiv preprint arXiv:1701.00160 (2016).
GAN VS OTHER IMAGE HANDLING MODELS

• GAN have the capability to predict the corresponding new frame in a video.
• In the case of Image handling, it can also give a boost to the resolution of an
Image.
• If there is a necessity of Image to Image interpretation, for this purpose GAN
model can be used. It means a firmly new image can be generated from an
old image with the help of a GAN model.
• With the help of GAN model text to an image generation is also possible
which means provide the description in the form of text to the GAN model,
and it can generate a pragmatic photo of the specification.

39
PROBLEMS WITH GANS

• Probability Distribution is Implicit

• Not straightforward to compute P(X).
• Thus Vanilla GANs are only good for Sampling/Generation.

• Training is Hard
• Non-Convergence
• Mode-Collapse

Goodfellow, Ian. "NIPS 2016 Tutorial: Generative Adversarial Networks." arXiv preprint arXiv:1701.00160 (2016).
WHY USE GANS FOR GENERATION?

• Can be trained using back-propagation for Neural Network based

Generator/Discriminator functions.
• Sharper images can be generated.
• Faster to sample from the model distribution: single forward pass
generates a single sample.
Applications of GAN

42
IMAGE AND VIDEO SYNTHESIS

¡ GANs have been extensively used for image and video synthesis.
¡ They can generate realistic images of faces, objects, and even landscapes.
¡ These images can be used for various purposes, such as art, advertising, or
video game development. GANs can also be used for video synthesis,
generating new frames that follow the same pattern as the real video.

43
IMAGE GENERATION

44
FACE INPAINTING

45
Facial inpainting is used for generating facial features for missing pixels in a face image.
STYLE TRANSFER

¡ Style transfer is applying one image's style to another. GANs can perform style transfer by training a
network to generate an image with the content of one image and the style of another. This technique
has been used to generate images in the style of famous artists, such as Van Gogh or Picasso.

46
 DATA AUGMENTATION

Data augmentation is the process of

generating new samples from an existing
dataset to increase size. GANs can be
used for data augmentation by
generating new samples similar to the
real samples but different enough to
provide additional training data. This
technique can be helpful in situations
with limited data available for training a
deep learning model.
47
IMAGE RESTORATION

¡ GANs can also be used for image restoration, generating a high-quality version of a low-
quality image. This technique is proper when the original image is of low quality, such as in
medical or satellite imaging. GANs can also remove noise from images and enhance image
resolution.

48
 DEEPFAKE

Deepfakes are the manipulation of facial appearance through deep

generative methods.
Deepfakes are a machine-aided synthetic media technology used for the
generation or manipulation of text and video that can appear quite
realistic to the untrained eye, and they have the potential for considerable
implications across culture, geopolitics, and security.

Deepfakes often transform existing source content where one person is swapped for another. They also create
entirely original content where someone is represented doing or saying something they didn't do or say.

The greatest danger posed by deepfakes is their ability to spread false

information that appears to come from trusted sources. For example, in 2022 a 49
deepfake video was released of Ukrainian president Volodymyr Zelenskyy asking
his troops to surrender.
TEXT-TO-IMAGE GENERATION
¡ GANs can also be used for generating images based on textual descriptions. This technique can be
helpful when a visual representation of a text description is needed, such as designing products or
creating visual aids for the visually impaired.

50
¡ THANK YOU..

51