![List of Figures
1 Example of a GAN architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Architecture of a GAN. The generator and the discriminator are synchronously
trained. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 Example of the xCos architecture taken from [13] . . . . . . . . . . . . . . . . 14
4 Learning to Simplify architecture. Image taken from [20] . . . . . . . . . . . . 18
5 Figure from [20] showing the results of Learning to Simplify approach. . . . . 19
6 Figure from [19] that highlights the comparison between the Mastering sketch-
ing’s results and the ones of the Learning to Simplify. . . . . . . . . . . . . . 21
7 Results obtained from the three trained models ([19]). . . . . . . . . . . . . . 21
8 Figure taken from [9] in which there is a comparison between different known
edge detection techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
9 Example of outputs obtained by using ArtLine . . . . . . . . . . . . . . . . . 24
10 Output image obtained applying Artline and one of the four models of Learn-
ing to Simplify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
11 Output image obtained applying the XDoG operator . . . . . . . . . . . . . . 26
12 Output image obtained applying Mastering sketching to the output image of
XDoG operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
13 (a) traditional GAN architecture, (b) Style GAN architecture. Image taken
from [22]. The mapping network consists of 8 layers, while the synthesis
network g consists of 18 layers (two for each resolution starting from 42
).
The “A” block stands for the affine transform, and the “B” is the block that
applies per-channel scaling factor to the input Gaussian noise. . . . . . . . . . 30
14 StyleGAN2 architecture. Image taken from [12]. Fig (a) and (b) show the
StyleGAN architecture and its detailed view. In fig (c), the StyleGAN archi-
tecture in which the AdaIN layers have been replaced with the modulation
is illustrated. Fig. (d) shows the revised architecture in which the instance
normalisation has been replaced with a “demodulation” operation. . . . . . . 33
15 Pixel2Style2Pixel architecture. Image taken from [18] . . . . . . . . . . . . . 35
16 ReStyle’s iterative inversion process. Image taken from [2] . . . . . . . . . . . 38
17 Simplified ReStyle encoder architecture. Image taken from [2] . . . . . . . . . 39
18 Average image used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
III](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-6-320.jpg)
![1 Introduction
Deep learning, once considered a marginal field, has become a key player in the rapidly evolv-
ing landscape of Artificial Intelligence (AI). In the past decade, the growth of deep learning
has led to the development of innovative architectures and methods that enable machines
to learn in ways previously thought impossible. From traditional supervised learning to
more advanced forms such as unsupervised and self-supervised learning, to the innovative
techniques of adversarial and reinforcement learning, AI has come a long way. Among these
forms of learning, generative modelling has emerged as a major area of focus and has seen
substantial growth.
Thanks to deep learning, there is now a better understanding of what it means for an AI
to learn and how to implement models that can self-learn. In particular, the advent of
adversarial learning has played a major role in fuelling the AI explosion, since it has a large
set of applications in generative modelling. Generative modelling refers to implementing AI
models that can generate new and varied content through a variety of learning methods.
With the ability to generate new content, AI is changing the way the world is experienced
and understood. In the context of generative modelling, the term “generate” is crucial. When
constructing a generative model, the focus is on creating new content, not simply modifying
or filtering what already exists. This content can be generated randomly or, in certain cases,
it can be generated based on a text or an image. The ability to conditionally control content
generation allows for a greater degree of flexibility and precision in generating content that
meets specific criteria [16].
The ultimate goal of generative modelling is to create new, unique content that was not
present in the original dataset. Generative modelling has numerous applications, including
image synthesis, data augmentation, and music generation, among others. Advances in
generative modelling are constantly being made, and new techniques are being developed,
making it a rapidly growing field of research with numerous opportunities for new discoveries
and breakthroughs.
1.1 Objective of the thesis and research question
Generative Adversarial Networks have gained significant attention in recent years due to
their ability to generate realistic images. However, it is challenging to evaluate the quality
1](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-9-320.jpg)
![be potentially dangerous in case of misused or if applied without considering ethical and
moral implications. Indeed, this technology could be used for the creation of fake identities.
1.3 Recently emerged technology
While conducting this research, similar techniques have emerged, including Stable Diffusion.
Stable Diffusion is a text-to-image model that allows users to easily produce high-quality
artwork in a few seconds based on a text prompt. Traditionally, the Stable Diffusion model
is trained to generate images based on text inputs. However, thanks to a diffusion-denoising
mechanism proposed by [15] the model can also be used for task such as text-guided image-
to-image translation and upscaling. The approach offer a similar solution to the problem
that this study was investigating, but ultimately, the research was carried on.
The results obtained from Stable Diffusion are remarkable due to its extensive training on a
massive dataset and on highly powerful GPUs. The long training time allowed the network
to learn intricate patterns and relationships in the data, resulting in high-quality outputs.
The use of powerful GPUs also ensured a faster training process, allowing the network to
converge quickly to a satisfactory solution.
1.4 Structure of the thesis
This section describes the thesis’s structure. Chapter 1 provides a comprehensive overview
of artificial intelligence and generative models. It also includes the objective and guiding
research question of the thesis, as well as several potential applications of the project.
Chapter 2 of the thesis is dedicated to a comprehensive review of the existing literature on
generative adversarial networks. In this chapter, the reader will gain an understanding of
the background and evolution of these networks.
In chapter 3 a detailed discussion on the creation of the dataset is presented, including
the selection criteria, data pre-processing and preparation processes. This chapter will dig
deeper into the methods used to build the dataset, the reasons for selecting the specific data
and how it was processed. Additionally, it explores the theories behind the technologies
used to process the images.
Chapter 4 provides a comprehensive overview of the specific generative adversarial networks
that are used in the project, including their architecture and training methodologies.
Chapter 5 is dedicated to a discussion about the training and testing setup.
3](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-11-320.jpg)
![2 Literature Review
This chapter provides an in-depth overview of the field of generative modelling. It lays out
the theoretical background of the thesis, beginning with a general explanation of how GANs
work and then exploring some useful theoretical concepts.
2.1 GAN
The innovative concept of Generative Adversarial Networks (GANs) was first introduced in
2014 by Goodfellow and his colleagues from the University of Montreal [7]. GANs are a type
of deep learning model that has the ability to generate new content, such as text or images,
based on a given dataset. GANs have been the focus of significant research in recent years
and have made significant contributions to the field of computer vision. The technology has
advanced in areas like generating realistic images, transforming images, modifying facial
features, and similar topics, but also in other areas such as music generation.
Before the advent of GANs, machine learning algorithms were highly effective in recognising
patterns in existing data and could be utilised for tasks like classification and prediction.
However, generating new data presented a challenge for these algorithms. The solution
proposed in the GAN paper leverages the strengths of machine learning algorithms – their
ability to classify and predict – to overcome their weaknesses in generating new content.
The approach involves setting two algorithms against each other in a kind of competition,
leading to a continuous process of improvement. This approach has proven to be highly
effective in generating new, diverse content and has been widely adopted in the field of deep
learning. Before diving into the intricacies of GANs, it is essential to understand how these
networks work. A widely used analogy can provide a clearer insight.
Imagine there is an art forger and an art authenticator. The forger’s goal is to produce a
counterfeit piece of art that the authenticator will consider genuine. The authenticator is
trained to recognise genuine art by examining real pieces, while the forger learns by creating
art that is then evaluated by the authenticator. In GANs, the forger is the generator, which
strives to produce examples that are indistinguishable from the real data in the training
set, thus trying to fool the discriminator. On the other hand, the discriminator acts as the
art authenticator, evaluating the authenticity of the generated pieces. These two networks
5](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-13-320.jpg)
![are in a constant battle, each trying to outsmart the other. As the generator improves in
creating realistic data, the discriminator needs to enhance its capability in detecting fake
examples. Similarly, as the discriminator becomes better in differentiating true from false,
the generator must strive to generate even more convincing fake data.
The most critical aspect of this architecture is that the generator has no direct connection to
the actual image, its only way to learn is through its interaction with the discriminator [16].
Fig. 1 illustrates a simple GAN architecture based on the analogy.
Figure 1: Example of a GAN architecture
2.1.1 GAN structure and training
A GAN has a similar structure to that of an autoencoder, where the discriminator serves
as the encoder. However, instead of producing an encoded representation, it only outputs
whether the input is real or fake. On the other hand, the generator acts like the decoder
and takes in random data from the latent space, learning to generate new content that can
trick the discriminator into thinking it is real. Both the generator and discriminator are
trained in a competitive manner, with the generator striving to maximise the error of the
discriminator and the discriminator trying to minimise it.
6](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-14-320.jpg)
![The generator and the discriminator are two distinct deep neural networks. The generator
is a function G that takes a random noise vector z as input and maps it to the data space,
producing a set of synthetic samples. These samples are fed into the discriminator network
D, together with the real samples x. This network outputs a single scalar that represents the
probability that the sample came from the real data distribution rather than the synthetic
data distribution. Therefore, the discriminator learns to use traditional supervised learning
techniques, dividing inputs into two classes (real or fake).
The GAN training process is a two-player that involves updating the weights of the generator
and discriminator networks using gradients computed from the loss function. The loss
function, also called cost function, in a GAN is designed to quantify the difference between
the generated data and the real data. The objective is to find the balance between the
generator and discriminator losses, where the generator loss represents the ability of the
generator to produce realistic synthetic data and the discriminator loss represents the ability
of the discriminator to distinguish between real and fake data.
There are several cost functions that may be used, and they usually differ in terms of the
cost function used for the generator.
The basic loss function used in GAN is the binary cross-entropy:
L(ŷ, y) = [y × log(ŷ) + (1 − y) × log(1 − ŷ)] (1)
where y represents the original data and ŷ the generated one. In training, the discriminator
assumes y = 1 for real data, while for the generated data it is set: ŷ = D(x). Substituting
this information in equation 1 the discriminator loss can be obtained:
L(D(x), 1) = log(D(x)) (2)
For the generator’s output data, it is assumed that y = 0, because it outputs fake data,
and ŷ = G(D(z)) for the generated data, where z is a random noise vector sampled from a
noise distribution pz(z) (e.g., uniform or Gaussian distribution). Therefore, the loss for the
generator is:
L(D(G(z)), 0) = log(1 − D(G(z))) (3)
To obtain the total discriminator loss the two equations above have to be combined and
7](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-15-320.jpg)
![maximised:
L(D)
= max[log(D(x)) + log(1 − D(G(z)))] (4)
The generator has to fool the discriminator, hence it has the opposite goal, therefore to
obtain its loss function its minimum has to be calculated:
L(G)
= min[log(D(x)) + log(1 − D(G(z)))] (5)
The loss function is designed to encourage the generator to produce realistic synthetic data
and the discriminator to distinguish between real and fake data with high accuracy.
The interdependence of each player’s cost on the other player’s parameters, along with
the lack of control over the other player’s parameters, makes this scenario more accurately
described as a game rather than an optimisation problem. The loss function in a GAN is an
important component of the training process, as it determines the quality of the generated
data and the ability of the discriminator to accurately distinguish between real and fake
data.
The training process of GANs is a zero-sum game in which the objective function of this
minimax optimisation problem is:
min max V (D, G) = min
G
max
D
Ex∼pdata(x)[log D(x)] + Ez∼pz(z)[log(1 − D(G(z)))] (6)
where x is a real image from the true data distribution pdata, and z is a noise vector
sampled from the noise distribution pz. Ex and Ez denote the expected log likelihood from
the different outputs of both real and generated images.
In training a GAN, it is important to ensure that both the generator and discriminator
models are developed in a balanced manner. The success of the GAN largely depends on
the alignment of the real and generated distributions, and this can only be achieved if neither
of the models is overly dominant. In addition to train the generator and the discriminator,
the errors on their outputs are backpropagated into the models as gradients of the loss
functions, as shown in Fig. 2. These loss functions are used to guide the learning process and
ensure that the generator produces high-quality samples. If the discriminator is too good at
identifying fake versus real images early in the training process, the generator may struggle
to produce convincing fake images, making it difficult for the two models to improve. If the
8](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-16-320.jpg)
![Figure 2: Architecture of a GAN. The generator and the discriminator are synchronously trained.
generator is successful in fooling the discriminator, the loss will backpropagate through the
discriminator network to improve its performance. On the other hand, if the discriminator
effectively distinguishes between fake and real samples, the loss will backpropagate through
the generator network to improve its ability to generate samples that are similar to real
samples. The update rules for the discriminator and the generator are shown in equations 7
and 8 respectively. The weights of each model are represented by θd and θg, respectively,
and m represents the total number of samples in a batch used to test the models before
updating them. Algorithm 0 optimises Eq. 6 as show by [7].
From a mathematical point of view, if the discriminator gets too good at identifying the
samples, this may increase the divergence between the generated expected distribution. In
case the generated and the real distributions become too diverse or do not overlap, then the
generator will suffer from vanishing gradient loss.
The vanishing gradient problem is a critical issue in training deep neural networks. The
problem occurs when the gradients of the parameters with respect to the loss become ex-
tremely small, making it difficult for the optimizers to update the parameters of the network
effectively since they do not have any effect on the training model. The vanishing gradient
problem can lead to slow convergence or complete failure of the training process. One pos-
sible solution to this problem is to use the Wasserstein GAN architecture.
9](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-17-320.jpg)
![Algorithm 1 Minibatch stochastic gradient descent training of generative adversarial nets.
The number of steps to apply to the discriminator, k, is a hyperparameter.
for number of training iteration do
for k steps do
• Sample minibatch of m noise samples {z(1)
, ..., z(m)
} from noise prior pg(z)
• Sample minibatch of m examples {x(1)
, ..., x(m)
} from data generating distribu-
tion ppdata(x).
• Update the discriminator by ascending its stochastic gradient:
∇θd
1
m
m
X
i=1
[log D(x(i)
) + log(1 − D(G(z(i)
))] (7)
end for
• Sample minibatch of m noise samples {z(1)
, ..., z(m)
} from noise prior pg(z).
• Update the generator by descending its stochastic gradient:
∇θg
1
m
m
X
i=1
[log(1 − D(G(z(i)
))] (8)
end for
The gradient-based updates can use any standard gradient-based learning rule.
The goal is to make the expected and generated distribution match as closely as possible.
The GAN continues this process back and forth until an equilibrium is reached between the
generator and discriminator loss. Over time, the two networks learn from each other and
converge to a state where the generator produces samples that are almost indistinguishable
from real data, and the discriminator is able to correctly identify which samples are real
and which are fake with high accuracy.
This basic design is referred to as the Vanilla GAN, as it represents the original model
from which more advanced GAN architectures have evolved, yet still retaining the two key
components of the GAN structure.
2.2 Wasserstein-GAN
The Wasserstein-GAN, also known as WGAN, is a variant of the Vanilla GAN and it was
introduced by Arjovsky et al. in 2017 [4]. WGAN aims to improve the generator model’s
capability of approximating the data distribution of a given training dataset by introducing
a different training mechanism. Instead of using a discriminator to classify the generated
10](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-18-320.jpg)
![images as real or fake, WGAN uses a critic that scores the authenticity of an image. It uses
the Wasserstein distance (also known as Earth-Mover’s distance), which is a metric that
quantifies the minimum cost of moving mass from one distribution to another to convert
the source distribution q to the target distribution p. It is used to measure the distance
between the distribution of real data and the distribution of generated data.
In WGAN, the discriminator is trained to be a function that outputs a scalar value instead
of a probability, and the generator is trained to minimise the Wasserstein distance.
One formulation for the Wasserstein distance is:
W(Pr, Pg) = sup
f∈F
Ex∼Pr [f(x)] − Ex∼Pg [f(x)] (9)
where F is defined as the set of 1-Lipschitz functions, which are those satisfying ∥f∥L ≤ 1
where ∥ · ∥L represents the Lipschitz norm.
The WGAN objective function can be formulated as:
min
G
max
D∈F
Ex∼Pr
[D(x)] − Ex̃∼Pg
[D(x̃)] (10)
where Pr and Pg are respectively the real and generated distribution, x̃ = G(z) with
z ∼ p(z). The above formulation needs the critic to be 1-Lipschitz. This condition is
ensured by clipping the weights W of the discriminator to be within a compact range of
[−c, c], where c is a user-defined hyper-parameter. Specifically, each weight wi,j is clipped
to this range.
WGAN-GP
One step towards stable training of GANs is made by WGAN, however, it may still generate
inadequate samples or not converge. The cause of such issues is frequently attributed to
the utilisation of weight clipping in WGAN, which enforces a Lipschitz constraint on the
critic and can result in undesirable behaviour, for example, the convergence of a very deep
WGAN critic may often fail. Therefore, to address this issue an alternative to weight clipping
has been proposed by [8] to ensure smooth training, which is the Wasserstein GAN with
Gradient Penalty (WGAN-GP). The WGAN-GP method replaces weight clipping with a
gradient penalty, introducing an extra loss term that encourages the discriminator to have
11](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-19-320.jpg)
![a gradient with a norm that is near 1. The resultant optimization problem is given by:
L = Ex̃∼Pg
[D(x̃)] − Ex∼Pr
[D(x)] + λEx̂∼Px̂
[(∥∇x̂D(x̂)∥2 − 1)2
] (11)
where the first two terms are the original critic loss of the WGAN, while the last term is the
gradient penalty, in which λ defines the gradient penalty coefficient, x̂ is a random sample
along the straight line between real and generated samples and px̂ is the distribution of x̂.
2.3 ProGAN
ProGAN, short for Progressive Generative Adversarial Network, is a deep learning model
that was developed to generate high-resolution, photorealistic images. It was introduced by
Karras et al. in 2017 [21], it is a type of generative model, which means that it can generate
new, synthetic images based on the data it is trained on.
The main idea behind ProGAN is to use a series of generators and discriminators to
generate high-resolution images incrementally, starting from low-resolution images (e.g.
4×4) and gradually increasing the resolution until a high-resolution image is obtained (e.g.
1024×1024). This approach is called progressive growing, and it allows the network to learn
the finer details of an image in an incremental manner. The generator starts with a random
noise input and generates an image with low resolution, which is then evaluated by the
discriminator. The discriminator determines whether the generated image is real or not and
provides feedback to the generator, allowing it to improve.
One of the benefits of ProGAN is that it generates high-resolution images that are photore-
alistic, which makes it useful for various applications. Additionally, ProGAN is trained on
large datasets, which means that it can create diverse and varied images. Another advantage
of ProGAN is its ability to produce images at different resolutions, which allows for a level
of control over the generated images.
Some drawbacks of the progressive growth method in a GAN are the prolonged training time
involved in constructing subsequent models at higher resolutions, the need for extra resources
in terms of data preparation and storage, and of course the non-negligible computational
time.
12](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-20-320.jpg)
![2.4 Evaluation metric
2.4.1 xCos
The state-of-the-art face verification models employ a methodology that involves extracting
deep features from a pair of face images and subsequently computing the cosine similarity
or the L2-distance of the paired features. The determination of whether the two images are
of the same person is based on whether the similarity value exceeds a pre-defined threshold
value. However, the approach proposed in the paper “xCos: An Explainable Cosine Metric
for Face Verification Task” [13] offers a distinctive approach in which the verification result is
obtained by the integration of the local similarity map and the attention map. It is based on
the insight that humans tend to compare different facial features to determine whether two
face images belong to the same individual. xCos leverages this understanding to evaluate
the similarity of facial features between two images. It is built using a grid-based feature
extraction approach, in which each face image is divided into multiple local regions. The
facial features of each local region are then extracted and compared with the features of
the corresponding regions in the other image. The resulting similarities are then aggregated
across all the local regions to compute a final similarity score. It also includes an atten-
tion mechanism that identifies the specific facial features that contribute the most to the
similarity score. This attention mechanism allows for a more interpretable and explainable
verification result, as it highlights the specific facial features that the model is prioritising.
The proposed architecture, showed in Fig. 3, for face recognition consists of a modified
CNN backbone and two branches for xCos and identification. The primary responsibility
of the CNN backbone is to extract facial features for each identity. To preserve the position
information of each feature point, the final flatten and fully-connected layers of the backbone,
such as ArcFace [5] or CosFace [24], are replaced with a 1 by 1 convolution. On the xCos
branch, to compute a patched cosine map S (cospatch), the two feature maps of compared
images are element-wise evaluated using cosine similarity. At the same time, an attention
weight map W is generated using the attention mechanism based on the two feature maps.
The final xCos similarity value is achieved by performing a weighted sum of the patched
cosine map S, using the attention weight map W.
The xCos branch is trained using cosine similarity computed by another face recognition
13](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-21-320.jpg)
![Figure 3: Example of the xCos architecture taken from [13]
model, such as ArcFace. The identification branch flattens the extracted feature and feeds
it into another fully connected layer for predicting the identity. The training is stabilised
by using a common face recognition loss, such as the one used in ArcFace, which is denoted
as LID.
14](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-22-320.jpg)
![3 Dataset preparation
This chapter provides an in-depth discussion on the process of the dataset creation, as well
as the dataset utilised in the experimentation. Furthermore, the theoretical foundations of
the tools employed in the dataset creation are comprehensively presented.
3.1 Dataset selection
Since the objective of this thesis is to generate a realistic photo of a face starting from a
handmade sketch, a neural network had to be trained on a large dataset of face images
and their corresponding sketches. However, finding such a dataset was not easy. Online
the CUHK dataset (The Chinese University of Hong Kong dataset) [1] is available, it is a
collection of images used for various computer vision and machine learning tasks, such as
image classification, object detection, and face recognition. The specific content and size of
the CUHK dataset may vary depending on the source. The dataset includes the CUHK Face
Sketch FERET Database (CUFSF) [25, 26] which is composed of 1194 pair of images with
both a photo of a face and a sketch of it. Each face includes an artist-drawn sketch based on
a photo that was taken with the individual in a neutral expression, under normal lighting
conditions, and in a frontal pose. This database is made for the purpose of researching face
sketch synthesis and face sketch recognition. Unluckily, it does not contain enough images,
and it requires data augmentation techniques in order to use it.
Another dataset available online is the FFHQ (Flickr-Faces-HQ) dataset [17], which is a
large-scale, high-quality dataset of face images, created to advance research in generative
models and computer vision. The dataset contains 70 000 high-resolution (1024 × 1024) im-
ages of diverse human faces, each with different poses, lighting, expressions, and ethnicities.
Since it is a very large dataset, it can be used to create the needed dataset composed of
pairs of sketches and photos, but it requires some image processing techniques in order to
obtain the sketches.
The challenge here is to find a technique to transform an image into a sketch. The idea is to
extract the face contours using an edge detection technique, while preserving the essential
details of the face and making the result look like a hand-drawn sketch.
To achieve these results, two approaches were considered to build the dataset. The first
approach involves using two separate networks: Artline [14] and a network for the sketch
15](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-23-320.jpg)
![simplification task. In the second approach, it is also used a network for simplifying the
sketches, but instead of using a neural network to obtain the sketches, the extended difference
of Gaussian (XDoG) edge detector [9] is applied. In order to produce the desired results, both
approaches need to be carefully analysed and compared. In this way, the most appropriate
approach can be chosen and further developed to create a realistic handmade sketch.
Now the theoretical foundation underlying the techniques that are employed for constructing
the dataset are presented before discussing how they have been used to reach the final goal.
3.2 ArtLine
ArtLine is a deep neural network designed specifically to turn photographs into line art
drawings. It is a recent project developed by Vijish Madhavan [14] interested in generative
modelling, that follows the discoveries found by Jason Antic in his DeOldify project [3],
utilising advancements and tools developed by the team at Fast.ai. ArtLine, as DeOldify, is
based on a self-attention GAN (SAGAN) that is progressively resized to achieve a desired
resolution. SAGAN was introduced in a 2018 research paper by Zhang et al. [30] and from
then on it has been applied to a wide range of image generation tasks, including facial image
synthesis, artistic image generation, and more. The results have been very promising, with
SAGAN producing high-quality and highly detailed images that are comparable to those
produced by state-of-the-art methods.
The self-attention mechanism allows the generator network to weigh the contribution of
different parts of an image in generating the final result. This allows the network to focus
on important features of the image and produce more accurate results. Additionally, the
use of self-attention also allows the network to better capture the relationships between
different parts of the image, resulting in more coherent and visually appealing images.
ArtLine has been trained on the APDrawing dataset, which is a dataset obtained from
another generative adversarial network able to generate high-quality sketch drawings from
a given image, called APDrawingGAN [28]. The APDrawingGAN is an innovative GAN-
based architecture that has the capability to generate professional-level artistic portraits.
It uses a combination of a global network and local networks, which allows for dedicated
drawing strategies to be learned for specific facial features. This leads to a more precise and
accurate portrayal of facial features in the generated drawings. To train APDrawingGAN, a
unique artistic drawing dataset was constructed, containing high-resolution portrait photos
16](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-24-320.jpg)
![and corresponding professional artistic drawings. The results of extensive experiments and a
user study demonstrate that APDrawingGAN produces significantly better artistic drawings
than existing methods. Since the results obtained using this dataset were good only in the
case of ID photos, the author of ArtLine combined this dataset with an Anime sketch dataset
to obtain good results also with other kinds of pictures.
The training process of ArtLine is similar to that of a NoGAN, which is a type of genera-
tive model that trains without using the discriminator network. Therefore, the initial loss
function for the generator is based on the features extracted from a pre-trained VGG model.
3.3 Sketch simplification
3.3.1 Learning to simplify
The model Learning to Simplify (LtS) [20] consists of a Fully Convolutional Network (FCN)
to learn the mapping between rough sketches and clean drawings. The technique of simpli-
fying rough sketches utilises Convolutional Neural Networks (CNNs) to simplify the image.
Through a series of convolution operations, the sketch lines are grouped, and a simplifica-
tion is directly produced. The CNNs are trained using a unique dataset of rough sketches
and their corresponding simplifications, which the authors have created. This data-driven
method has several benefits. Firstly, the model learns automatically from the training data
the necessary heuristics for sketch simplification. Secondly, since convolution operations can
be executed efficiently on GPUs, the processing time is less than a second for most of the
images. In contrast to typical CNN architectures, which utilise fully connected layers, their
approach uses solely convolutional layers with sparse connections. These sparse connections
allow their CNN to process images of varying resolutions and aspect ratios with ease. The
results of the study show that the FCN is able to produce high-quality clean drawings from
rough sketches with remarkable accuracy.
The model has three main components: an encoder that compresses the image, a processing
unit that extracts the critical lines, and a decoder that converts the simplified represen-
tation back into a greyscale image of the same resolution as the input (Fig. 4). All these
components use convolutional layers for their processing.
The model uses a down- and up-convolution architecture that may look like simple filter
banks. It has a larger number of channels in lower-resolution parts, such as 1024 channels
17](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-25-320.jpg)
![Figure 4: Learning to Simplify architecture. Image taken from [20]
when the size is 1/8, to ensure that the necessary information for clean lines is carried
through. The model uses padding in its convolutional layers to guarantee that the output
has the same size as the input when a stride of 1 is used. Instead of pooling layers, the authors
of this model used convolutional layers with a higher stride to lower the resolution from the
previous layers. To increase the resolution and ensure that the output of the model has the
same dimensions as the input, they used fractional strides. Overall, the model leverages the
ability of CNNs to learn and extract essential information, making it an effective solution
for sketch simplification. The LtS model has been trained by the authors using pairs of
rough and simplified sketches representing the input and the target, respectively, and they
used the weighted mean square error criterion as loss:
l(Y, Y ∗, M) = ∥M ⊙ (Y − Y ∗)∥2
F RO (12)
where Y is the model output, Y ∗ is the target output, M is the loss map, ⊙ is the matrix
element-wise multiplication or Haddamard product, and ∥ · ∥F RO is the Frobenius norm.
They selected a loss map that minimises loss on the thicker lines, to prevent the model from
primarily focusing on the thicker lines while neglecting the thinner lines. The loss map is
built by examining the histogram surrounding each pixel in the target label (ground truth)
and it is defined as:
M(u, v) =
1 if I(u, v) = 1
min(α exp(−H(I, u, v)) + β, 1) else
(13)
18](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-26-320.jpg)
![The value of the local normalized histogram bin in which the pixel (u, v) of image I falls
into is represented as H(I, u, v).
An example of the results that can be obtained with this approach can be seen in Fig. 5.
Figure 5: Figure from [20] showing the results of Learning to Simplify approach.
3.3.2 Mastering Sketching
Mastering sketching [19] is another approach for training networks on the sketch simplifi-
cation task, similar to Learning to Simplify (3.3.1). The authors of the paper introduce
adversarial augmentation, a fusion of unsupervised and adversarial training for structured
prediction tasks. The proposed method can improve the quality and generalization of exist-
ing networks, while also permitting the use of unlabelled examples. The authors explained
that the existing methods to achieve the sketch simplification task requires large amounts of
labelled data, which is time-consuming and an expensive process. Additionally, the resulting
data may not accurately represent the true rough sketches drawn without such references,
and the availability of such data is often limited, which can lead to overfitting or poor gen-
eralization. The proposed method aims to address these issues by leveraging unsupervised
data and adversarial training.
Their approach combines a fully convolutional network for sketch simplification, which is
the same as the one used in the Learning to Simplify, with a discriminator network that is
able to distinguish real line drawings from those generated by the network. The training
of the sketch simplification network involves two objectives: simplifying the input sketches
and fooling the discriminator network by generating sketches that are indistinguishable from
real ones. The use of the discriminator network allows the simplification network to learn
19](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-27-320.jpg)
![the structure of real sketches and generate simplified versions that are similar in appearance
to the training sketches.
They trained a variation of a conditional GAN where, instead of using a random input z, they
used a deterministic prediction S : x 7→ y = S(x). For the supervised adversarial training
they trained together the prediction model S and the discriminator model D : y 7→ D(y) ∈ R
which is not conditioned on the input x, since they assumed that S is associated with a
supervised training loss L(S(x), y∗
). The training consists in maximising the output of D
while at the same time minimising the loss of the model with the structured output, resulting
in the following optimisation:
min
S
max
D
E(x,y∗)∼ρx,y
[α log D(y∗
) + α log(1 − D(S(x))) + L(S(x), y∗
)] (14)
where ρx,y is the supervise training data consisting of input-output pairs, α is the weighting
hyperparameter, y and y∗
are the output image and the ground truth data corresponding
to the input x.
From equation 14, it can be noted that the first term in the expectation value depends only
on y, while the second depends only on x. This indicates that it can be done a training
using these terms separately with unsupervised data. By separating the first two terms in
the expectation value of equation 14, then the supervised adversarial objective function can
be used to incorporate the unsupervised data into the training.
Under the conditions that there are available an extensive collection of input data ρx, output
data ρy, and a dataset ρx,y consisting of fully-annotated pairs, then the optimisation function
can be adjusted as follows:
min
S
max
D
E(x,y∗)∼ρx,y
[L(S(x), y∗
) + α log D(y∗
) + α log(1 − D(S(x)))]
+βEy∼ρy
[log D(y)] + βEx∼ρx
[log(1 − D(S(x))]
(15)
where β is a weighting hyperparameter for the unsupervised data term.
For the simplification network, they did not use the weighted MSE loss as in 3.3.1, but
instead they used the MSE loss defined as:
L(S(x), y∗
) = ∥S(x) − y∗
∥2
(16)
20](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-28-320.jpg)
![where ∥ · ∥ is the Euclidean norm, S(x) is the deterministic prediction of a given input x,
y and y∗
are the output image and the ground truth data corresponding to the input x,
respectively. The results using this approach can be seen in Fig. 6.
Figure 6: Figure from [19] that highlights the comparison between the Mastering sketching’s
results and the ones of the Learning to Simplify.
The authors also proposed a pencil drawing generation solution that is the inverse problem
of sketch simplification. Indeed, they swapped the input and output data used for the
sketch simplification and trained three models: one with the MSE loss and two models with
adversarial augmentation for different artists, one based on an artist with a dirty and faded
pencil style and the other based on an artist with clearer pencil style. Fig. 7 illustrates the
results of the three models.
Figure 7: Results obtained from the three trained models ([19]).
21](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-29-320.jpg)
![3.4 Extended Difference of Gaussian
The Extended Difference of Gaussian (XDoG) is an edge detector operator based on the
Difference of Gaussian (DoG). It comes from the idea that edges in an image correspond
to rapid changes in intensity, which can be detected by subtracting two images that have
been smoothed with different Gaussian kernels. It can be used to increase the visibility of
edges and other features in digital images. The capacity of the DoG operator to extract
information about the edges in the image can be explained by considering the Gaussian
filter from a signal processing point of view. The Gaussian filter is defined as:
Gσ(x) =
1
2πσ2
e− x2
2σ2
(17)
where x refers to a two-dimensional coordinate, and σ represents the standard deviation
of the Gaussian distribution in the spatial domain. It is a low-pass filter which attenuates
or eliminates high frequencies. The difference between the two Gaussian with different σ
is equivalent to a subtraction between a blurred version of the original image and a less
blurred version of it, and it is defined as:
Dσ,k(x) = Gσ(x) − Gkσ(x) ≈ −(k − 1)σ2
∇2
G (18)
where the standard deviations of these Gaussian filters determine the scale at which edges are
detected, with larger standard deviations detecting larger edges. This difference is a band-
pass filter which attenuates all frequencies that are not between the cut-off frequencies of
the two Gaussians. This results in a new image that highlights regions of the original image
where the intensity changes rapidly.
The XDoG approach is inspired by biological models proposed by Young et al. [29], which
were based on the DoG model. Winnemöller et al. [27] extended the DoG model to allow
for greater control over the inhibitory effect of the larger Gaussian, which is allowed to vary,
resulting in a more advanced edge detection technique with the following equation:
Dσ,k,τ (x) = Gσ(x) − τ · Gkσ(x) (19)
This allows for more fine-grained control over the output, resulting in more precise edge
detection and image stylization. In particular, it is possible to achieve a wider range of
22](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-30-320.jpg)
![stylistic effects using the following continuous ramp function:
Tϵ,φ(u) =
1 u ≥ ϵ
1 + tanh(φ · (u − ϵ)) otherwise
(20)
Equations 19 and 20 taken together is referred to as the XDoG filter Tϵ,φ(Dσ,k,τ ∗ I) for a
given image I.
There are a lot of known edge detectors that can be applied to find the edges of objects
present in an image, and they have their advantages and disadvantages (as shown in [9]).
Since the objective was not only to extract contours but also to obtain something that looks
like a sketch done by an artist, not all the edge detectors that are known could be applied.
As can be seen from Figure 8, some edge detectors work well in extracting edges, like the
Sobel filter and the Canny edge detector, but the results obtained are not satisfactory from
an artistic point of view. Hence, the XDoG operator has been chosen since it satisfies the
final purpose.
Figure 8: Figure taken from [9] in which there is a comparison between different known edge
detection techniques.
23](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-31-320.jpg)
![4 Implementation
In this chapter, the generative adversarial networks used to generate synthetic images start-
ing from a sketch are described.
4.1 StyleGAN
StyleGAN [22] is a state-of-the-art deep learning generative model developed by NVIDIA in
2018 to produce realistic-looking images. The model is based on Generative Adversarial Net-
work (GAN) architecture, in particular, it is built upon the foundation of the ProGAN (2.3),
and it takes inspiration from the concept of “style transfer” introduced by [6] to generate
unique images. An approach that involves the use of neural representations to separate and
recombine the content and style of images.
StyleGAN is designed to generate high-resolution images of faces, but can also be adapted
to generate other types of images as well. Its key feature includes the ability to generate
high-resolution images while also controlling various aspects of the image style, such as
colour, texture, and overall composition.
The main idea behind StyleGAN is to use a deep neural network that is trained to generate
images from a random noise input. This noise input is then transformed into a feature
representation, which is fed into a generator network. The generator network uses a series
of convolutional layers to upsample the feature representation and generate the final image.
The generator in StyleGAN begins by using a learned constant input and then modifies the
style of the image in every convolution layer based on the latent code. This direct control
allows for the adjustment of the strength of the image’s features at various scales. The
inclusion of noise directly injected into the network results in the automatic, unsupervised
separation of high-level attributes such as pose and identity from stochastic variations like
freckles and hair in the generated images. This design makes it possible to perform intuitive
mixing and interpolation operations that are specific to different scales.
The design of StyleGAN is different from the one of the traditional GANs, since the latent
code is not provided to the generator through an input layer. However, the generator starts
by using a 4 × 4 × 512 constant to start the image synthesis process, and uses a non-linear
mapping network to map the input latent code z to an intermediate space. The non-linear
28](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-36-320.jpg)
![allows for a more linear and less entangled representation of various factors of variation.
The output of the mapping network is 512 × 1, the same size as the input layer. The Style-
GAN generator model is now referred to as the “synthesis network” due to the integration
of the new mapping network into its architecture, its architecture can be seen in Fig. 13(b),
while Fig. 13(a) shows a traditional GAN architecture.
Figure 13: (a) traditional GAN architecture, (b) Style GAN architecture. Image taken from [22].
The mapping network consists of 8 layers, while the synthesis network g consists of 18 layers (two
for each resolution starting from 42
). The “A” block stands for the affine transform, and the “B” is
the block that applies per-channel scaling factor to the input Gaussian noise.
The AdaIN (Adaptive Instance Normalization) layer [10] is an extension of the Instance
Normalization layer which can adapt to different styles instead of being able to normalise
to a single, specified style. The AdaIN layer integrates information from a learned encoding
of the style of the image into the generator network to allow for control over the style of the
generated image. This layer is added at each resolution level of the synthesis network to
provide a unique visual expression of features at each level. The AdaIN modules shift and
scale the output through the layers in the synthesis network, promoting the significance of
relevant filters in each convolutional layer. This helps the generator differentiate between
important and insignificant features. It can be considered as an internal feedback mecha-
30](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-38-320.jpg)
![The experiment conducted by Karras et al. [22] shows that the redesign of the generator not
only preserves image quality, but also it significantly enhances it. To demonstrate this, the
Fréchet Inception Distance (FID) was computed for various generator architectures on a few
available datasets. The authors’ results suggest that a style-based design is superior to the
conventional GAN generator architecture in all aspects. This superiority is demonstrated
through established quality metrics, and their findings on the separation of high-level fea-
tures and stochastic effects, as well as the linearity of the intermediate latent space, all
point towards a deeper understanding and control of GAN synthesis. Furthermore, their
path length metric can be included as a regularisation term during training, and a similar
approach could be applied to the linear separability metric.
4.2 StyleGAN2
StyleGAN2 (2020) is an improvement of (4.1) for synthesising high-resolution images de-
veloped by NVIDIA. This new version is based on the previous one, but with some dif-
ferences [12]. The authors of StyleGAN2 discovered that styles were not only capable of
capturing and transferring the essence of an image, but also transferring visual distortions
such as water droplets or smudges. They linked these visual distortions to the AdaIN lay-
ers, and therefore they made some changes to them in order to eliminate the artefacts. To
overcome this issue, they came up with a new approach to the AdaIN layers, which were
previously used as direct inputs to the network. Instead, they integrated the AdaIN layers
within the convolutional layers.
The authors replaced the AdaIN operator with a weight modulation and demodulation step,
where the modulation adjusts each feature map of the convolution based on the style.
The authors discovered that the strong location preference for facial features in StyleGAN
images was due to progressive growing. To address this, they took inspiration from the
“Multi-Scale Gradients for Generative Adversarial Networks” paper by Animesh Karnewar
and Oliver Wang [11] and incorporated the idea of multiple scale gradient using a single
architecture. They developed a new architecture, which can be seen in Fig. 14, that employs
multiple scales of image generation, which is achieved through the utilisation of a resnet-
style skip connection that links lower resolution feature maps to the final generated image.
A further difference between the two version of StyleGAN is the introduction of a normali-
32](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-40-320.jpg)
![Figure 14: StyleGAN2 architecture. Image taken from [12]. Fig (a) and (b) show the StyleGAN
architecture and its detailed view. In fig (c), the StyleGAN architecture in which the AdaIN layers
have been replaced with the modulation is illustrated. Fig. (d) shows the revised architecture in
which the instance normalisation has been replaced with a “demodulation” operation.
sation term to the loss function to make the latent space more uniform and smoother. By
doing so, the mapping of images in and out of the latent space becomes more controlled, en-
abling more controlled image generation. Additionally, this improved relationship between
the latent space and generated images allows creating images along a path in the latent
space. The regularisation term added to the loss is driven by the goal of maintaining the
expected lengths of vectors regardless of their directions, and it is formulated as:
Ew,y∼N (0,1) = (∥Jw
T
y∥2 − a)2
(22)
where y are random images with normally distributed pixel intensities, w ∼ f(z), where z
are normally distributed. The local metric properties of the generator’s mapping function
g(w) : W → Y are represented by the Jacobian matrix Jw = ∂g(w)/∂w. The value
of the constant a is dynamically determined during the optimisation process through the
calculation of the long-term exponential moving average of the vector lengths ∥Jw
T
y∥2. This
allows the optimisation to automatically find an appropriate global scale for the process.
The regularisation term is known as path length regularisation, and its implementation has
proven to be highly beneficial in practice, leading to the development of more stable and
consistent models. This makes the process of exploring and experimenting with different
architectural designs much more manageable. Additionally, it has been noted that the
smoother nature of the generator as a result of this regularisation makes it significantly
33](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-41-320.jpg)
![easier to perform inversion.
Additionally, the authors [12] observed that there was no need to compute the regularisation
term every time the loss was computed, therefore in this way they could obtain a decrease
in the computational cost and also in the overall memory usage. They applied a lazy
regularisation and evaluate the regularisation terms every k training iterations.
To summarise the improvements made in the second version of StyleGAN are:
1. weight demodulation
2. path length regularisation
3. lazy regularisation
4. no growing
These improvements made it also possible to speed up the learning time of the model up to
40% and optimise the images quality.
4.3 pSp Framework
Pixel2Style2Pixel (pSp) is a novel image translation framework that leverages the powerful
representation capabilities of a pre-trained StyleGAN2 generator and the extended W+
latent space [18]. The pSp framework is designed to tackle the task of transforming an
input image into a target image, preserving the content of the original image while adopting
the style of another. In the paper, a new encoder network is proposed, it is based on a
Feature Pyramid Network capable of directly encoding real images into the W+ space.
The encoder that the authors built is able to directly reconstruct real input images, enabling
latent space manipulations without the need for time-consuming optimisation. However,
this approach is limited as the input image must be invertible, meaning that it must have
a latent code that can reconstruct it. This is a critical limitation since in conditional image
generation, the input image does not stay in the same StyleGAN domain. The solution
proposed is to utilise the encoder jointly with the pre-trained StyleGAN generator as a
complete solution for image-to-image translation, thereby overcoming the limitation of the
encoder’s requirement.
The input images are transformed into the desired output latents through direct encoding,
which are then processed by StyleGAN to generate the desired output images. In this
34](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-42-320.jpg)
![way, StyleGAN can be used for image-to-image translation also when the input and output
images belong to different domains.
The straightforward way to encode an input image into the W+ is to extend the encoder
backbone with a feature pyramid which allows to produce three levels of features maps of
spatial resolution (coarse, medium and fine). The styles are extracted from different scales
of the pyramid using an intermediate network called map2style, and are inserted directly
into the fixed, pre-trained StyleGAN2 generator in correspondence to their scale to generate
the final output image. The map2style is a small fully convolutional network which reduces
the spatial size using a set of 2-strided convolutions followed by LeakyReLU activation
functions. It is trained to extract the learned style from the corresponding feature map for
each of the 18 target styles. Figure 15 illustrates the pSp architecture.
Figure 15: Pixel2Style2Pixel architecture. Image taken from [18]
Additionally, they wanted to design the encoder to learn the latent code based on an average
style vector in order to improve the initialisation. Therefore, they defined an average style
vector w̄ of the pretrained generator, and given an input image x the output of their model
is defined as:
pSp(x) := G(E(x) + w̄) (23)
where E(·) is their encoder while G(·) is the StyleGAN2 generator.
For the loss function, the authors used a weighted combination of several functions, since
they observe that this combination allowed to obtain a more accurate encoding into Style-
GAN, defined as:
L(x) = λ1L2(x) + λ2LLP IP S(x) + λ3Lreg(x) + λ4LID(x) (24)
35](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-43-320.jpg)
![where λ1, λ2, λ3, λ4 are constants used to define the loss weights, and L2 loss is the pixel-wise
loss:
L2 = ∥x − pSp(x)∥2 (25)
LLP IP S is the LPIPS loss to learn perceptual similarities, with F(·) denoting the perceptual
feature extractor:
LLP IP S = ∥F(x − pSp(x))∥2 (26)
Lreg is the regularisation loss to guide the encoder to produce latent style vectors that are
closer to the average latent vector:
Lreg = ∥E(x) − w̄∥2 (27)
and finally LID is a loss measuring the cosine similarity between the output image and its
source since their aim was to preserve the input identity:
LID(x) = 1 − ⟨R(x) − R(pSp(x))⟩ (28)
where R represents the ArcFace pretrained network [5], which is a type of loss function that
aims to increase the separability between classes by adding a margin to the cosine similarity
between features of an image and the weight vector of the classifier.
4.4 Restyle
In the area of unconditional image synthesis, Generative Adversarial Networks have recently
achieved outstanding results. It is essential for trained GANs to be able to invert an image
into its matching latent code, since doing so allows manipulating real-world images while
exploiting the network’s rich semantics. One of these models is StyleGAN, which is capable
of synthesising highly realistic images, as discussed in 4.1. However, there is a major draw-
back to StyleGAN: it only allows for the synthesis of new images in a specific style, meaning
that it cannot change the style of an existing image.
This limitation has motivated the development of the ReStyle algorithm [2], which is a
residual-based StyleGAN encoder that can change the style of an existing image. The
ReStyle algorithm is based on the iterative refinement of a residual encoding that transforms
36](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-44-320.jpg)
![the input image into a feature representation compatible with the StyleGAN generator. In
this way, ReStyle permits to transfer the style from a pre-trained StyleGAN2 model to a
given input image.
In the paper “ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement” [2],
the authors presented a new encoder-based inversion approach for a more accurate image
inversion. They observed that achieving precise inversion in a single attempt inflicts a
heavy constraint on the training process, so they introduced a novel method, called ReStyle,
that employs an iterative process to carry out the inversion. This method uses a feedback
mechanism to repeatedly feed the output of each iteration back into the encoder together
with the original input image. This enables the encoder to use the knowledge gained from
previous iterations to focus on the important parts necessary to reconstruct the input image
with a certain level of accuracy.
The objective is to train an encoder E capable of inverting real-world images into the
extended W+ latent space of a previously trained StyleGAN generator G. Where the goal
must be achieved through multiple steps N > 1, with each step consisting of a single forward
pass through both the encoder and the StyleGAN generator.
From a mathematical point of view, the purpose was to generate an image ŷ = G(E(x)),
where x is a given input image, in such a way that ŷ ≈ x. To reach this goal, the encoder
was trained while the StyleGAN generator remained fixed, since it is pre-trained.
ReStyle training
To train the encoder network and enable it to perform the inversion task, a set of losses are
introduced. The losses used, which are also used by most encoder-based methods, are a L2
loss, which measures the difference between the reconstructed image and the original image
pixel-wise, and a perceptual loss, such as LPIPS, which measures the difference between
the two images in terms of their high-level features. The training process of the encoder is
guided by the losses, which are computed at each forward step. The encoder weights are then
updated according to the computed losses via back-propagation. In the inversion process at
step t, ReStyle operates by concatenating the current prediction of the reconstructed image
ŷ with an input image x. This results in an extended 6-channel input, represented as xt:
xt := x∥ŷt (29)
37](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-45-320.jpg)
![At this point the encoder E has to compute the residual code based on the latent code just
predicted, and then it updates the new prediction for the latent code using the previous
latent code wt, as shown in equations 30 and 31, respectively.
∆t := E(xt) (30)
wt+1 ←
− ∆t + wt (31)
The updated prediction of the reconstructed image is obtained by passing the latent code
wt+1 to the generator, as defined in equation 32. Finally, the updated estimate ŷt+1 is used
in the following step, as specified by equation 29.
ŷt+1 := G(wt+1) (32)
The initialization of the inversion process consists in starting with an average latent code
w0 and its corresponding image ŷ0, the overall process can be seen in Figure 16.
Figure 16: ReStyle’s iterative inversion process. Image taken from [2]
ReStyle Encoder architecture
The ReStyle encoder (Fig. 17) is designed as a variation of the architecture described in 4.3
and by [23]. Instead of obtaining the style features from three intermediate levels in the
encoder, all style vectors are derived from the final 16×16 feature map. To do so, k different
map2style blocks are employed to reduce the size of the feature map and produce the
corresponding 512-dimensional style input, where k is the number of style inputs in the
StyleGAN generator, as introduced in 4.3.
38](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-46-320.jpg)
![Figure 17: Simplified ReStyle encoder architecture. Image taken from [2]
ReStyle conclusion
The ReStyle scheme has been shown to produce highly realistic results in a variety of
experiments. It is capable of transforming the style of an image while preserving its content,
which is a crucial property for style transfer algorithms. Additionally, the ReStyle algorithm
is highly flexible, as it can be trained on different styles by using different pre-trained
StyleGAN models. This allows for the creation of a large variety of style transfer models,
each with a unique style.
39](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-47-320.jpg)
![5 Setup for training and testing
Since the goal of this thesis is not to create a new type of generative adversarial network, a
pre-existing one have been utilised. The aim was to train the existing network, specifically
the ReStyle scheme, discussed in 4.4, over a variation of the encoder proposed in the pSp
framework (4.3), to generate high-quality face images when conditioned with face sketches.
The dataset used to train the model is the one obtained utilising the techniques explained
in 3.
5.1 Training’s setup
The official version of ReStyle published by Alaluf [2] is used as a base model to perform
the training.
The model has been trained on NVIDIA GRID A100D-2-20C GPU, with 11.4 CUDA, and
20GB of memory.
The training process involves setting up the dataset to be used with ReStyle, therefore the
following updates in the ReStyle directory are needed. The config/path_configs.py has to
be update indicating the position where the images, for both training and testing, are saved:
dataset_paths = {
’s_train ’: ’ffhq/sketch/’,
’f_train ’: ’ffhq/orig/’,
’s_test ’: ’ffhq/sketch_test /’,
’f_test ’: ’ffhq/orig_test/’
}
The configs/paths_config.py has to be updated, defining the path to the images that have
to be used for training and testing. The dataset has to be configured in this way:
DATASETS = {
’sketch_to_face ’: {
’transforms ’: transforms_config . SketchToImageTransforms ,
’train_source_root ’: dataset_paths [’s_train ’],
’train_target_root ’: dataset_paths [’f_train ’],
’test_source_root ’: dataset_paths [’s_test ’],
’test_target_root ’: dataset_paths [’f_test ’],
}
}
40](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-48-320.jpg)
![In the config/transforms_configs.py this code has to be added:
class SketchToImageTransforms ( TransformsConfig ):
def __init__(self , opts):
super( SketchToImageTransforms , self).__init__(opts)
def get_transforms (self):
transforms_dict = {
’transform_gt_train ’: transforms.Compose ([
transforms.Resize ((256 , 256)),
transforms.ToTensor (),
transforms.Normalize ([0.5 , 0.5, 0.5] , [0.5 , 0.5, 0.5]) ]),
’transform_source ’: transforms.Compose ([
transforms.Resize ((256 , 256)),
transforms.ToTensor ()]),
’transform_test ’: transforms.Compose ([
transforms.Resize ((256 , 256)),
transforms.ToTensor (),
transforms.Normalize ([0.5 , 0.5, 0.5] , [0.5 , 0.5, 0.5]) ]),
’transform_inference ’: transforms.Compose ([
transforms.Resize ((256 , 256)),
transforms.ToTensor (),
transforms.Normalize ([0.5 , 0.5, 0.5] , [0.5 , 0.5, 0.5]) ])
}
return transforms_dict
Additionally, the StyleGAN and IR-SE50 pre-trained models have to be downloaded and
placed in the pretrained_models folder. Once all of these steps are performed, the training
phase can start using the following command:
python scripts/ train_restyle_psp .py
--dataset_type = sketch_to_face
--encoder_type = BackboneEncoder
--exp_dir=experiment/ restyle_psp_s_encode
--workers =8
--batch_size =8
--test_batch_size =8
--test_workers =8
--val_interval =5000
--save_interval =10000
--start_from_latent_avg
--lpips_lambda =0.8
41](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-49-320.jpg)
![5.2 Testing’s setup
For the purpose of generating synthetic images with the trained model, a graphical user
interface has been developed, along with a client-server architecture. All these three are
done in Python using the RPyC library. This setup enables the model to be tested in a
convenient and efficient manner, as the graphical interface provides an easy-to-use interface
for inputting data and analysing the results, while the client permits to test the model from
the command line inputting existing sketches.
Graphical user interface
The graphical user interface was done using Tkinter library, which allows users to draw on
a canvas using a black pen. Additionally, users can use a rubber to delete portions of the
sketch, or directly remove the previous inserted lines using the “undo” button, or also clear
the canvas, with the “delete” button, and start again from scratch. The rubber and the pen
can vary in thickness between 1 (thinner) and 10 (thicker). Furthermore, three dashed lines
are present, one is vertical in the centre of the interface and the other two are horizontal and
placed at eye and chin level. Those lines guide the users in positioning main facial features
like the eyes, nose, and mouth. The Fig. 20 shows a sketch drawn on the interface.
Once the “send” button is pressed, the drawing is sent to the server. To start the interface,
the following command has to be executed.
python interface.py [-h] [-p PORT] [-a HOST]
Options:
-h this help
-p PORT remote server port , default: 18862
-a HOST remote server address , default: 140.105.164.237
Client architecture
An image can also be generated from the command line typing:
python client_next .py [-h] -i IMAGE [-o OUT] [-p PORT] [-a HOST]
Options:
-h this help
-i IMAGE input image
-o OUT output file , default: output_results .jpg
-p PORT remote server port , default: 18862
-a HOST remote server address , default: 140.105.164.237
44](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-52-320.jpg)
![Figure 20: Graphical interface
The input sketch has to be specified, while there are other parameters that can be set are
not mandatory.
Server architecture
To be able to generate the image based on the sketches the server has to be up, therefore the
following command has to be executed otherwise it will not be possible to generate images.
python server_next .py [-h] -m MODEL [-t TYPE] [-p PORT]
Options:
-h this help
-m MODEL pretrained model
-t TYPE type of the mode , either psp or e4e , default: psp
-p PORT remote server port , default: 18862
Once an image is sent to the server, the restyle encoder is applied to generate the output
image, which is then sent back as a byte stream and then saved as a PNG image. The server
allows also multiple clients since it used the ThreadedServer class of RPyC.
45](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-53-320.jpg)
![References
[1] Cuhk. http://mmlab.ie.cuhk.edu.hk/archive/facesketch.html.
[2] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: A residual-based style-
gan encoder via iterative refinement. In Proceedings of the IEEE/CVF International
Conference on Computer Vision (ICCV), October 2021.
[3] Jason Antic. Deoldify. https://github.com/jantic/DeOldify.
[4] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017.
[5] Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. Arcface: Additive
angular margin loss for deep face recognition. arXiv:1801.07698, 2018.
[6] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using
convolutional neural networks. In 2016 IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pages 2414–2423, 2016.
[7] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In
Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors,
Advances in Neural Information Processing Systems, volume 27. Curran Associates,
Inc., 2014.
[8] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron
Courville. Improved training of wasserstein gans, 2017.
[9] Sven C. Olsen Holger Winnemöller, Jan Eric Kyprianidis. Xdog: An extended
difference-of-gaussians compendium including advanced image stylization. Computers
& Graphics, 36, 2012.
[10] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive
instance normalization. In ICCV, 2017.
[11] Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative
adversarial networks. arXiv preprint arXiv:1903.06048, 2019.
60](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-68-320.jpg)
![[12] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo
Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
[13] Yu-Sheng Lin, Zhe-Yu Liu, Yu-An Chen, Yu-Siang Wang, Ya-Liang Chang, and Win-
ston H. Hsu. Xcos: An explainable cosine metric for face verification task. ACM Trans.
Multimedia Comput. Commun. Appl., 17(3s), nov 2021.
[14] Vijish Madhavan. Artline. https://github.com/vijishmadhavan/ArtLine.
[15] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and
Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential
equations. In International Conference on Learning Representations, 2022.
[16] Lanham Micheal. Generating a new reality. In Generating a New Reality, 2021.
[17] NVIDIA. Ffhq dataset. https://github.com/NVlabs/ffhq-dataset.
[18] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro,
and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image transla-
tion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
June 2021.
[19] Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. Mastering Sketching: Adver-
sarial Augmentation for Structured Prediction. ACM Transactions on Graphics (TOG),
37(1), 2018.
[20] Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. Learning to
Simplify: Fully Convolutional Networks for Rough Sketch Cleanup. ACM Transactions
on Graphics (SIGGRAPH), 35(4), 2016.
[21] Karras Tero, Aila Timo, Laine Samuli, and Lehtinen Jaakko. Progressive growing of
gans for improved quality, stability, and variation. In ICLR 2018, 2018.
[22] Timo Aila Tero Karras, Samuli Laine. A style-based generator architecture for genera-
tive adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), 2019.
[23] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing
an encoder for stylegan image manipulation. arXiv preprint arXiv:2102.02766, 2021.
61](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-69-320.jpg)
![[24] H. J. Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng
Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In
IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[25] X. Wang and X. Tang. Face photo-sketch synthesis and recognition. In IEEE Transac-
tions on Pattern Analysis and Machine Intelligence (PAMI), pages 1955–1967, 2009.
[26] X. Wang and X. Tang. Coupled information-theoretic encoding for face photo-sketch
recognition.). In Proceedings of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR, 2011.
[27] H. Winnemöller, S.C. Olsen, and B. Gooch. Real-time video abstraction. ACM Trans
Graph, 25:1221–1226, 2006.
[28] Ran Yi, Yong-Jin Liu, Yu-Kun Lai, and Paul L Rosin. APDrawingGAN: Generating
artistic portrait drawings from face photos with hierarchical gans. In IEEE Conference
on Computer Vision and Pattern Recognition (CVPR ’19), pages 10743–10752, 2019.
[29] R.A. Young. The gaussian derivative model for spatial vision: I. retinal mechanisms.
Spatial Vision, 2, 1987.
[30] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention
generative adversarial networks. In Proceedings of the 36 th International Conference
on Machine Learning, 2019.
62](https://image.slidesharecdn.com/gallone-in2000143-tesi-230228132406-5291bf6e/85/Evaluation-of-conditional-images-synthesis-generating-a-photorealistic-image-from-a-face-sketch-70-320.jpg)
Evaluation of conditional images synthesis: generating a photorealistic image from a face sketch
- 1. UNIVERSITÀ DEGLI STUDI DI TRIESTE Dipartimento di Ingegneria e Architettura Corso di Laurea Magistrale in Ingegneria Elettronica e Informatica Evaluation of conditional images synthesis: generating a photorealistic image from a face sketch LAUREANDA: Samantha Gallone RELATORE: Chiar.mo Prof. Andrea De Lorenzo ANNO ACCADEMICO 2021–2022
- 2. Abstract Over the last decade, Generative Adversarial Networks (GANs) have made significant progress in image generation. Their capacity to generate high-quality, complex images by learning the distribution of the input data and generating new samples that are indistinguishable from real ones opened up new possibilities for research advancements and innovations. Among all the different opportunities created by GANs, realistic images of human faces, modification of image styles, voice and text generation stand out. The research focuses on exploring the concept of conditional Generative Adversarial Net- works, which are neural networks capable of being conditioned during the image generation process. This conditioning allows to have the control over the content of the generated image, and it is used to produce a realistic face image from an input sketch. The thesis is guided by the research question: “Can people accurately distinguish between a conditionally generated photorealistic image and real images of similar appearance?”. To address the question, a GAN model has been trained on a vast dataset of face sketches and the corresponding images. Afterwards, the performance of the model has been qualitatively evaluated through human assessment. The models and techniques used in this thesis already exist in the literature, however this study differs in the way they have been trained. For the purpose of the thesis, the gener- ative model has been trained on a novel dataset specifically created using computer vision techniques. Finally, the results and findings are presented, followed by the encountered limitations and the challenges faced by GANs in generating realistic images.
- 3. Sommario Nell’ultimo decennio, le reti generative avversarie (GANs) hanno fatto progressi significativi nella generazione di immagini. La loro capacità di generare immagini complesse e di alta qualità apprendendo la distribuzione dei dati di input e generando nuovi campioni indistin- guibili da quelli reali offre nuove possibilità di progresso e innovazione nella ricerca. Tra tutte le diverse opportunità create dalle GAN, spiccano la generazione di immagini realistiche di volti umani, la modifica dello stile di un’immagine, la generazione di voce e testo. La ricerca si concentra sull’esplorazione del concetto di reti generative avversarie condizio- nali, ossia reti neurali in grado di essere condizionate durante il processo di generazione di un’immagine. Questo condizionamento permette di avere il controllo sul contenuto del- l’immagine generata, ed è usato per produrre immagini di volti realistici partendo da un disegno. La tesi è guidata dalla domanda di ricerca: “Possono le persone distinguere accuratamente tra un’immagine fotorealistica generata in modo condizionale e immagini reali di aspetto simile?”. Per rispondere a questa domanda è stato addestrato un modello GAN su un vasto dataset di disegni di volti e le immagini corrispondenti. Successivamente le prestazioni del modello sono state valutate qualitativamente attraverso la valutazione effettuata da diverse persone. I modelli e le tecniche utilizzati in questa tesi esistono già in letteratura, tuttavia lo studio si differenzia nel modo in cui sono stati addestrati. Ai fini della tesi, il modello generativo è stato allenato su un nuovo set di dati appositamente creato utilizzando tecniche di computer vision. Infine, vengono presentati i risultati e le scoperte, seguiti dalle limitazioni incontrate e dalle sfide affrontate dalle GAN nella generazione di immagini realistiche.
- 4. Contents 1 Introduction 1 1.1 Objective of the thesis and research question . . . . . . . . . . . . . . . . . . 1 1.2 Possible applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Recently emerged technology . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.4 Structure of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Literature Review 5 2.1 GAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.1 GAN structure and training . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Wasserstein-GAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 ProGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4 Evaluation metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4.1 xCos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3 Dataset preparation 15 3.1 Dataset selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2 ArtLine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.3 Sketch simplification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.3.1 Learning to simplify . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.3.2 Mastering Sketching . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4 Extended Difference of Gaussian . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.5 Dataset building . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4 Implementation 28 4.1 StyleGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.2 StyleGAN2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.3 pSp Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.4 Restyle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5 Setup for training and testing 40 5.1 Training’s setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.2 Testing’s setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 I
- 5. 6 Evaluation and results 46 6.1 Survey’s results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 7 Conclusions 58
- 6. List of Figures 1 Example of a GAN architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Architecture of a GAN. The generator and the discriminator are synchronously trained. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3 Example of the xCos architecture taken from [13] . . . . . . . . . . . . . . . . 14 4 Learning to Simplify architecture. Image taken from [20] . . . . . . . . . . . . 18 5 Figure from [20] showing the results of Learning to Simplify approach. . . . . 19 6 Figure from [19] that highlights the comparison between the Mastering sketch- ing’s results and the ones of the Learning to Simplify. . . . . . . . . . . . . . 21 7 Results obtained from the three trained models ([19]). . . . . . . . . . . . . . 21 8 Figure taken from [9] in which there is a comparison between different known edge detection techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 9 Example of outputs obtained by using ArtLine . . . . . . . . . . . . . . . . . 24 10 Output image obtained applying Artline and one of the four models of Learn- ing to Simplify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 11 Output image obtained applying the XDoG operator . . . . . . . . . . . . . . 26 12 Output image obtained applying Mastering sketching to the output image of XDoG operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 13 (a) traditional GAN architecture, (b) Style GAN architecture. Image taken from [22]. The mapping network consists of 8 layers, while the synthesis network g consists of 18 layers (two for each resolution starting from 42 ). The “A” block stands for the affine transform, and the “B” is the block that applies per-channel scaling factor to the input Gaussian noise. . . . . . . . . . 30 14 StyleGAN2 architecture. Image taken from [12]. Fig (a) and (b) show the StyleGAN architecture and its detailed view. In fig (c), the StyleGAN archi- tecture in which the AdaIN layers have been replaced with the modulation is illustrated. Fig. (d) shows the revised architecture in which the instance normalisation has been replaced with a “demodulation” operation. . . . . . . 33 15 Pixel2Style2Pixel architecture. Image taken from [18] . . . . . . . . . . . . . 35 16 ReStyle’s iterative inversion process. Image taken from [2] . . . . . . . . . . . 38 17 Simplified ReStyle encoder architecture. Image taken from [2] . . . . . . . . . 39 18 Average image used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 III
- 7. 19 Training results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 20 Graphical interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 21 Disappointing result generated from a sketch too small . . . . . . . . . . . . . 46 22 Result obtained from a sketch made with a thick pen . . . . . . . . . . . . . . 47 23 Impressive result of an image generated from a sketch . . . . . . . . . . . . . 47 24 Result of an image generated from a simple sketch . . . . . . . . . . . . . . . 48 25 Examples of two similar sketches . . . . . . . . . . . . . . . . . . . . . . . . . 48 26 Examples of discarder images . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 27 Examples of the structure of the survey . . . . . . . . . . . . . . . . . . . . . 49 28 Results for the first 10 sketches (a). . . . . . . . . . . . . . . . . . . . . . . . . 50 28 Results for the second 10 sketches (b). . . . . . . . . . . . . . . . . . . . . . . 51 29 Sketch with the greatest percentage of incorrect responses . . . . . . . . . . . 52 30 Sketch with the greatest percentage of correct responses . . . . . . . . . . . . 53 31 Sketch with the lowest average time . . . . . . . . . . . . . . . . . . . . . . . 53 32 Sketch with the highest average time . . . . . . . . . . . . . . . . . . . . . . . 54 33 Images associated with sketch 0 and 2 . . . . . . . . . . . . . . . . . . . . . . 55 34 Scatter plot representing the relation between the average time and the per- centage of correct responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 35 Sketch of the most dreamt man and its corresponding generated image . . . . 57 IV
- 8. List of Tables 1 Summary of the results obtained . . . . . . . . . . . . . . . . . . . . . . . . . 52 2 Results of the binomial test . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 V
- 9. 1 Introduction Deep learning, once considered a marginal field, has become a key player in the rapidly evolv- ing landscape of Artificial Intelligence (AI). In the past decade, the growth of deep learning has led to the development of innovative architectures and methods that enable machines to learn in ways previously thought impossible. From traditional supervised learning to more advanced forms such as unsupervised and self-supervised learning, to the innovative techniques of adversarial and reinforcement learning, AI has come a long way. Among these forms of learning, generative modelling has emerged as a major area of focus and has seen substantial growth. Thanks to deep learning, there is now a better understanding of what it means for an AI to learn and how to implement models that can self-learn. In particular, the advent of adversarial learning has played a major role in fuelling the AI explosion, since it has a large set of applications in generative modelling. Generative modelling refers to implementing AI models that can generate new and varied content through a variety of learning methods. With the ability to generate new content, AI is changing the way the world is experienced and understood. In the context of generative modelling, the term “generate” is crucial. When constructing a generative model, the focus is on creating new content, not simply modifying or filtering what already exists. This content can be generated randomly or, in certain cases, it can be generated based on a text or an image. The ability to conditionally control content generation allows for a greater degree of flexibility and precision in generating content that meets specific criteria [16]. The ultimate goal of generative modelling is to create new, unique content that was not present in the original dataset. Generative modelling has numerous applications, including image synthesis, data augmentation, and music generation, among others. Advances in generative modelling are constantly being made, and new techniques are being developed, making it a rapidly growing field of research with numerous opportunities for new discoveries and breakthroughs. 1.1 Objective of the thesis and research question Generative Adversarial Networks have gained significant attention in recent years due to their ability to generate realistic images. However, it is challenging to evaluate the quality 1
- 10. of image generation in these networks. This thesis aims to explore the extent to which generative adversarial networks can produce conditioned images. The investigation is guided by the desired to comprehend the abilities of these networks. The thesis primary goal is to determine the level at which people can differentiate between images generated from sketches and real images that resemble the sketches. The research is not centred on developing a new neural network, but rather on evaluating the images produced by existing networks. The pre-trained StyleGAN network was utilised in this research, and the weights were employed to train a scheme called ReStyle over a variation of the pixel2style2pixel encoder on a custom dataset. The research question that this thesis wants to answer is the following: Can people accurately distinguish between a conditionally generated photorealistic image and real images of similar appearance? To assess the quality of the generated images, a survey was designed where participants were shown a sketch of a face and asked to identify which of the four additional images inspired the sketch. The results of the survey were analysed to determine the human ability to recognise generated images from authentic ones. The images were generated based on sketches created by users on a tablet device, and they are very similar to real people’s faces, thus making it challenging for humans to differentiate between authentic and fake images. 1.2 Possible applications There are numerous potential applications for the generation of conditional images based on sketches, as the full extent of this technology’s capabilities has yet to be explored. The following illustrates examples of the possible uses. For example, the generation of conditional images based on sketches can be used in the entertainment industry, where the technology could be used to create digital characters for video games, movies, and animations. But also the creation of custom avatars for social media and virtual reality. While one of the most impactful implementation of this research would be a sketch-based image retrieval system in criminal investigations. The technology could be useful for law enforcement, allowing forensic artists to create photorealistic renderings of suspects based on eyewitness sketches. Unfortunately, not all the possible applications are “safe”, since there are some that could 2
- 11. be potentially dangerous in case of misused or if applied without considering ethical and moral implications. Indeed, this technology could be used for the creation of fake identities. 1.3 Recently emerged technology While conducting this research, similar techniques have emerged, including Stable Diffusion. Stable Diffusion is a text-to-image model that allows users to easily produce high-quality artwork in a few seconds based on a text prompt. Traditionally, the Stable Diffusion model is trained to generate images based on text inputs. However, thanks to a diffusion-denoising mechanism proposed by [15] the model can also be used for task such as text-guided image- to-image translation and upscaling. The approach offer a similar solution to the problem that this study was investigating, but ultimately, the research was carried on. The results obtained from Stable Diffusion are remarkable due to its extensive training on a massive dataset and on highly powerful GPUs. The long training time allowed the network to learn intricate patterns and relationships in the data, resulting in high-quality outputs. The use of powerful GPUs also ensured a faster training process, allowing the network to converge quickly to a satisfactory solution. 1.4 Structure of the thesis This section describes the thesis’s structure. Chapter 1 provides a comprehensive overview of artificial intelligence and generative models. It also includes the objective and guiding research question of the thesis, as well as several potential applications of the project. Chapter 2 of the thesis is dedicated to a comprehensive review of the existing literature on generative adversarial networks. In this chapter, the reader will gain an understanding of the background and evolution of these networks. In chapter 3 a detailed discussion on the creation of the dataset is presented, including the selection criteria, data pre-processing and preparation processes. This chapter will dig deeper into the methods used to build the dataset, the reasons for selecting the specific data and how it was processed. Additionally, it explores the theories behind the technologies used to process the images. Chapter 4 provides a comprehensive overview of the specific generative adversarial networks that are used in the project, including their architecture and training methodologies. Chapter 5 is dedicated to a discussion about the training and testing setup. 3
- 12. Chapter 6 is specifically devoted to the evaluation of the results obtained through the experiments conducted in this study. This chapter presents a comprehensive analysis of the performance of the proposed Generative Adversarial Network (GAN) model in generating realistic face images from input sketches. The evaluation is carried out from a qualitative evaluation through human assessment. The main goal of this chapter is to provide an in- depth evaluation of the performance of the proposed model and to draw conclusions based on the obtained results. In the final chapter, chapter 7, there will be a brief discussion on the achieved results, the limitations of the model, and potential areas of future work. 4
- 13. 2 Literature Review This chapter provides an in-depth overview of the field of generative modelling. It lays out the theoretical background of the thesis, beginning with a general explanation of how GANs work and then exploring some useful theoretical concepts. 2.1 GAN The innovative concept of Generative Adversarial Networks (GANs) was first introduced in 2014 by Goodfellow and his colleagues from the University of Montreal [7]. GANs are a type of deep learning model that has the ability to generate new content, such as text or images, based on a given dataset. GANs have been the focus of significant research in recent years and have made significant contributions to the field of computer vision. The technology has advanced in areas like generating realistic images, transforming images, modifying facial features, and similar topics, but also in other areas such as music generation. Before the advent of GANs, machine learning algorithms were highly effective in recognising patterns in existing data and could be utilised for tasks like classification and prediction. However, generating new data presented a challenge for these algorithms. The solution proposed in the GAN paper leverages the strengths of machine learning algorithms – their ability to classify and predict – to overcome their weaknesses in generating new content. The approach involves setting two algorithms against each other in a kind of competition, leading to a continuous process of improvement. This approach has proven to be highly effective in generating new, diverse content and has been widely adopted in the field of deep learning. Before diving into the intricacies of GANs, it is essential to understand how these networks work. A widely used analogy can provide a clearer insight. Imagine there is an art forger and an art authenticator. The forger’s goal is to produce a counterfeit piece of art that the authenticator will consider genuine. The authenticator is trained to recognise genuine art by examining real pieces, while the forger learns by creating art that is then evaluated by the authenticator. In GANs, the forger is the generator, which strives to produce examples that are indistinguishable from the real data in the training set, thus trying to fool the discriminator. On the other hand, the discriminator acts as the art authenticator, evaluating the authenticity of the generated pieces. These two networks 5
- 14. are in a constant battle, each trying to outsmart the other. As the generator improves in creating realistic data, the discriminator needs to enhance its capability in detecting fake examples. Similarly, as the discriminator becomes better in differentiating true from false, the generator must strive to generate even more convincing fake data. The most critical aspect of this architecture is that the generator has no direct connection to the actual image, its only way to learn is through its interaction with the discriminator [16]. Fig. 1 illustrates a simple GAN architecture based on the analogy. Figure 1: Example of a GAN architecture 2.1.1 GAN structure and training A GAN has a similar structure to that of an autoencoder, where the discriminator serves as the encoder. However, instead of producing an encoded representation, it only outputs whether the input is real or fake. On the other hand, the generator acts like the decoder and takes in random data from the latent space, learning to generate new content that can trick the discriminator into thinking it is real. Both the generator and discriminator are trained in a competitive manner, with the generator striving to maximise the error of the discriminator and the discriminator trying to minimise it. 6
- 15. The generator and the discriminator are two distinct deep neural networks. The generator is a function G that takes a random noise vector z as input and maps it to the data space, producing a set of synthetic samples. These samples are fed into the discriminator network D, together with the real samples x. This network outputs a single scalar that represents the probability that the sample came from the real data distribution rather than the synthetic data distribution. Therefore, the discriminator learns to use traditional supervised learning techniques, dividing inputs into two classes (real or fake). The GAN training process is a two-player that involves updating the weights of the generator and discriminator networks using gradients computed from the loss function. The loss function, also called cost function, in a GAN is designed to quantify the difference between the generated data and the real data. The objective is to find the balance between the generator and discriminator losses, where the generator loss represents the ability of the generator to produce realistic synthetic data and the discriminator loss represents the ability of the discriminator to distinguish between real and fake data. There are several cost functions that may be used, and they usually differ in terms of the cost function used for the generator. The basic loss function used in GAN is the binary cross-entropy: L(ŷ, y) = [y × log(ŷ) + (1 − y) × log(1 − ŷ)] (1) where y represents the original data and ŷ the generated one. In training, the discriminator assumes y = 1 for real data, while for the generated data it is set: ŷ = D(x). Substituting this information in equation 1 the discriminator loss can be obtained: L(D(x), 1) = log(D(x)) (2) For the generator’s output data, it is assumed that y = 0, because it outputs fake data, and ŷ = G(D(z)) for the generated data, where z is a random noise vector sampled from a noise distribution pz(z) (e.g., uniform or Gaussian distribution). Therefore, the loss for the generator is: L(D(G(z)), 0) = log(1 − D(G(z))) (3) To obtain the total discriminator loss the two equations above have to be combined and 7
- 16. maximised: L(D) = max[log(D(x)) + log(1 − D(G(z)))] (4) The generator has to fool the discriminator, hence it has the opposite goal, therefore to obtain its loss function its minimum has to be calculated: L(G) = min[log(D(x)) + log(1 − D(G(z)))] (5) The loss function is designed to encourage the generator to produce realistic synthetic data and the discriminator to distinguish between real and fake data with high accuracy. The interdependence of each player’s cost on the other player’s parameters, along with the lack of control over the other player’s parameters, makes this scenario more accurately described as a game rather than an optimisation problem. The loss function in a GAN is an important component of the training process, as it determines the quality of the generated data and the ability of the discriminator to accurately distinguish between real and fake data. The training process of GANs is a zero-sum game in which the objective function of this minimax optimisation problem is: min max V (D, G) = min G max D Ex∼pdata(x)[log D(x)] + Ez∼pz(z)[log(1 − D(G(z)))] (6) where x is a real image from the true data distribution pdata, and z is a noise vector sampled from the noise distribution pz. Ex and Ez denote the expected log likelihood from the different outputs of both real and generated images. In training a GAN, it is important to ensure that both the generator and discriminator models are developed in a balanced manner. The success of the GAN largely depends on the alignment of the real and generated distributions, and this can only be achieved if neither of the models is overly dominant. In addition to train the generator and the discriminator, the errors on their outputs are backpropagated into the models as gradients of the loss functions, as shown in Fig. 2. These loss functions are used to guide the learning process and ensure that the generator produces high-quality samples. If the discriminator is too good at identifying fake versus real images early in the training process, the generator may struggle to produce convincing fake images, making it difficult for the two models to improve. If the 8
- 17. Figure 2: Architecture of a GAN. The generator and the discriminator are synchronously trained. generator is successful in fooling the discriminator, the loss will backpropagate through the discriminator network to improve its performance. On the other hand, if the discriminator effectively distinguishes between fake and real samples, the loss will backpropagate through the generator network to improve its ability to generate samples that are similar to real samples. The update rules for the discriminator and the generator are shown in equations 7 and 8 respectively. The weights of each model are represented by θd and θg, respectively, and m represents the total number of samples in a batch used to test the models before updating them. Algorithm 0 optimises Eq. 6 as show by [7]. From a mathematical point of view, if the discriminator gets too good at identifying the samples, this may increase the divergence between the generated expected distribution. In case the generated and the real distributions become too diverse or do not overlap, then the generator will suffer from vanishing gradient loss. The vanishing gradient problem is a critical issue in training deep neural networks. The problem occurs when the gradients of the parameters with respect to the loss become ex- tremely small, making it difficult for the optimizers to update the parameters of the network effectively since they do not have any effect on the training model. The vanishing gradient problem can lead to slow convergence or complete failure of the training process. One pos- sible solution to this problem is to use the Wasserstein GAN architecture. 9
- 18. Algorithm 1 Minibatch stochastic gradient descent training of generative adversarial nets. The number of steps to apply to the discriminator, k, is a hyperparameter. for number of training iteration do for k steps do • Sample minibatch of m noise samples {z(1) , ..., z(m) } from noise prior pg(z) • Sample minibatch of m examples {x(1) , ..., x(m) } from data generating distribu- tion ppdata(x). • Update the discriminator by ascending its stochastic gradient: ∇θd 1 m m X i=1 [log D(x(i) ) + log(1 − D(G(z(i) ))] (7) end for • Sample minibatch of m noise samples {z(1) , ..., z(m) } from noise prior pg(z). • Update the generator by descending its stochastic gradient: ∇θg 1 m m X i=1 [log(1 − D(G(z(i) ))] (8) end for The gradient-based updates can use any standard gradient-based learning rule. The goal is to make the expected and generated distribution match as closely as possible. The GAN continues this process back and forth until an equilibrium is reached between the generator and discriminator loss. Over time, the two networks learn from each other and converge to a state where the generator produces samples that are almost indistinguishable from real data, and the discriminator is able to correctly identify which samples are real and which are fake with high accuracy. This basic design is referred to as the Vanilla GAN, as it represents the original model from which more advanced GAN architectures have evolved, yet still retaining the two key components of the GAN structure. 2.2 Wasserstein-GAN The Wasserstein-GAN, also known as WGAN, is a variant of the Vanilla GAN and it was introduced by Arjovsky et al. in 2017 [4]. WGAN aims to improve the generator model’s capability of approximating the data distribution of a given training dataset by introducing a different training mechanism. Instead of using a discriminator to classify the generated 10
- 19. images as real or fake, WGAN uses a critic that scores the authenticity of an image. It uses the Wasserstein distance (also known as Earth-Mover’s distance), which is a metric that quantifies the minimum cost of moving mass from one distribution to another to convert the source distribution q to the target distribution p. It is used to measure the distance between the distribution of real data and the distribution of generated data. In WGAN, the discriminator is trained to be a function that outputs a scalar value instead of a probability, and the generator is trained to minimise the Wasserstein distance. One formulation for the Wasserstein distance is: W(Pr, Pg) = sup f∈F Ex∼Pr [f(x)] − Ex∼Pg [f(x)] (9) where F is defined as the set of 1-Lipschitz functions, which are those satisfying ∥f∥L ≤ 1 where ∥ · ∥L represents the Lipschitz norm. The WGAN objective function can be formulated as: min G max D∈F Ex∼Pr [D(x)] − Ex̃∼Pg [D(x̃)] (10) where Pr and Pg are respectively the real and generated distribution, x̃ = G(z) with z ∼ p(z). The above formulation needs the critic to be 1-Lipschitz. This condition is ensured by clipping the weights W of the discriminator to be within a compact range of [−c, c], where c is a user-defined hyper-parameter. Specifically, each weight wi,j is clipped to this range. WGAN-GP One step towards stable training of GANs is made by WGAN, however, it may still generate inadequate samples or not converge. The cause of such issues is frequently attributed to the utilisation of weight clipping in WGAN, which enforces a Lipschitz constraint on the critic and can result in undesirable behaviour, for example, the convergence of a very deep WGAN critic may often fail. Therefore, to address this issue an alternative to weight clipping has been proposed by [8] to ensure smooth training, which is the Wasserstein GAN with Gradient Penalty (WGAN-GP). The WGAN-GP method replaces weight clipping with a gradient penalty, introducing an extra loss term that encourages the discriminator to have 11
- 20. a gradient with a norm that is near 1. The resultant optimization problem is given by: L = Ex̃∼Pg [D(x̃)] − Ex∼Pr [D(x)] + λEx̂∼Px̂ [(∥∇x̂D(x̂)∥2 − 1)2 ] (11) where the first two terms are the original critic loss of the WGAN, while the last term is the gradient penalty, in which λ defines the gradient penalty coefficient, x̂ is a random sample along the straight line between real and generated samples and px̂ is the distribution of x̂. 2.3 ProGAN ProGAN, short for Progressive Generative Adversarial Network, is a deep learning model that was developed to generate high-resolution, photorealistic images. It was introduced by Karras et al. in 2017 [21], it is a type of generative model, which means that it can generate new, synthetic images based on the data it is trained on. The main idea behind ProGAN is to use a series of generators and discriminators to generate high-resolution images incrementally, starting from low-resolution images (e.g. 4×4) and gradually increasing the resolution until a high-resolution image is obtained (e.g. 1024×1024). This approach is called progressive growing, and it allows the network to learn the finer details of an image in an incremental manner. The generator starts with a random noise input and generates an image with low resolution, which is then evaluated by the discriminator. The discriminator determines whether the generated image is real or not and provides feedback to the generator, allowing it to improve. One of the benefits of ProGAN is that it generates high-resolution images that are photore- alistic, which makes it useful for various applications. Additionally, ProGAN is trained on large datasets, which means that it can create diverse and varied images. Another advantage of ProGAN is its ability to produce images at different resolutions, which allows for a level of control over the generated images. Some drawbacks of the progressive growth method in a GAN are the prolonged training time involved in constructing subsequent models at higher resolutions, the need for extra resources in terms of data preparation and storage, and of course the non-negligible computational time. 12
- 21. 2.4 Evaluation metric 2.4.1 xCos The state-of-the-art face verification models employ a methodology that involves extracting deep features from a pair of face images and subsequently computing the cosine similarity or the L2-distance of the paired features. The determination of whether the two images are of the same person is based on whether the similarity value exceeds a pre-defined threshold value. However, the approach proposed in the paper “xCos: An Explainable Cosine Metric for Face Verification Task” [13] offers a distinctive approach in which the verification result is obtained by the integration of the local similarity map and the attention map. It is based on the insight that humans tend to compare different facial features to determine whether two face images belong to the same individual. xCos leverages this understanding to evaluate the similarity of facial features between two images. It is built using a grid-based feature extraction approach, in which each face image is divided into multiple local regions. The facial features of each local region are then extracted and compared with the features of the corresponding regions in the other image. The resulting similarities are then aggregated across all the local regions to compute a final similarity score. It also includes an atten- tion mechanism that identifies the specific facial features that contribute the most to the similarity score. This attention mechanism allows for a more interpretable and explainable verification result, as it highlights the specific facial features that the model is prioritising. The proposed architecture, showed in Fig. 3, for face recognition consists of a modified CNN backbone and two branches for xCos and identification. The primary responsibility of the CNN backbone is to extract facial features for each identity. To preserve the position information of each feature point, the final flatten and fully-connected layers of the backbone, such as ArcFace [5] or CosFace [24], are replaced with a 1 by 1 convolution. On the xCos branch, to compute a patched cosine map S (cospatch), the two feature maps of compared images are element-wise evaluated using cosine similarity. At the same time, an attention weight map W is generated using the attention mechanism based on the two feature maps. The final xCos similarity value is achieved by performing a weighted sum of the patched cosine map S, using the attention weight map W. The xCos branch is trained using cosine similarity computed by another face recognition 13
- 22. Figure 3: Example of the xCos architecture taken from [13] model, such as ArcFace. The identification branch flattens the extracted feature and feeds it into another fully connected layer for predicting the identity. The training is stabilised by using a common face recognition loss, such as the one used in ArcFace, which is denoted as LID. 14
- 23. 3 Dataset preparation This chapter provides an in-depth discussion on the process of the dataset creation, as well as the dataset utilised in the experimentation. Furthermore, the theoretical foundations of the tools employed in the dataset creation are comprehensively presented. 3.1 Dataset selection Since the objective of this thesis is to generate a realistic photo of a face starting from a handmade sketch, a neural network had to be trained on a large dataset of face images and their corresponding sketches. However, finding such a dataset was not easy. Online the CUHK dataset (The Chinese University of Hong Kong dataset) [1] is available, it is a collection of images used for various computer vision and machine learning tasks, such as image classification, object detection, and face recognition. The specific content and size of the CUHK dataset may vary depending on the source. The dataset includes the CUHK Face Sketch FERET Database (CUFSF) [25, 26] which is composed of 1194 pair of images with both a photo of a face and a sketch of it. Each face includes an artist-drawn sketch based on a photo that was taken with the individual in a neutral expression, under normal lighting conditions, and in a frontal pose. This database is made for the purpose of researching face sketch synthesis and face sketch recognition. Unluckily, it does not contain enough images, and it requires data augmentation techniques in order to use it. Another dataset available online is the FFHQ (Flickr-Faces-HQ) dataset [17], which is a large-scale, high-quality dataset of face images, created to advance research in generative models and computer vision. The dataset contains 70 000 high-resolution (1024 × 1024) im- ages of diverse human faces, each with different poses, lighting, expressions, and ethnicities. Since it is a very large dataset, it can be used to create the needed dataset composed of pairs of sketches and photos, but it requires some image processing techniques in order to obtain the sketches. The challenge here is to find a technique to transform an image into a sketch. The idea is to extract the face contours using an edge detection technique, while preserving the essential details of the face and making the result look like a hand-drawn sketch. To achieve these results, two approaches were considered to build the dataset. The first approach involves using two separate networks: Artline [14] and a network for the sketch 15
- 24. simplification task. In the second approach, it is also used a network for simplifying the sketches, but instead of using a neural network to obtain the sketches, the extended difference of Gaussian (XDoG) edge detector [9] is applied. In order to produce the desired results, both approaches need to be carefully analysed and compared. In this way, the most appropriate approach can be chosen and further developed to create a realistic handmade sketch. Now the theoretical foundation underlying the techniques that are employed for constructing the dataset are presented before discussing how they have been used to reach the final goal. 3.2 ArtLine ArtLine is a deep neural network designed specifically to turn photographs into line art drawings. It is a recent project developed by Vijish Madhavan [14] interested in generative modelling, that follows the discoveries found by Jason Antic in his DeOldify project [3], utilising advancements and tools developed by the team at Fast.ai. ArtLine, as DeOldify, is based on a self-attention GAN (SAGAN) that is progressively resized to achieve a desired resolution. SAGAN was introduced in a 2018 research paper by Zhang et al. [30] and from then on it has been applied to a wide range of image generation tasks, including facial image synthesis, artistic image generation, and more. The results have been very promising, with SAGAN producing high-quality and highly detailed images that are comparable to those produced by state-of-the-art methods. The self-attention mechanism allows the generator network to weigh the contribution of different parts of an image in generating the final result. This allows the network to focus on important features of the image and produce more accurate results. Additionally, the use of self-attention also allows the network to better capture the relationships between different parts of the image, resulting in more coherent and visually appealing images. ArtLine has been trained on the APDrawing dataset, which is a dataset obtained from another generative adversarial network able to generate high-quality sketch drawings from a given image, called APDrawingGAN [28]. The APDrawingGAN is an innovative GAN- based architecture that has the capability to generate professional-level artistic portraits. It uses a combination of a global network and local networks, which allows for dedicated drawing strategies to be learned for specific facial features. This leads to a more precise and accurate portrayal of facial features in the generated drawings. To train APDrawingGAN, a unique artistic drawing dataset was constructed, containing high-resolution portrait photos 16
- 25. and corresponding professional artistic drawings. The results of extensive experiments and a user study demonstrate that APDrawingGAN produces significantly better artistic drawings than existing methods. Since the results obtained using this dataset were good only in the case of ID photos, the author of ArtLine combined this dataset with an Anime sketch dataset to obtain good results also with other kinds of pictures. The training process of ArtLine is similar to that of a NoGAN, which is a type of genera- tive model that trains without using the discriminator network. Therefore, the initial loss function for the generator is based on the features extracted from a pre-trained VGG model. 3.3 Sketch simplification 3.3.1 Learning to simplify The model Learning to Simplify (LtS) [20] consists of a Fully Convolutional Network (FCN) to learn the mapping between rough sketches and clean drawings. The technique of simpli- fying rough sketches utilises Convolutional Neural Networks (CNNs) to simplify the image. Through a series of convolution operations, the sketch lines are grouped, and a simplifica- tion is directly produced. The CNNs are trained using a unique dataset of rough sketches and their corresponding simplifications, which the authors have created. This data-driven method has several benefits. Firstly, the model learns automatically from the training data the necessary heuristics for sketch simplification. Secondly, since convolution operations can be executed efficiently on GPUs, the processing time is less than a second for most of the images. In contrast to typical CNN architectures, which utilise fully connected layers, their approach uses solely convolutional layers with sparse connections. These sparse connections allow their CNN to process images of varying resolutions and aspect ratios with ease. The results of the study show that the FCN is able to produce high-quality clean drawings from rough sketches with remarkable accuracy. The model has three main components: an encoder that compresses the image, a processing unit that extracts the critical lines, and a decoder that converts the simplified represen- tation back into a greyscale image of the same resolution as the input (Fig. 4). All these components use convolutional layers for their processing. The model uses a down- and up-convolution architecture that may look like simple filter banks. It has a larger number of channels in lower-resolution parts, such as 1024 channels 17
- 26. Figure 4: Learning to Simplify architecture. Image taken from [20] when the size is 1/8, to ensure that the necessary information for clean lines is carried through. The model uses padding in its convolutional layers to guarantee that the output has the same size as the input when a stride of 1 is used. Instead of pooling layers, the authors of this model used convolutional layers with a higher stride to lower the resolution from the previous layers. To increase the resolution and ensure that the output of the model has the same dimensions as the input, they used fractional strides. Overall, the model leverages the ability of CNNs to learn and extract essential information, making it an effective solution for sketch simplification. The LtS model has been trained by the authors using pairs of rough and simplified sketches representing the input and the target, respectively, and they used the weighted mean square error criterion as loss: l(Y, Y ∗, M) = ∥M ⊙ (Y − Y ∗)∥2 F RO (12) where Y is the model output, Y ∗ is the target output, M is the loss map, ⊙ is the matrix element-wise multiplication or Haddamard product, and ∥ · ∥F RO is the Frobenius norm. They selected a loss map that minimises loss on the thicker lines, to prevent the model from primarily focusing on the thicker lines while neglecting the thinner lines. The loss map is built by examining the histogram surrounding each pixel in the target label (ground truth) and it is defined as: M(u, v) = 1 if I(u, v) = 1 min(α exp(−H(I, u, v)) + β, 1) else (13) 18
- 27. The value of the local normalized histogram bin in which the pixel (u, v) of image I falls into is represented as H(I, u, v). An example of the results that can be obtained with this approach can be seen in Fig. 5. Figure 5: Figure from [20] showing the results of Learning to Simplify approach. 3.3.2 Mastering Sketching Mastering sketching [19] is another approach for training networks on the sketch simplifi- cation task, similar to Learning to Simplify (3.3.1). The authors of the paper introduce adversarial augmentation, a fusion of unsupervised and adversarial training for structured prediction tasks. The proposed method can improve the quality and generalization of exist- ing networks, while also permitting the use of unlabelled examples. The authors explained that the existing methods to achieve the sketch simplification task requires large amounts of labelled data, which is time-consuming and an expensive process. Additionally, the resulting data may not accurately represent the true rough sketches drawn without such references, and the availability of such data is often limited, which can lead to overfitting or poor gen- eralization. The proposed method aims to address these issues by leveraging unsupervised data and adversarial training. Their approach combines a fully convolutional network for sketch simplification, which is the same as the one used in the Learning to Simplify, with a discriminator network that is able to distinguish real line drawings from those generated by the network. The training of the sketch simplification network involves two objectives: simplifying the input sketches and fooling the discriminator network by generating sketches that are indistinguishable from real ones. The use of the discriminator network allows the simplification network to learn 19
- 28. the structure of real sketches and generate simplified versions that are similar in appearance to the training sketches. They trained a variation of a conditional GAN where, instead of using a random input z, they used a deterministic prediction S : x 7→ y = S(x). For the supervised adversarial training they trained together the prediction model S and the discriminator model D : y 7→ D(y) ∈ R which is not conditioned on the input x, since they assumed that S is associated with a supervised training loss L(S(x), y∗ ). The training consists in maximising the output of D while at the same time minimising the loss of the model with the structured output, resulting in the following optimisation: min S max D E(x,y∗)∼ρx,y [α log D(y∗ ) + α log(1 − D(S(x))) + L(S(x), y∗ )] (14) where ρx,y is the supervise training data consisting of input-output pairs, α is the weighting hyperparameter, y and y∗ are the output image and the ground truth data corresponding to the input x. From equation 14, it can be noted that the first term in the expectation value depends only on y, while the second depends only on x. This indicates that it can be done a training using these terms separately with unsupervised data. By separating the first two terms in the expectation value of equation 14, then the supervised adversarial objective function can be used to incorporate the unsupervised data into the training. Under the conditions that there are available an extensive collection of input data ρx, output data ρy, and a dataset ρx,y consisting of fully-annotated pairs, then the optimisation function can be adjusted as follows: min S max D E(x,y∗)∼ρx,y [L(S(x), y∗ ) + α log D(y∗ ) + α log(1 − D(S(x)))] +βEy∼ρy [log D(y)] + βEx∼ρx [log(1 − D(S(x))] (15) where β is a weighting hyperparameter for the unsupervised data term. For the simplification network, they did not use the weighted MSE loss as in 3.3.1, but instead they used the MSE loss defined as: L(S(x), y∗ ) = ∥S(x) − y∗ ∥2 (16) 20
- 29. where ∥ · ∥ is the Euclidean norm, S(x) is the deterministic prediction of a given input x, y and y∗ are the output image and the ground truth data corresponding to the input x, respectively. The results using this approach can be seen in Fig. 6. Figure 6: Figure from [19] that highlights the comparison between the Mastering sketching’s results and the ones of the Learning to Simplify. The authors also proposed a pencil drawing generation solution that is the inverse problem of sketch simplification. Indeed, they swapped the input and output data used for the sketch simplification and trained three models: one with the MSE loss and two models with adversarial augmentation for different artists, one based on an artist with a dirty and faded pencil style and the other based on an artist with clearer pencil style. Fig. 7 illustrates the results of the three models. Figure 7: Results obtained from the three trained models ([19]). 21
- 30. 3.4 Extended Difference of Gaussian The Extended Difference of Gaussian (XDoG) is an edge detector operator based on the Difference of Gaussian (DoG). It comes from the idea that edges in an image correspond to rapid changes in intensity, which can be detected by subtracting two images that have been smoothed with different Gaussian kernels. It can be used to increase the visibility of edges and other features in digital images. The capacity of the DoG operator to extract information about the edges in the image can be explained by considering the Gaussian filter from a signal processing point of view. The Gaussian filter is defined as: Gσ(x) = 1 2πσ2 e− x2 2σ2 (17) where x refers to a two-dimensional coordinate, and σ represents the standard deviation of the Gaussian distribution in the spatial domain. It is a low-pass filter which attenuates or eliminates high frequencies. The difference between the two Gaussian with different σ is equivalent to a subtraction between a blurred version of the original image and a less blurred version of it, and it is defined as: Dσ,k(x) = Gσ(x) − Gkσ(x) ≈ −(k − 1)σ2 ∇2 G (18) where the standard deviations of these Gaussian filters determine the scale at which edges are detected, with larger standard deviations detecting larger edges. This difference is a band- pass filter which attenuates all frequencies that are not between the cut-off frequencies of the two Gaussians. This results in a new image that highlights regions of the original image where the intensity changes rapidly. The XDoG approach is inspired by biological models proposed by Young et al. [29], which were based on the DoG model. Winnemöller et al. [27] extended the DoG model to allow for greater control over the inhibitory effect of the larger Gaussian, which is allowed to vary, resulting in a more advanced edge detection technique with the following equation: Dσ,k,τ (x) = Gσ(x) − τ · Gkσ(x) (19) This allows for more fine-grained control over the output, resulting in more precise edge detection and image stylization. In particular, it is possible to achieve a wider range of 22
- 31. stylistic effects using the following continuous ramp function: Tϵ,φ(u) = 1 u ≥ ϵ 1 + tanh(φ · (u − ϵ)) otherwise (20) Equations 19 and 20 taken together is referred to as the XDoG filter Tϵ,φ(Dσ,k,τ ∗ I) for a given image I. There are a lot of known edge detectors that can be applied to find the edges of objects present in an image, and they have their advantages and disadvantages (as shown in [9]). Since the objective was not only to extract contours but also to obtain something that looks like a sketch done by an artist, not all the edge detectors that are known could be applied. As can be seen from Figure 8, some edge detectors work well in extracting edges, like the Sobel filter and the Canny edge detector, but the results obtained are not satisfactory from an artistic point of view. Hence, the XDoG operator has been chosen since it satisfies the final purpose. Figure 8: Figure taken from [9] in which there is a comparison between different known edge detection techniques. 23
- 32. 3.5 Dataset building To build the needed dataset, the following approaches have been followed: 1. ArtLine combined with Learning to Simplify/Mastering sketching 2. Extended Difference of Gaussian (XDoG) edge detector together with Mastering sketch- ing The initial idea was to take the images and transform them into line art drawings using ArtLine and then simplify the results by removing some lines from the obtained image. The results obtained applying only ArtLine can be seen in Fig. 9, they are good and give the impression that they are made by an artist. Nevertheless, they do not seem like sketches made with a pencil because all the people’s hair and beard are coloured in black. (a) Original imag (b) ArtLine’s output (c) Original image (d) ArtLine’s output Figure 9: Example of outputs obtained by using ArtLine Therefore, there was a need to try to simplify these images, by applying sketch simplification which can be obtained thanks to Learning to Simplify and Mastering sketching. The LtS approach can be used with a model obtained by training the network using only the MSE loss, while the mastering sketching has the following models available: • a model obtained from training the network using MSE and GAN loss using both supervised and unsupervised training data 24
- 33. • a model for pencil drawing generation based on an artist with a dirty and faded pencil style (pencil1) • model for pencil drawing generation based on an artist with a clearer overlay pencil style (pencil2) The performance of the four models was evaluated and compared. The results of these models can be visualised in Fig. 10. The aim of this evaluation was to determine which model provided the best results for the desired application. After careful testing, it was discovered that the model with both the MSE and GAN loss, and the model pencil1 offered remarkable results, even if in different scenarios. The ArtLine network, as already specified, had the tendency to produce images with a high number of dark pixels, particularly in areas such as hair, beard and sometimes even in the background. To address this issue, the model was selected based on the number of dark pixels present in the image. When the image had less than 800 000 dark pixels, the MSE+GAN model was utilised. On the other hand, for images with more dark pixels, the model pencil1 was applied. (a) MSE model (b) MSE + GAN model (c) pencil1 model (d) pencil2 model (e) MSE model (f) MSE + GAN model (g) pencil1 model (h) pencil2 model Figure 10: Output image obtained applying Artline and one of the four models of Learning to Simplify After undergoing these two steps, the dataset obtained was lacking in quality as several sketches had lost some key facial features, such as lines of the lips or wrinkles. Applying the 25
- 34. LtS or the Mastering sketching methods to simplify the results from the Artline network was not the optimal solution, as LtS is designed to convert rough sketches into clean, simplified drawings. Although the inverted models of Mastering sketching performed well, some details were lost after applying ArtLine. The outcome of the training phase of the model responsible for generating a photo of a face from a sketch was disappointing, despite it had been trained for several days. This was due to the low quality of the dataset, which resulted in an insufficient number of high-quality sketches. To overcome this problem, further improvements were required to ensure that the dataset captured the necessary features for producing accurate facial sketches. The new approach considered was a two-step procedure, starting with the application of an edge detection operator and then simplifying the results. While there are many available edge detection algorithms, they each have their own strengths and weaknesses, therefore not all of them were suitable for this particular task, as the goal was not only to extract edges but also to produce results that resemble an artist’s sketch. Hence, the edge detector operator chosen was the XDoG operator. The results obtained with this operator are much more like drawings made by artists and are far better than the ones obtained using Artline. In Fig. 11 can be seen the results obtained with this operator used with these parameters: • ϵ = 0.01 • k = 200 • σ1 = 0.05, σ2 = k ∗ σ1 = 10 • ϕ = 10 (a) (b) Figure 11: Output image obtained applying the XDoG operator Even if the results were very good, the model trained with the MSE and GAN loss of 26
- 35. Mastering sketching was applied in order to simplify the “sketches”. The results had to be simplified also because the final goal is to obtain a photo of a face even if a person with no drawing ability does the sketch. The result of this further step can be seen in Fig. 12, they are significantly better compared to the ones obtained from the first approach, and they preserved a lot of details that were previously lost. (a) (b) Figure 12: Output image obtained applying Mastering sketching to the output image of XDoG operator 27
- 36. 4 Implementation In this chapter, the generative adversarial networks used to generate synthetic images start- ing from a sketch are described. 4.1 StyleGAN StyleGAN [22] is a state-of-the-art deep learning generative model developed by NVIDIA in 2018 to produce realistic-looking images. The model is based on Generative Adversarial Net- work (GAN) architecture, in particular, it is built upon the foundation of the ProGAN (2.3), and it takes inspiration from the concept of “style transfer” introduced by [6] to generate unique images. An approach that involves the use of neural representations to separate and recombine the content and style of images. StyleGAN is designed to generate high-resolution images of faces, but can also be adapted to generate other types of images as well. Its key feature includes the ability to generate high-resolution images while also controlling various aspects of the image style, such as colour, texture, and overall composition. The main idea behind StyleGAN is to use a deep neural network that is trained to generate images from a random noise input. This noise input is then transformed into a feature representation, which is fed into a generator network. The generator network uses a series of convolutional layers to upsample the feature representation and generate the final image. The generator in StyleGAN begins by using a learned constant input and then modifies the style of the image in every convolution layer based on the latent code. This direct control allows for the adjustment of the strength of the image’s features at various scales. The inclusion of noise directly injected into the network results in the automatic, unsupervised separation of high-level attributes such as pose and identity from stochastic variations like freckles and hair in the generated images. This design makes it possible to perform intuitive mixing and interpolation operations that are specific to different scales. The design of StyleGAN is different from the one of the traditional GANs, since the latent code is not provided to the generator through an input layer. However, the generator starts by using a 4 × 4 × 512 constant to start the image synthesis process, and uses a non-linear mapping network to map the input latent code z to an intermediate space. The non-linear 28
- 37. mapping function is a function f : Z → W where Z is the input latent space and W is the intermediate latent space. This mapping network outputs a vector that controls the style of the generated image by integrating it into the generator model through the adaptive instance normalisation layer. This vector provides the ability to dictate the style of the synthetic image. For simplicity, the authors set the dimensionality of both the input latent space and the intermediate latent space to 512 and the mapping is done using a 8-layer Multi- Layer Perceptron (MLP). This choice of dimensionality is arbitrary and could potentially be changed in other implementations. The purpose of the mapping network is to convert an input vector into an intermediate vector that governs different visual attributes. This is a challenging conversion since the input vector must adhere to the probability density of the training data, therefore it con- ducts to an entanglement of features. To solve this issue, the mapping network is used to create an intermediate vector that does not have to follow the distribution of the training data, allowing for a disentangled representation. Disentanglement representation refers to the ability of a machine learning model to identify and separate the underlying factors of variation in a set of data. This means that the model is able to identify and isolate the different aspects of the data that contribute to its overall appearance, such as colour, shape, texture, and lighting, among others. Given the inapplicability of previous methods for determining disentanglement in the latent space in this case, the authors introduced two new automated metrics: perceptual path length and linear separability. These metrics enable them to quantify the disentanglement of the generator. Perceptual path length measures the smoothness of the generated images in the latent space. It is based on the idea that a small step in the latent space should correspond to a small change in the generated image. Therefore, models with low perceptual path length are expected to have a more continuous and smooth latent space. Linear separability, on the other hand, measures the degree to which different factors of variation are disentangled and can be independently manipulated in the latent space. A model with high linear separability is expected to have a latent space in which each dimension corresponds to a specific factor of variation, such as pose, identity, or background. This means that each dimension can be manipulated independently to generate new images with a specific combination of features. Their results show that compared to a conventional generator architecture, their generator 29
- 38. allows for a more linear and less entangled representation of various factors of variation. The output of the mapping network is 512 × 1, the same size as the input layer. The Style- GAN generator model is now referred to as the “synthesis network” due to the integration of the new mapping network into its architecture, its architecture can be seen in Fig. 13(b), while Fig. 13(a) shows a traditional GAN architecture. Figure 13: (a) traditional GAN architecture, (b) Style GAN architecture. Image taken from [22]. The mapping network consists of 8 layers, while the synthesis network g consists of 18 layers (two for each resolution starting from 42 ). The “A” block stands for the affine transform, and the “B” is the block that applies per-channel scaling factor to the input Gaussian noise. The AdaIN (Adaptive Instance Normalization) layer [10] is an extension of the Instance Normalization layer which can adapt to different styles instead of being able to normalise to a single, specified style. The AdaIN layer integrates information from a learned encoding of the style of the image into the generator network to allow for control over the style of the generated image. This layer is added at each resolution level of the synthesis network to provide a unique visual expression of features at each level. The AdaIN modules shift and scale the output through the layers in the synthesis network, promoting the significance of relevant filters in each convolutional layer. This helps the generator differentiate between important and insignificant features. It can be considered as an internal feedback mecha- 30
- 39. nism, where the mapping network learns to encode the W vector by focusing on the most relevant features. The AdaIN operation starts by normalising each channel to have zero mean and unit variance, and then it provides style information to apply scales and biases to each channel. The scale and bias values adjust the relative importance of each feature in the subsequent convolution operation. However, the original statistics of the input are not taken into account because of the normalisation step. As a result, each AdaIN operation only controls one convolution before being overridden by the next AdaIN operation. The AdaIN operation receives a feature map xi and a style input y = (ys, yb), and it is defined as: AdaIN(xi, y) = ys,i xi − µ(xi) σ(xi) + yb,i (21) where µ(xi) is the mean, σ(xi) is the variance, while for scaling and shifting ys,i and yb,i are used respectively. Since y consists in a scaling and shifting factors, its dimensional- ity is twice the number of feature maps on that layer. This process helps to preserve the content information from the content input while allowing the style information from the style input to be applied to the output. Additionally, each layer of the synthesis network is fed with uncorrelated Gaussian noise before the AdaIN operation in order to generate stochastic details. Incorporating random noise at each style block enables more precise con- trol over the model’s generated details, including finer features like freckles, beard, wrinkles, and dimples. Without this noise, these small details would be overpowered by larger, more prominent features. It is evident that the noise only impacts the stochastic elements, pre- serving the overall composition and high-level features such as identity. Moreover, the mixing regularisation technique is used to promote even more the localisation of styles in the generator. During training, a portion of the images is generated using two random latent codes instead of just one. This process, called “style mixing”, involves switch- ing from one latent code to another at a randomly chosen point in the synthesis network. The mapping network processes the latent codes, z1 and z2, producing the corresponding styles w1 and w2, which respectively control the styles applied before and after the crossover point. This regularisation method prevents the network from assuming a correlation between adjacent styles. The authors of StyleGAN used the WGAN-GP loss function (2.2) for evaluating their method. 31
- 40. The experiment conducted by Karras et al. [22] shows that the redesign of the generator not only preserves image quality, but also it significantly enhances it. To demonstrate this, the Fréchet Inception Distance (FID) was computed for various generator architectures on a few available datasets. The authors’ results suggest that a style-based design is superior to the conventional GAN generator architecture in all aspects. This superiority is demonstrated through established quality metrics, and their findings on the separation of high-level fea- tures and stochastic effects, as well as the linearity of the intermediate latent space, all point towards a deeper understanding and control of GAN synthesis. Furthermore, their path length metric can be included as a regularisation term during training, and a similar approach could be applied to the linear separability metric. 4.2 StyleGAN2 StyleGAN2 (2020) is an improvement of (4.1) for synthesising high-resolution images de- veloped by NVIDIA. This new version is based on the previous one, but with some dif- ferences [12]. The authors of StyleGAN2 discovered that styles were not only capable of capturing and transferring the essence of an image, but also transferring visual distortions such as water droplets or smudges. They linked these visual distortions to the AdaIN lay- ers, and therefore they made some changes to them in order to eliminate the artefacts. To overcome this issue, they came up with a new approach to the AdaIN layers, which were previously used as direct inputs to the network. Instead, they integrated the AdaIN layers within the convolutional layers. The authors replaced the AdaIN operator with a weight modulation and demodulation step, where the modulation adjusts each feature map of the convolution based on the style. The authors discovered that the strong location preference for facial features in StyleGAN images was due to progressive growing. To address this, they took inspiration from the “Multi-Scale Gradients for Generative Adversarial Networks” paper by Animesh Karnewar and Oliver Wang [11] and incorporated the idea of multiple scale gradient using a single architecture. They developed a new architecture, which can be seen in Fig. 14, that employs multiple scales of image generation, which is achieved through the utilisation of a resnet- style skip connection that links lower resolution feature maps to the final generated image. A further difference between the two version of StyleGAN is the introduction of a normali- 32
- 41. Figure 14: StyleGAN2 architecture. Image taken from [12]. Fig (a) and (b) show the StyleGAN architecture and its detailed view. In fig (c), the StyleGAN architecture in which the AdaIN layers have been replaced with the modulation is illustrated. Fig. (d) shows the revised architecture in which the instance normalisation has been replaced with a “demodulation” operation. sation term to the loss function to make the latent space more uniform and smoother. By doing so, the mapping of images in and out of the latent space becomes more controlled, en- abling more controlled image generation. Additionally, this improved relationship between the latent space and generated images allows creating images along a path in the latent space. The regularisation term added to the loss is driven by the goal of maintaining the expected lengths of vectors regardless of their directions, and it is formulated as: Ew,y∼N (0,1) = (∥Jw T y∥2 − a)2 (22) where y are random images with normally distributed pixel intensities, w ∼ f(z), where z are normally distributed. The local metric properties of the generator’s mapping function g(w) : W → Y are represented by the Jacobian matrix Jw = ∂g(w)/∂w. The value of the constant a is dynamically determined during the optimisation process through the calculation of the long-term exponential moving average of the vector lengths ∥Jw T y∥2. This allows the optimisation to automatically find an appropriate global scale for the process. The regularisation term is known as path length regularisation, and its implementation has proven to be highly beneficial in practice, leading to the development of more stable and consistent models. This makes the process of exploring and experimenting with different architectural designs much more manageable. Additionally, it has been noted that the smoother nature of the generator as a result of this regularisation makes it significantly 33
- 42. easier to perform inversion. Additionally, the authors [12] observed that there was no need to compute the regularisation term every time the loss was computed, therefore in this way they could obtain a decrease in the computational cost and also in the overall memory usage. They applied a lazy regularisation and evaluate the regularisation terms every k training iterations. To summarise the improvements made in the second version of StyleGAN are: 1. weight demodulation 2. path length regularisation 3. lazy regularisation 4. no growing These improvements made it also possible to speed up the learning time of the model up to 40% and optimise the images quality. 4.3 pSp Framework Pixel2Style2Pixel (pSp) is a novel image translation framework that leverages the powerful representation capabilities of a pre-trained StyleGAN2 generator and the extended W+ latent space [18]. The pSp framework is designed to tackle the task of transforming an input image into a target image, preserving the content of the original image while adopting the style of another. In the paper, a new encoder network is proposed, it is based on a Feature Pyramid Network capable of directly encoding real images into the W+ space. The encoder that the authors built is able to directly reconstruct real input images, enabling latent space manipulations without the need for time-consuming optimisation. However, this approach is limited as the input image must be invertible, meaning that it must have a latent code that can reconstruct it. This is a critical limitation since in conditional image generation, the input image does not stay in the same StyleGAN domain. The solution proposed is to utilise the encoder jointly with the pre-trained StyleGAN generator as a complete solution for image-to-image translation, thereby overcoming the limitation of the encoder’s requirement. The input images are transformed into the desired output latents through direct encoding, which are then processed by StyleGAN to generate the desired output images. In this 34
- 43. way, StyleGAN can be used for image-to-image translation also when the input and output images belong to different domains. The straightforward way to encode an input image into the W+ is to extend the encoder backbone with a feature pyramid which allows to produce three levels of features maps of spatial resolution (coarse, medium and fine). The styles are extracted from different scales of the pyramid using an intermediate network called map2style, and are inserted directly into the fixed, pre-trained StyleGAN2 generator in correspondence to their scale to generate the final output image. The map2style is a small fully convolutional network which reduces the spatial size using a set of 2-strided convolutions followed by LeakyReLU activation functions. It is trained to extract the learned style from the corresponding feature map for each of the 18 target styles. Figure 15 illustrates the pSp architecture. Figure 15: Pixel2Style2Pixel architecture. Image taken from [18] Additionally, they wanted to design the encoder to learn the latent code based on an average style vector in order to improve the initialisation. Therefore, they defined an average style vector w̄ of the pretrained generator, and given an input image x the output of their model is defined as: pSp(x) := G(E(x) + w̄) (23) where E(·) is their encoder while G(·) is the StyleGAN2 generator. For the loss function, the authors used a weighted combination of several functions, since they observe that this combination allowed to obtain a more accurate encoding into Style- GAN, defined as: L(x) = λ1L2(x) + λ2LLP IP S(x) + λ3Lreg(x) + λ4LID(x) (24) 35
- 44. where λ1, λ2, λ3, λ4 are constants used to define the loss weights, and L2 loss is the pixel-wise loss: L2 = ∥x − pSp(x)∥2 (25) LLP IP S is the LPIPS loss to learn perceptual similarities, with F(·) denoting the perceptual feature extractor: LLP IP S = ∥F(x − pSp(x))∥2 (26) Lreg is the regularisation loss to guide the encoder to produce latent style vectors that are closer to the average latent vector: Lreg = ∥E(x) − w̄∥2 (27) and finally LID is a loss measuring the cosine similarity between the output image and its source since their aim was to preserve the input identity: LID(x) = 1 − ⟨R(x) − R(pSp(x))⟩ (28) where R represents the ArcFace pretrained network [5], which is a type of loss function that aims to increase the separability between classes by adding a margin to the cosine similarity between features of an image and the weight vector of the classifier. 4.4 Restyle In the area of unconditional image synthesis, Generative Adversarial Networks have recently achieved outstanding results. It is essential for trained GANs to be able to invert an image into its matching latent code, since doing so allows manipulating real-world images while exploiting the network’s rich semantics. One of these models is StyleGAN, which is capable of synthesising highly realistic images, as discussed in 4.1. However, there is a major draw- back to StyleGAN: it only allows for the synthesis of new images in a specific style, meaning that it cannot change the style of an existing image. This limitation has motivated the development of the ReStyle algorithm [2], which is a residual-based StyleGAN encoder that can change the style of an existing image. The ReStyle algorithm is based on the iterative refinement of a residual encoding that transforms 36
- 45. the input image into a feature representation compatible with the StyleGAN generator. In this way, ReStyle permits to transfer the style from a pre-trained StyleGAN2 model to a given input image. In the paper “ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement” [2], the authors presented a new encoder-based inversion approach for a more accurate image inversion. They observed that achieving precise inversion in a single attempt inflicts a heavy constraint on the training process, so they introduced a novel method, called ReStyle, that employs an iterative process to carry out the inversion. This method uses a feedback mechanism to repeatedly feed the output of each iteration back into the encoder together with the original input image. This enables the encoder to use the knowledge gained from previous iterations to focus on the important parts necessary to reconstruct the input image with a certain level of accuracy. The objective is to train an encoder E capable of inverting real-world images into the extended W+ latent space of a previously trained StyleGAN generator G. Where the goal must be achieved through multiple steps N > 1, with each step consisting of a single forward pass through both the encoder and the StyleGAN generator. From a mathematical point of view, the purpose was to generate an image ŷ = G(E(x)), where x is a given input image, in such a way that ŷ ≈ x. To reach this goal, the encoder was trained while the StyleGAN generator remained fixed, since it is pre-trained. ReStyle training To train the encoder network and enable it to perform the inversion task, a set of losses are introduced. The losses used, which are also used by most encoder-based methods, are a L2 loss, which measures the difference between the reconstructed image and the original image pixel-wise, and a perceptual loss, such as LPIPS, which measures the difference between the two images in terms of their high-level features. The training process of the encoder is guided by the losses, which are computed at each forward step. The encoder weights are then updated according to the computed losses via back-propagation. In the inversion process at step t, ReStyle operates by concatenating the current prediction of the reconstructed image ŷ with an input image x. This results in an extended 6-channel input, represented as xt: xt := x∥ŷt (29) 37
- 46. At this point the encoder E has to compute the residual code based on the latent code just predicted, and then it updates the new prediction for the latent code using the previous latent code wt, as shown in equations 30 and 31, respectively. ∆t := E(xt) (30) wt+1 ← − ∆t + wt (31) The updated prediction of the reconstructed image is obtained by passing the latent code wt+1 to the generator, as defined in equation 32. Finally, the updated estimate ŷt+1 is used in the following step, as specified by equation 29. ŷt+1 := G(wt+1) (32) The initialization of the inversion process consists in starting with an average latent code w0 and its corresponding image ŷ0, the overall process can be seen in Figure 16. Figure 16: ReStyle’s iterative inversion process. Image taken from [2] ReStyle Encoder architecture The ReStyle encoder (Fig. 17) is designed as a variation of the architecture described in 4.3 and by [23]. Instead of obtaining the style features from three intermediate levels in the encoder, all style vectors are derived from the final 16×16 feature map. To do so, k different map2style blocks are employed to reduce the size of the feature map and produce the corresponding 512-dimensional style input, where k is the number of style inputs in the StyleGAN generator, as introduced in 4.3. 38
- 47. Figure 17: Simplified ReStyle encoder architecture. Image taken from [2] ReStyle conclusion The ReStyle scheme has been shown to produce highly realistic results in a variety of experiments. It is capable of transforming the style of an image while preserving its content, which is a crucial property for style transfer algorithms. Additionally, the ReStyle algorithm is highly flexible, as it can be trained on different styles by using different pre-trained StyleGAN models. This allows for the creation of a large variety of style transfer models, each with a unique style. 39
- 48. 5 Setup for training and testing Since the goal of this thesis is not to create a new type of generative adversarial network, a pre-existing one have been utilised. The aim was to train the existing network, specifically the ReStyle scheme, discussed in 4.4, over a variation of the encoder proposed in the pSp framework (4.3), to generate high-quality face images when conditioned with face sketches. The dataset used to train the model is the one obtained utilising the techniques explained in 3. 5.1 Training’s setup The official version of ReStyle published by Alaluf [2] is used as a base model to perform the training. The model has been trained on NVIDIA GRID A100D-2-20C GPU, with 11.4 CUDA, and 20GB of memory. The training process involves setting up the dataset to be used with ReStyle, therefore the following updates in the ReStyle directory are needed. The config/path_configs.py has to be update indicating the position where the images, for both training and testing, are saved: dataset_paths = { ’s_train ’: ’ffhq/sketch/’, ’f_train ’: ’ffhq/orig/’, ’s_test ’: ’ffhq/sketch_test /’, ’f_test ’: ’ffhq/orig_test/’ } The configs/paths_config.py has to be updated, defining the path to the images that have to be used for training and testing. The dataset has to be configured in this way: DATASETS = { ’sketch_to_face ’: { ’transforms ’: transforms_config . SketchToImageTransforms , ’train_source_root ’: dataset_paths [’s_train ’], ’train_target_root ’: dataset_paths [’f_train ’], ’test_source_root ’: dataset_paths [’s_test ’], ’test_target_root ’: dataset_paths [’f_test ’], } } 40
- 49. In the config/transforms_configs.py this code has to be added: class SketchToImageTransforms ( TransformsConfig ): def __init__(self , opts): super( SketchToImageTransforms , self).__init__(opts) def get_transforms (self): transforms_dict = { ’transform_gt_train ’: transforms.Compose ([ transforms.Resize ((256 , 256)), transforms.ToTensor (), transforms.Normalize ([0.5 , 0.5, 0.5] , [0.5 , 0.5, 0.5]) ]), ’transform_source ’: transforms.Compose ([ transforms.Resize ((256 , 256)), transforms.ToTensor ()]), ’transform_test ’: transforms.Compose ([ transforms.Resize ((256 , 256)), transforms.ToTensor (), transforms.Normalize ([0.5 , 0.5, 0.5] , [0.5 , 0.5, 0.5]) ]), ’transform_inference ’: transforms.Compose ([ transforms.Resize ((256 , 256)), transforms.ToTensor (), transforms.Normalize ([0.5 , 0.5, 0.5] , [0.5 , 0.5, 0.5]) ]) } return transforms_dict Additionally, the StyleGAN and IR-SE50 pre-trained models have to be downloaded and placed in the pretrained_models folder. Once all of these steps are performed, the training phase can start using the following command: python scripts/ train_restyle_psp .py --dataset_type = sketch_to_face --encoder_type = BackboneEncoder --exp_dir=experiment/ restyle_psp_s_encode --workers =8 --batch_size =8 --test_batch_size =8 --test_workers =8 --val_interval =5000 --save_interval =10000 --start_from_latent_avg --lpips_lambda =0.8 41
- 50. --l2_lambda =1 --w_norm_lambda =0 --id_lambda =0.1 --input_nc =6 --n_iters_per_batch =5 --output_size =1024 --stylegan_weights = pretrained_models /stylegan2 -ffhq -config -f.pt where dataset_type is used to specify the name of the dataset to employ and encoder_type define the encoder type to apply, that in the case of facial domain is the BackboneEncoder. The parameter exp_dir indicates the output directory in which the generated images and the weights of the network will be saved. workers and test_workers represent the number of subprocesses that are activated to speed up the training process by dividing the workload among them. The batch_size and test_batch_size are the number of samples that are passed into the training loop at each iteration. The parameter val_interval is used as a validation parameter to evaluate the performance of the model when the global step is a multiple of this parameter. save_interval parameter is used to define after how many steps the model has to be saved. To indicate the will to start from an average image, the parameter start_from_latent_avg has to be included, and this image is the one showed in Fig. 18. The parameter lpips_lambda, l2_lambda, w_norm_lambda and id_lambda are the λi used to weight differently the various losses defined in 4.3. To specify the number of iteration for each batch and the output image size, the input parameters to set are n_iter_per_batch and output_size respectively. The last parameter is used to specify the path to the directory which contains the weights of the pretrained StyleGAN2 generator. Figure 18: Average image used 42
- 51. Initially, the first dataset was used to train the network for a duration of three days. However, the results obtained from this training were not up to the expected standards. Subsequently, the second dataset was employed for training the network from scratch for three days. This resulted in a noticeable improvement in the network’s performance. Nevertheless, some crucial features were still missing from the output when testing the model. For example, when the sketch presented a person with the beard in the generated image this detail did not appear, and the same happened for details like dimples and wrinkles. To address this, the network was further trained for other three days. Indeed, the model was trained for a total of 70’000 iterations and the end result was a trained generative adversarial network capable of producing realistic face images when conditioned on input sketches. In Fig. 19 it can be seen the result of the training phase. (a) Train: 30’000-th iteration (b) Train: 70’000-th iteration Figure 19: Training results 43
- 52. 5.2 Testing’s setup For the purpose of generating synthetic images with the trained model, a graphical user interface has been developed, along with a client-server architecture. All these three are done in Python using the RPyC library. This setup enables the model to be tested in a convenient and efficient manner, as the graphical interface provides an easy-to-use interface for inputting data and analysing the results, while the client permits to test the model from the command line inputting existing sketches. Graphical user interface The graphical user interface was done using Tkinter library, which allows users to draw on a canvas using a black pen. Additionally, users can use a rubber to delete portions of the sketch, or directly remove the previous inserted lines using the “undo” button, or also clear the canvas, with the “delete” button, and start again from scratch. The rubber and the pen can vary in thickness between 1 (thinner) and 10 (thicker). Furthermore, three dashed lines are present, one is vertical in the centre of the interface and the other two are horizontal and placed at eye and chin level. Those lines guide the users in positioning main facial features like the eyes, nose, and mouth. The Fig. 20 shows a sketch drawn on the interface. Once the “send” button is pressed, the drawing is sent to the server. To start the interface, the following command has to be executed. python interface.py [-h] [-p PORT] [-a HOST] Options: -h this help -p PORT remote server port , default: 18862 -a HOST remote server address , default: 140.105.164.237 Client architecture An image can also be generated from the command line typing: python client_next .py [-h] -i IMAGE [-o OUT] [-p PORT] [-a HOST] Options: -h this help -i IMAGE input image -o OUT output file , default: output_results .jpg -p PORT remote server port , default: 18862 -a HOST remote server address , default: 140.105.164.237 44
- 53. Figure 20: Graphical interface The input sketch has to be specified, while there are other parameters that can be set are not mandatory. Server architecture To be able to generate the image based on the sketches the server has to be up, therefore the following command has to be executed otherwise it will not be possible to generate images. python server_next .py [-h] -m MODEL [-t TYPE] [-p PORT] Options: -h this help -m MODEL pretrained model -t TYPE type of the mode , either psp or e4e , default: psp -p PORT remote server port , default: 18862 Once an image is sent to the server, the restyle encoder is applied to generate the output image, which is then sent back as a byte stream and then saved as a PNG image. The server allows also multiple clients since it used the ThreadedServer class of RPyC. 45
- 54. 6 Evaluation and results Throughout this thesis, a few experiments were conducted to evaluate the performance of the trained model in terms of quality of the generated images. The results of these experiments have been consolidated in this section. To assess the quality of conditional image synthesis, a survey was prepared, comprising 20 distinct sketches, each associated with four images. Among the four images, one was generated from the sketch, while the other three were selected from the FFHQ dataset based on the similarity metric described in 2.4.1. The FFHQ dataset has been chosen due to the fact that it was the dataset used to train the model. The 20 images were chosen from 3256 sketches done by people during a scientific research festival on a graphic tablet. These sketches were made on the graphical interface described in 5.2. In the process of collecting the sketches, it was observed that if the size of the sketches was too small or if the position of the eyes was too high or low on the graphical interface, the resulting generated image did not accurately resemble the original sketch. To address this issue, two horizontal lines and a vertical line, discussed in section 5.2, were added. This allowed individuals to have a better understanding of the appropriate size of the sketch and the correct placement of the main facial characteristics to ensure accurate image generation. Furthermore, it has been observed that the sketches made using a pen with a thickness greater than three tend to result in bad images, despite the talent of the artist who created the sketch. Fig. 21 shows an instance of unsatisfactory outcome achieved from a highly realistic sketch, which was produced using a very small scale. Figure 21: Disappointing result generated from a sketch too small As illustrated in Fig. 22, the use of a pen too thick can have a negative effect on the 46
- 55. generated images, for example it can generate unrealistic hair and colours. However, upon closer examination of the generated image, it can be observed that certain details of the original sketch are still present. Figure 22: Result obtained from a sketch made with a thick pen Impressive results can be achieved when talented people create the sketches. It is worth noting that the success of the generated image is highly dependent on the quality of the input sketch. Where the term “quality” is related to the dimension of the sketch and the pen’s thickness. Therefore, it is essential to carefully consider the conditions under which the sketch is created to obtain optimal results. An example of a very breathtaking result can be seen in Fig. 23, while Fig. 24 depicts a straightforward input sketch and its corresponding generated image that exhibits noticeable similarity. Figure 23: Impressive result of an image generated from a sketch Given the diverse range of people who contributed to the sketch dataset, consisting of individuals with varying age groups and drawing abilities, it became necessary to filter out sketches that were unrealistic. Furthermore, some sketches displayed minor differences, as shown in Fig. 25, or only portrayed the outline of the face without any distinguishable feature (Fig. 26). 47
- 56. Figure 24: Result of an image generated from a simple sketch (a) (b) Figure 25: Examples of two similar sketches (a) Empty sketch (b) Sketch with just the contour of the face (c) Sketch done with a thick pen Figure 26: Examples of discarder images 48
- 57. Therefore, the initial set of sketches was manually processed to remove such sketches, leaving only the more realistic ones. Subsequently, the remaining images underwent further analysis, and the poorest sketches were discarded. Finally, the xCos metric was applied to the filtered dataset to identify the most similar images to the sketches. The survey was built in such a way that 5 random images out of the 20 are displayed. The survey’s participants were instructed to choose the image that in their opinion had inspired the realisation of the sketch. An example of how the images are displayed in the survey is shown in Fig. 27. (a) (b) Figure 27: Examples of the structure of the survey 49
- 58. 6.1 Survey’s results Out of the 202 responses collected from the survey, only 155 were considered for the analysis because the remaining 47 came from people try to complete the survey a second time. Figure 28 (a) and (b) provide an overview of the results obtained from the survey. The figure displays the total number of correct and incorrect responses for each sketch, indicated by green and red bars, respectively. The sketches are labelled on the x-axis. Table 1 shows the percentage of correct responses and the average response time in seconds for each sketch in the survey. By looking at the column of the percentage of correct responses, two values stand out, namely the percentage of correct responses for the sketches with the IDs 2 and 8. These two sketches have reported the smallest and the greatest percentage of correct responses, respectively. The sketch with ID 2 can be seen in Fig. 29, where in Fig 29(a) each bar represents the number of responses obtained for each image. The third image in the plot obtained the highest number of responses, which may be attributed to the thick eyebrows depicted in the Figure 28: Results for the first 10 sketches (a). 50
- 59. Figure 28: Results for the second 10 sketches (b). sketch, and also because the drawing gives the impression that it is a male subject. The average time spent for each image is shown in Figure 29(b). The results obtained for the sketch with ID 8 are shown in Fig. 30, where it can be observed that this sketch obtained the highest number of correct responses, most likely because the correct image was easily identifiable. Looking at the average time, for almost all the images the time taken to choose a response is the same, exception made for sketch 19 and 10 that respectively obtained a lower and higher value than average time. In Figure 31, the number of correct/incorrect responses and the average time for each similar image associated to sketch 19 are shown, and the same information are represented in Fig. 32 for sketch 10. Before drawing any conclusions, it is of extreme importance to determine whether the pro- portion of correct answers is due to an actual understanding of the difference between the proposed images and people could actually determine which of the images was inspired by 51
- 60. SketchID % correct responses average time (s) 0 23.4 % 19.79 1 67.3 % 14.13 2 18.8 % 16.59 3 87.0 % 11.04 4 57.4 % 14.08 5 73.1 % 16.08 6 80.4 % 14.52 7 86.2 % 10.09 8 93.0 % 12.48 9 73.6 % 14.28 10 53.8 % 21.19 11 92.2 % 14.00 12 82.9 % 19.45 13 37.9 % 12.55 14 64.0 % 13.78 15 37.9 % 19.53 16 70.3 % 15.43 17 72.2 % 13.62 18 46.9 % 18.90 19 80.0 % 9.43 Mean: 64.9 % 15.05 Table 1: Summary of the results obtained (a) (b) Figure 29: Sketch with the greatest percentage of incorrect responses 52
- 61. (a) (b) Figure 30: Sketch with the greatest percentage of correct responses (a) (b) Figure 31: Sketch with the lowest average time the sketch, rather than just guessing the correct answer. Therefore, the binomial test was performed, to calculate the likelihood of getting a certain number of successes, based on the assumption that the result was randomly obtained. The test essentially checks if the results 53
- 62. (a) (b) Figure 32: Sketch with the highest average time are statistically significant or just due to random chance. The null hypothesis considered in this test is that there is no significant difference between the observed proportion of success and the expected proportion of success by chance, while the p-value represents the probability of observing more extreme outcomes than the observed ones. If the p-value is below a predetermined significance level, the null hypothesis is rejected, and it is concluded that the observed outcome is unlikely to be due to chance alone. For each question in the survey there are four possible responses, but just one is correct, therefore the probability of guessing it correctly is 25%. The significance value alpha is set to 0.05 for all tests. The null hypothesis for the test states that the proportion of correct answers is equal to 25%, while the alternative hypothesis was that the proportion was greater than 25%. The results of the analysis, shown in Table 2, indicate that, for 18 out of the 20 questions, the p-value is less than alpha. This suggests that the percentage of correct responses is significantly higher than chance, meaning that it is unlikely that the correct responses are the result of random choices. For the questions related to sketch 0 and 2, the null hypothesis cannot be rejected, suggesting that the proportion of correct answer is not significantly different from chance. As can be noticed from Figure 33, the images proposed for these sketches appear to be very similar, 54
- 63. SketchID p-value 0 0.65 1 0.00 2 0.88 3 0.00 4 0.00 5 0.00 6 0.00 7 0.00 8 0.00 9 0.00 10 0.00 11 0.00 12 0.00 13 0.02 14 0.00 15 0.02 16 0.00 17 0.00 18 0.00 19 0.00 Table 2: Results of the binomial test making the selection more challenging. Overall, these findings suggest that the participants generally performed better than chance. Furthermore, this statistical test was also performed on the overall results, considering the number of correct responses out of the total responses, that is 509 out of 775. The p-value obtained in this case is 0.00. (a) (b) Figure 33: Images associated with sketch 0 and 2 55
- 64. An additional analysis was conducted to examine the correlation between the response time and the accuracy of the choices made. In particular, the Spearman correlation coefficient was used, as it measures the strength and direction of the association between two variables. Specifically, it assesses the monotonic relationship between the two variables, meaning that it measures how well the relationship between the variables can be described by a monotonic function. In order to determine if such correlation exists, the two variable considered are the percent- age of correct responses and the average time spent to choose the response for each sketch. The Spearman correlation coefficient obtained is −0.56 with a p-value of 0.01, results that indicate statistically significant negative correlation between the two variables. Therefore, as the average time spent to answer increases, the number of correct responses decreases. This outcome can be attributed to the possibility that individuals who paid closer attention to the images may have noticed several details that are also present in the sketch and in the images, which could have influenced their choice. However, additional investigations are required to determine whether or not individuals were genuinely misled by these de- tails. Figure 34 shows this negative correlation between the average time spent per response and the percentage of correct responses. The percentage of responses is represented on the x-axis, and the time spent per response is represented on the y-axis. Figure 34: Scatter plot representing the relation between the average time and the percentage of correct responses A further test was conducted during the process of collecting sketches, mentioned earlier 56
- 65. in this chapter, several people expressed their curiosity in testing the model’s capability to generate an image of the most dreamt person in the world as claimed by many people since 2006. The phenomenon was called “This Man” and it also has a dedicate website where people experiences are collected and shared, including the description of this man. Given the interest and the curiosity generated, the GAN model was used to attempt to produce the image corresponding to the man sketch. By feeding the sketch of this famous man into the trained model, the objective was to generate a visual representation of this person. The resulting image is illustrated in Figure 35, and it reveals the face of a woman maintaining the primary facial characteristics of the original sketch. Figure 35: Sketch of the most dreamt man and its corresponding generated image 57
- 66. 7 Conclusions In this thesis, a generative adversarial network (GAN) is trained on a large dataset of face images to explore its capability in synthesizing photorealistic images from face sketches. Then the performance of the model was qualitatively assessed to evaluate the results ob- tained. The results and progress attained throughout the process draw attention on the outstanding accomplishments of the model and its successfulness in generating photorealistic images from face sketches. Furthermore, the model’s achievements highlight the promising potential it can have if applied to various fields, such as digital art, entertainment, and criminal investigations. However, despite its remarkable performance, the model still has certain drawbacks that need to be addressed. Firstly, it may not be able to generate photorealistic images for all types of sketches, particularly those that are incomplete or poorly-drawn. Secondly, it does not generate images of all races equally, factor that can lead to a biased output. Moreover, it turned out to be challenging to generate images of children and younger people, as the model tended to generate an adult face most of the time. Lastly, the model is currently not able to capture some specific features present in the input sketches, such as piercings, tattoos, and freckles, therefore, when these are present they are not reported in the portrait. In addition, according to the analysis conducted on the qualitative assessment’s result, the ultimate outcome of the model highlights that the generated images are perceived to be realistic with a high degree of resemblance between sketches and generated portrait. Future developments in this field can potentially involve a more in-depth exploration of the use of Stable Diffusion when applied to the generation of photorealistic images based on face sketches. Given the speed at which this area of research is evolving, several further techniques and approaches have the potential to be leveraged in the future to improve the performance of the current model. For example, additional researches can focus on address- ing the identified limitations, and enhancing the model’s ability to capture such undervalued details. The proposed approach can also be adapted and applied to other domains beyond face image synthesis, such as generating photorealistic images of objects and scenes from sketches. Future researches can and should also explore the use of other evaluation metrics, 58
- 67. such as quantitative measures and other user studies. Finally, this research provides valuable insights on the potential of GANs in image generation and unwraps new possibilities for future research in this field. The results obtained in this study can be used as a foundation for further investigation and development of GAN-based models for various applications. 59
- 68. References [1] Cuhk. http://mmlab.ie.cuhk.edu.hk/archive/facesketch.html. [2] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: A residual-based style- gan encoder via iterative refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021. [3] Jason Antic. Deoldify. https://github.com/jantic/DeOldify. [4] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017. [5] Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. arXiv:1801.07698, 2018. [6] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2414–2423, 2016. [7] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. [8] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans, 2017. [9] Sven C. Olsen Holger Winnemöller, Jan Eric Kyprianidis. Xdog: An extended difference-of-gaussians compendium including advanced image stylization. Computers & Graphics, 36, 2012. [10] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017. [11] Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative adversarial networks. arXiv preprint arXiv:1903.06048, 2019. 60
- 69. [12] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020. [13] Yu-Sheng Lin, Zhe-Yu Liu, Yu-An Chen, Yu-Siang Wang, Ya-Liang Chang, and Win- ston H. Hsu. Xcos: An explainable cosine metric for face verification task. ACM Trans. Multimedia Comput. Commun. Appl., 17(3s), nov 2021. [14] Vijish Madhavan. Artline. https://github.com/vijishmadhavan/ArtLine. [15] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022. [16] Lanham Micheal. Generating a new reality. In Generating a New Reality, 2021. [17] NVIDIA. Ffhq dataset. https://github.com/NVlabs/ffhq-dataset. [18] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image transla- tion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021. [19] Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. Mastering Sketching: Adver- sarial Augmentation for Structured Prediction. ACM Transactions on Graphics (TOG), 37(1), 2018. [20] Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup. ACM Transactions on Graphics (SIGGRAPH), 35(4), 2016. [21] Karras Tero, Aila Timo, Laine Samuli, and Lehtinen Jaakko. Progressive growing of gans for improved quality, stability, and variation. In ICLR 2018, 2018. [22] Timo Aila Tero Karras, Samuli Laine. A style-based generator architecture for genera- tive adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. [23] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. arXiv preprint arXiv:2102.02766, 2021. 61
- 70. [24] H. J. Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. [25] X. Wang and X. Tang. Face photo-sketch synthesis and recognition. In IEEE Transac- tions on Pattern Analysis and Machine Intelligence (PAMI), pages 1955–1967, 2009. [26] X. Wang and X. Tang. Coupled information-theoretic encoding for face photo-sketch recognition.). In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR, 2011. [27] H. Winnemöller, S.C. Olsen, and B. Gooch. Real-time video abstraction. ACM Trans Graph, 25:1221–1226, 2006. [28] Ran Yi, Yong-Jin Liu, Yu-Kun Lai, and Paul L Rosin. APDrawingGAN: Generating artistic portrait drawings from face photos with hierarchical gans. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR ’19), pages 10743–10752, 2019. [29] R.A. Young. The gaussian derivative model for spatial vision: I. retinal mechanisms. Spatial Vision, 2, 1987. [30] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proceedings of the 36 th International Conference on Machine Learning, 2019. 62