GACNN: TRAINING DEEP CONVOLUTIONAL NEURAL NETWORKS WITH

GENETIC ALGORITHM

KARAN DIXIT
(1MV17CS046)

Student, CSE Dept.

Sir M. Visvesvaraya Institute of Technology

([email protected])

programming languages, like Python, and helpful libraries,

ABSTRACT such as Keras.

Convolutional Neural Networks (CNNs) have gained a Genetic Algorithm (GA), as one of the subsets of
significant attraction in the recent years due to their Evolutionary Algorithms, is a global optimization method
increasing real-world applications. Their performance is inspired by the process of natural selection for solving both
highly dependent to the network structure and the selected constrained and unconstrained optimization problems.
optimization method for tuning the network parameters. In Genetic algorithm repeatedly modifies a population of
this paper, we propose novel yet efficient methods for individual solutions by selecting the best individuals from the
training convolutional neural networks. The most of current current population as parents and using them to produce
state of the art learning method for CNNs are based on children for the next generation through a number of
Gradient decent. In contrary to the traditional CNN training bio-inspired operators. Over successive generations, the
methods, we propose to optimize the CNNs using methods population "evolves" towards an optimal solution. As the
based on Genetic Algorithms (GAs). These methods are training process of a CNN is basically an optimization
carried out using three individual GA schemes, Steady-State, problem, intuition suggests that GA can be used to do that.
Generational, and Elitism. We present new genetic operators
for crossover, mutation and also an innovative encoding In this work, there is a attempt to train two different deep
paradigm of CNNs to chromosomes aiming to reduce the convolutional neural network architectures doing an image
resulting chromosome’s size by a large factor. We compare classification task over two different modern datasets using
the effectiveness and scalability of our encoding with the methods based on genetic algorithm. These methods are
traditional encoding. Furthermore, the performance of carried out using three individual GA schemes, Steady-State,
individual GA schemes used for training the networks were Generational, and Elitism. Our training methods involve
compared with each other in means of convergence rate and novel genetic operators for crossover and mutation. In
overall accuracy. Finally, our new encoding alongside the addition, we introduce the Accordion chromosome structure,
superior GA-based training scheme is compared to an innovative encoding paradigm of the networks to
Backpropagation training with Adam optimization. chromosomes that reduces the chromosome’s size by a
large factor, leading to faster operations time.
1. Introduction

Recent years has seen the rapid growth of machine learning

applications to the real world problems, specially deep 2. Background
learning. [1] Convolutional Neural Networks (CNNs), being
one of the many classes of deep learning algorithms, are 2.1 Convolutional Neural Networks
proving to be one of the most effective and popular tools
used in fields such as computer vision and speech
Convolutional neural networks are a specialized kind of
recognition. Convolutional neural networks, with their
artificial neural networks used widely in the field of image
exceptional generality in finding good solutions and a
and video analysis. They are excellent tools for finding
property to tolerate noisy and uncertain data, are becoming
patterns which are far too complex or numerous for a human
the go-to candidate for solving many problems. In addition,
programmer to extract and teach the machine to recognize.
implementation, modification, testing, and application of
Artificial neural networks themselves are biologically inspired
CNNs for working on large sized datasets are becoming
computation systems intended to replicate animal brains.
easier and more convenient, thanks to higher-end
They are made up of simple processing elements, called
neurons, that interact with each other using a network of
weighted connections. Artificial neural networks are a In a genetic algorithm, a population of solutions (called
remarkable method for classifying noisy and uncertain data individuals, members, etc.) to an optimization problem is
and they can be trained to exceptional accuracies in a very evolved towards better solutions.
desirable time. However, as the problem gets complicated
and the network’s input size starts to grow, they fail to scale
A typical genetic algorithm requires a genetic representation
their performance. This failure arises from their inability to
of the solution domain and a fitness function to evaluate the
detect certain important features in the data, and instead,
solution domain. A standard representation of each solution
trying to report results using only the raw data. So, the
is by using an array. The main property that makesbthis
model’s designer has to realize these important features into
genetic representations convenient is that its parts are easily
the networks manually. But in most cases, these features are
aligned due to their fixed size, which enables simple
way to complex to be coded into a model. Or even worse,
crossover operations. Once the genetic representation and
they might be overlooked or undetected. To address this
the fitness function are defined, the algorithm proceeds to
issue, convolutional neural networks come to the rescue. An
initialize a population of solutions and then improving it
illustration of a simple convolutional neural network is
through repetitive application of the selection, crossover, and
depicted in figure Figure 1.

mutation operators.

Next describes the flow of how a genetic algorithm generally

operates.

1. Initialization

2. Evaluation

3. Selection

4. Genetic Operators

5. Termination

Figure 1: A simple convolutional neural network with two 3. Methodology

convolution layers, two pooling layers, and a single fully
connected (dense) layer for the classification of images into
3.1 Genetic Algorithm for Training Convolutional Neural
four classes; dog, cat, goat, and bird.
Networks

So, once an instance of the problem’s data is inputted

The challenge of using genetic algorithm for training a deep
through the network, that data instance is flowed and
convolutional neural network is in mapping the problem from
changed through each filter, layer and neuron of the network
the domain of artificial neural networks literature to the
until reaching the last layer and eventually, being outputted
domain of genetic algorithm literature. Meaning, how GA can
as some values. To measure the performance of the
be used to train a network. The initial intuition suggest that
network, a proper loss function must be chosen to evaluate
the networks should act as members of a population for our
how much the output is correct for any given labeled input
algorithm. A network’s performance, somehow, should
instance. By changing the parameters of the network,
represent the network’s fitness, and the algorithm’s iterative
specially its filter and weight values, the error of performance
process should "evolve" the population of networks towards
(i.e. the value of loss function) for different instances of the
better accuracies.
problems data changes. Optimizing this value configuration
so the loss function reports back the lowest possible value is
known as the training of a convolutional neural network. The first part of the challenge is the encoding problem, i.e.
how to map a network to a chromosome. Traditional
encoding paradigms extracts every single trainable
2.2 Genetic Algorithm
parameter in a network’s structure and then encapsulate
them as an array construct to achieve a chromosome
In computer science and operations research, Genetic representation of the network. In this manner, each element
Algorithm (GA) is a metaheuristic inspired by the process of of the chromosome array holds only a single network
natural selection that was introduced by John Holland in parameter.
1960 based on the concept of Darwin’s theory of evolution.
GA is most commonly used to generate high-quality
solutions to optimization and search problems by relying on
bio-inspired operators such as mutation, crossover and
selection.
 the generations. This step marks the completion of one
iteration of the steady-state scheme.

4. Evaluation Results

For our evaluation, first, the Accordion encoding is compared

to the traditional encoding in means of convergence rate and
overall accuracy. To do this, each encoding was used
alongside the three individual GA training schemes for two
network architectures each doing a classification task over a
different dataset. From this comparison, we can also
determine which scheme performs better than the others.
Finally, the better encoding alongside the superior training
scheme was compared to two backpropagation methods,
Stochastic Gradient Descent and Adam training. For the first
part of the evaluation, the training methods were used for
two network architectures each doing a classification task
Figure2: Our proposed encoding paradigm of a over the MNIST and CIFAR10 datasets, respectively. The
network into a chromosome. architecture description of these networks are demonstrated
in Table 2 and Table 3. The network architecture for the
3.1.1 Steady-State Genetic Algorithm for training MNIST task is custom design. But the network architecture
Convolutional Neural Networks for the CIFAR10 task is the famous LeNet network.

The steady-state scheme for training a convolutional neural We must also note here a very important factor in the fidelity
network consists of the following steps: 1. Initialization: In of our results, that is the initialization used for all of the
this step, some networks equal to pop_size are initialized training methods for each network is the same. Meaning
using Keras, with their convolution filter and connection that, for example, for the MNIST network, a population is
weight values being assigned with a random number drawn initialized, and then, this one and only population is evolved
from a truncated normal distribution centered on zero using using different encoding and training schemes. This results
the Keras’ built in Glorot normal initializer. 2. Evaluation: In in the same starting accuracy point for every comparison that
the duration of this step, each network’s performance is will be made. Also, when comparing to backpropagation
evaluated based on its accuracy reported by Keras’ methods later, the fittest member of the initialization
model.evaluate() function. This particular step is utilized in a population is selected to be trained with backpropagation.
parallel fashion using the multiprocess library in Python,
allowing for a faster program run time. 3. Fitness
Assignment: Each network is assigned a fitness value fi
based on its evaluation. Here, we used the accuracy of each At Figure 12a, the network for the MNIST classification task
network as its fitness value. was trained using the steady-state scheme using one time
the traditional encoding and one time using the Accordion
4. Selection: A selection probability is assigned to each encoding. It can be derived from this result that the
network in this step. For our work, we used one of the most Accordion encoding performs only slightly better than the
famous selection strategies; Fitness Proportional Selection. traditional encoding. This evaluation is also carried out for
Or most commonly known as, Roulette Wheel selection. In the CIFAR10 classification task. As shown in the results for
this selection, the higher the fitness of a network, the more this evaluation in Figure 12b, even though that the initial
probable it is to be selected as parent for reproduction. accuracy in the population of networks that were meant to be
trained using the Accordion encoding is lower than of its
5. Crossover: In this process, two parents will spawn a new same for the traditional encoding, the population trained
child sharing some of their attributes. For this operator, the using the Accordion encoding manages to surpass the
fully folded chromosome is not considered. Instead, a traditional encoding multiple time in the evolution progress.
semi-folded chromosome structure, that each of its elements And additionally, evolution using the Accordion encoding
are either the convolution filters or the entirety of the ingoing achieves a higher accuracy threshold compared to the
weights of neuron in the fully connected section. traditional encoding in the same number of iterations. In
short, the Accordion encoding has a faster convergence rate
6. Mutation: During this process, one parents will spawn a and reaches better accuracies against the traditional
new child sharing most of its attributes. The process clones encoding.
an exact copy of the parent’s chromosome and randomly
selects some of the elements in the semi-folded
chromosome structure.

7. Replacement: The new child is inserted into the next

generation of population to replace the least fit member. By
this manner, it will discard the replacing member from the
population. in this way, pop_size is kept the same throughout
 for crossover and mutation and a new encoding paradigm of
the networks to the chromosomes. This new encoding
paradigm delivers a significant reduction in the chromosome
size, by considering the entirety of a layer (weather its
convolution or fully connected) as genes in the chromosome.
These methodologies are carried out using three different
schemes of genetic algorithm, steady-state, generational,
and elitism. We first evaluated the performance of our novel
Accordion encoding against the traditional encoding. Our
results demonstrated that the Accordion encoding performs
better than the traditional encoding in means of convergence
rate and overall accuracy. We derived that even though GA
training falls short in overall, it performs better in the early
stages of the training, specially as the problem gets more
complicated. This delivers a promising property to head-start
(a) the training process with genetic algorithm.

References

● Jürgen Schmidhuber. Deep learning in neural

networks: An overview. Neural networks,
61:85–117, 2015.

● François Chollet et al. Keras. https://keras.io,

2015.

● https://towardsdatascience.com/a-comprehensive-
guide-to-convolutional-neural-networks-the-eli5-wa
y-3bd2b1164a53
(b)
● David J Montana. Neural network weight selection
Figure 12: Comparison of the Accordion encoding and using genetic algorithms. Intelligent Hybrid
traditional encoding used in the steady-state training scheme Systems, 8(6):12–19, 1995.
of (a) the network for the MNIST classification task and (b)
the network for the CIFAR10 classification task.
● Yann LeCun and Corinna Cortes. MNIST
handwritten digit database. 2010.
6. Conclusion

In this work, we proposed novel methods for training deep

convolutional neural networks based on genetic algorithm.
Our proposed methodology involves novel genetic operators