MCTS guided Genetic Algorithm for optimization of

neural network weights

Akshay Hebbar Department of Engineering and Syracuse University, [email protected]
Computer Science NewYork

Abstract— In this research, we investigate the possibility Genetic algorithms are used for various problem
of applying a search strategy to genetic algorithms to domains such as decision trees, segmentation,
explore the entire genetic tree structure. Several methods classification, etc. However, in this paper, our focus
aid in performing tree searches; however, simpler
algorithms such as breadth-first, depth-first, and iterative
will be on the application of GA in optimizing neural
techniques are computation-heavy and often result in a long network weights.
execution time. Adversarial techniques are often the
preferred mechanism when performing a probabilistic Selection
search, yielding optimal results more quickly. The problem
we are trying to tackle in this paper is the optimization of
neural networks using genetic algorithms. Genetic
algorithms (GA) form a tree of possible states and provide Mutation Crossover
a mechanism for rewards via the fitness function. Monte
Carlo Tree Search (MCTS) has proven to be an effective
tree search strategy given states and rewards; therefore, we Figure 1. Simple Genetic Algorithm
will combine these approaches to optimally search for the
best result generated with genetic algorithms.
The Monte Carlo Tree Search approach was
Keywords—genetic algorithm, mcts, optimization,
reinforcement learning, neural network
developed in 2006 as an application to game-tree
search. This tree search works on the principles of
I. INTRODUCTION cumulated reward calculated from children’s nodes
Genetic algorithms belong to a subclass of and uses Q and N values to balance between
evolutionary algorithms that provide a method for exploration and expansion approaches. The
optimization based on genetic selection principles exploration approach considers the number of nodes
[8]. They serve a purpose in machine learning and visited and uses a quantitative approach to
research development, in addition to search tool discovering child nodes that have not been visited.
optimization systems. The approach used in genetic The expansion approach follows a qualitative
algorithms is analogous to biological concepts of strategy to discovering child nodes with Q-value
chromosome generation, with operators such as indicating the cumulative sum of rewards.
selection, crossover, mutation, and recombination.
GA is a population-based approach that aims to
provide a solution to successive generations. The
process of evolution using GA involves starting with
a random population and evolving it using crossover
and mutation operators to generate children. The
best-fit solution is then filtered, and the genetic
process is repeated until the objective is achieved. We
can observe that in the process of genetic algorithms,
a tree of possible solutions is generated, and the best-
fit solution is picked for successive iteration, which
limits the search space and computational resources. Figure 2. Outline of a Monte Carlo Tree Search

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©2023 IEEE

 A sufficient policy for MCTS - from prior research A quick calculation for a tree with branching factor
- has been found to be UCT (upper confidence trees). of 10 and depth 10 shows the search space to be
UCT provides an upper confidence bound to tree 1,111,111,111. Genetic algorithms often perform
search. This policy helps the search with balancing better when the size of the tree is large and with
exploration vs exploitation and navigate the search increase in tree size the number of nodes generated
space optimally. Given that the MCTS is an increases exponentially and thus the search space
adversarial tree search strategy, we may be able to itself.
apply it to the entire GA tree landscape to find
B. Maintaining the Integrity of the weights
optimal solutions rather than exploration based on the
fitness alone. Thus, we discuss the novel approach of
MCTS-GA for optimization of neural network in this A well-known issue with genetic algorithms is the
research. problem of competing conventions wherein the child
nodes generated because of evolution are not viable
II. PROBLEM AND DATA DESCRIPTION and have decreased fitness. An example in case of
neural network is a cross over operator applied to the
A. GA for Neural Network weights
weights of the neural network. The operator shuffles
the weights and applies a random mutation which can
The first problem to consider when applying GA propagate between layers and dimensions. The
to optimizing the neural network weights is to modified weights may not be appropriate for
confirm the validity of the child nodes generated in optimizing the loss function of the given neural
the process. The crossover and mutation operator network and may lead to an invalid solution.
applied on these weights may result in a suboptimal
solution in the lower generation, but these may C. Data Description
evolve to become better solutions in the later
generations. Conversely, a solution which had The diabetes dataset is chosen to forecast the onset
highest fitness in earlier generations may end up of diabetes mellitus. This dataset is originally from
providing a sub-optimal result in the later generation the National Institute of Diabetes and Digestive and
due to the nature of genetic operators used. Thus, the Kidney Diseases. The objective of the dataset is to
only way to identify the overall best solution is to let diagnostically predict whether a patient has diabetes,
the entire tree expand and calculate fitness of each based on certain diagnostic measurements included in
child node until a valuable solution is found. the dataset [1]. The dataset has been preprocessed by
However, this solution is computation heavy, and an balancing using under sampling and random
exhaustive tree search is rarely optimal even in cases shuffling. A minmax scaler is used to scale the data.
of smaller trees. The number of nodes for a given tree
which makes up for the search space is given by the
below formula.
! ! "#
!"#
, branching factor b and height h

Figure 4. Heatmap of the diabetes dataset

For this classification problem we have developed
a feedforward neural network with 4 hidden layers.
There are 8 input nodes and (16-8-4-1) hidden nodes
Figure 3. Application of Monte Carlo Tree Search applied respectively in each layer.
to Genetic Algorithm
 The neural network uses sigmoid activation The selection mechanism used in this approach is the
function, binary cross entropy loss function and k-way tournament selection. Tournament selection
Adam optimizer with 0.01 learning rate. The network applies selection of k random individuals from the
is trained for 200 epochs in batch size of 10. population and selects the best fit among the
The weights of this neural network are used as the randomly selected individuals.
point of optimization for our MCTS-GA approach.
Each layer weights are vectorized, combined and
labelled to form an individual upon which the
algorithm is applied.

III. APPROACH
In our approach we try to address both the issues
mentioned above by combining the approaches of
MCTS and GA. We can take advantage of the genetic
algorithm structure as it generates a tree with a
mechanism to evaluate the reward in terms of the
fitness of the individual. MCTS will use the same
underlying tree structure along with the fitness
function to calculate the Q-value which indicates the
cumulative sum of rewards; the N-value which
indicates the visit count of each node is also
maintained for the calculation of upper bounds. The
overall structure follows the complete expansion of
child nodes using GA and using MCTS with UCT
policy to search for the optimal nodes with fitness
function as a method to assign rewards. Figure 5. Application of Monte Carlo Tree Search applied
to Genetic Algorithm
The process of evolution using GA involves
starting with a random population which in our case
is the weights of the neural network. We generate the Crossover: A crossover operator is then applied on
initial population P0 of certain size by adding a the initial population to generate children. 1-point
random uniform distribution on each dimension of crossover is applied where we have restricted the
the weights of the neural network and creating crossover operator to each layer of the neural network
children with random weights. to mitigate the problem of competing conventions
and maintain the integrity of the weights of the neural
Next, we introduce the concept of genetic action
network.
for MCTS as illustrated by dashed box in figure 5.
Mutation: The mutation operator is used to
The genetic action consists of the following three
introduce random mutation in the population. For our
genetic operations applied on the parent population.
approach we have used random mutation on each
• Genetic Action - layer of the neural network by swapping the weight
o Selection values of 2 randomly chosen individuals. The
subsequent child nodes are selected with the UCT
o Crossover
policy of MCTS from the root node until a leaf node
o Mutation is found. The children among the leaf nodes are
selected randomly and their corresponding Q and N
values are backpropagated. The UCT policy using
Selection: Various selection strategies such as UCB1 is as follows.
roulette wheel, tournament, rank, steady state etc. are
available for filtering individuals of higher fitness.
 UCT = Q + UCB1 The results obtained from primary testing has
proven to be positive. The MCTS-GA approach was
$∗&' (*+)
UCB1 = ! , Ni is the ith child node and N is able to optimize the neural network weights for better
*-#
classification of the diabetes data and obtained better
the number of child nodes accuracy results.
Equation 1. Upper confidence bounds
The comparison of the results obtained to that of
The next step in the process is the application of the original neural network and canonical genetic
the concept of rollout (simulation) based on the algorithm are shown below.
MCTS approach. For the selected child node, an
TABLE I. ACCURACY RESULTS
evolutionary rollout operation is applied wherein the
individual is evolved using (µ+l) evolutionary Neural Net
- SGD
Neural Net
-ADAM
Genetic
Algorithm
MCTS-GA
strategy unlike the regular random action simulations
performed in prior research experiments [3]. The accuracy 0.49 0.72 0.73 0.745
reason for applying the evolutionary rollout is to find recall 0.42 0.73 0.77 0.78
out if the genetically mutated individual is of the
highest fitness possible for its phenotype. We define
this process as ageing the individual in comparison to The classification accuracy can also be noticed from
the biological phenomenon of ageing. Thus, the the roc-auc curves indicated below.
ageing process introduced in rollout determines the
best possible age (generation) in which the individual
would be best suited to genetically mutate again. The
rollout/ageing process will replace the individual if a
better fitness is found in later generations of the
evolutionary strategy.
Figure 6. ROC-AUC curve for neural network, genetic
This process is repeated until the specified tree algorithm and MCTS_GA respectively
depth is reached. This approach provides
computational flexibility in terms of configurable
parameters such as tree height, tournament selection, The confusion matrix for the three approaches
number of rollout generations and branching factor. compared are shown below.
These parameters can be used in combination with
each other as per the computational capacity
available. Thus, the application of genetic action and
evolutionary rollout in amalgamation with MCTS
provides the basis for the approach discussed in this
paper.

Figure 7. confusion matrix for neural network, genetic

IV. RESULTS algorithm and MCTS_GA respectively
The MCTS-GA approach is a novel mechanism V. DISCUSSION
for optimization and is aimed to be data agnostic. The The experiment confirms the working of MCTS-
genetic algorithm representation can be configured in GA on optimization of neural net weights. The
multiple ways which makes this approach suitable for optimization of weights and thus the classification is
wide range of optimization problems. In this seen to be better achieved by the MCTS-GA over the
experiment MCTS-GA was run using UCT for 20 genetic algorithm and feedforward neural network
generations (tree depth) with rollouts configured to approach. Although, the improvement is not large
10 generations and branching factor 5. The GA was MCTS-GA does run in a comparable time in
run for 200 generation and the neural net was run for comparison to the other two techniques. There is
200 epochs. scope for improvement of the algorithm discussed
here and the representations of different problems are
to be tested. In all, we discussed a novel approach that [5] Sheta, Alaa & Braik, Malik & Aljahdali, Sultan. (2012). Genetic
Algorithms: A tool for image segmentation. Proceedings of 2012
can prove to be a strong and valid technique for International Conference on Multimedia Computing and Systems,
optimization techniques in the future. ICMCS 2012. 84-90. 10.1109/ICMCS.2012.6320144.
[6] G. Lo Bosco, "A genetic algorithm for image segmentation," Proceedings
11th International Conference on Image Analysis and Processing,
Palermo, Italy, 2001, pp. 262-266.
REFERENCES [7] K. Rocki and R. Suda, "Large-Scale Parallel Monte Carlo Tree Search on
GPU," 2011 IEEE International Symposium on Parallel and Distributed
Processing Workshops and Phd Forum, Anchorage, AK, USA, 2011, pp.
[1] Smith JW, Everhart JE, Dickson WC, Knowler WC, Johannes RS. Using 2034-2037, doi: 10.1109/IPDPS.2011.370.
the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus.
Proc Annu Symp Comput Appl Med Care. 1988 Nov 9:261–5. [8] A. Lambora, K. Gupta and K. Chopra, "Genetic Algorithm- A Literature
Review," 2019 International Conference on Machine Learning, Big Data,
[2] Goldberg, David E. Genetic algorithms. pearson education India, 2013. Cloud and Parallel Computing (COMITCon), Faridabad, India, 2019, pp.
[3] Silver, D., Schrittwieser, J., Simonyan, K. et al. Mastering the game of 380-384, doi: 10.1109/COMITCon.2019.8862255.
Go without human knowledge. Nature 550, 354–359 (2017).I. [9] Jonathan C.T. Kuo, "Genetic Algorithm in Artificial Neural Network and
[4] K. W. Lee and H. N. Lam, "Optimising neural network weights using its Optimization Methods","medium.com",Web, 2020.
genetic algorithms: a case study," Proceedings of ICNN'95 - International
[10] Sagar Sharma, "MCTS", "https://towardsdatascience.com/", Web, 2018.
Conference on Neural Networks, Perth, WA, Australia, 1995, pp. 1384-
1388 vol.3.