OPTIMIZATION TECHNIQUES IN MACHINE LEARNING

BY

SAJILA FEIZ

Synopsis submitted to University of Lakki Marwat in the partial fulfillment of the

requirements for the degree of

Phd IN MATHEMATICS

DEPARTMENT OF MATHEMATICS
UNIVERSITY OF LAKKI MARWAT
SESSION (2024-27)

OPTIMIZATION TECHNIQUES IN MACHINE LEARNING

1. INTRODUCTION
Optimization techniques are a core part of machine learning (ML). These techniques allow us to
make machine learning models better by fine-tuning the values of their parameters to get
accurate predictions and efficient training. In ML, a model’s performance depends on how well it
can make correct predictions, and optimization is the process that adjusts the model’s parameters
to minimize error and improve accuracy. This process is essential for almost every type of ML,
from simple tasks like classifying emails into spam or not spam, to more complex tasks like
training autonomous cars to make safe driving decisions.

In simple terms, optimization in ML is like improving a recipe by adjusting the ingredients until
you achieve the best taste. Here, the "ingredients" are the parameters in the model, such as
weights and biases, and "taste" represents the model’s accuracy or performance. The goal of
optimization is to find the best combination of parameters that makes the model learn from the
data as effectively as possible. The way optimization does this is by measuring the model’s error
(or "loss") and adjusting the parameters to reduce this error over time.

One of the most basic optimization techniques is called gradient descent (GD). This technique
works by calculating the gradient (or slope) of the error with respect to each parameter. By
following the gradient in small steps, the model gradually reduces its error, improving its
accuracy with each iteration. While GD is a powerful and widely used method, it has some
limitations. For example, if the learning rate, which controls the size of each step, is set too high,
the model may never reach the best possible performance. Alternatively, if the learning rate is
too low, training will be slow. Also, GD can struggle with complex models where the error
function has many peaks and valleys, which makes it difficult for the model to find the best
solution.

To improve GD, researchers developed adaptive methods like Adam, RMSprop, and AdaGrad.
These methods adjust the learning rate automatically based on past updates, which helps in
speeding up the training process and avoiding issues like overshooting the best solution. Adam
(Adaptive Moment Estimation) is one of the most popular adaptive methods; it combines ideas
from other techniques and adjusts the learning rate more flexibly, making it well-suited for
training deep learning models with complex structures and large datasets.

In addition to gradient-based methods, there are optimization techniques that do not rely on
gradients, such as genetic algorithms and particle swarm optimization. These methods are
inspired by natural processes. For example, genetic algorithms simulate evolution by selecting
and combining "fitter" solutions over many generations to find the best result. Particle swarm
optimization is inspired by the behavior of flocks of birds or schools of fish that move together to
reach a target. These non-gradient methods are useful when the model’s function is complex or
not smooth, making it difficult to calculate a gradient.

Another important category in optimization is second-order methods, which use more detailed
information about the error function’s curvature. Techniques like Newton’s method and the
BFGS algorithm consider not only the slope but also the shape of the error curve. Although these
methods can make the optimization process faster and more accurate, they are computationally
demanding and are typically used in smaller problems.

Optimization techniques are essential in machine learning because they directly impact a model’s
ability to learn effectively. If an ML model is not properly optimized, it may either fail to learn
meaningful patterns (underfitting) or memorize the data without generalizing well to new data
(overfitting). Each type of optimization technique has its own advantages and disadvantages, and
choosing the right one depends on the specific problem, model, and data.

In recent years, optimization methods that combine the strengths of multiple techniques, known
as hybrid methods, have also become popular. These methods offer a more balanced approach,
especially for large and complex problems, where no single method can address all optimization
needs. The field of optimization in ML continues to grow, with researchers constantly looking
for new ways to make models learn faster, be more accurate, and use resources more efficiently.

.
2. LITERATURE REVIEW
Optimization in ML has evolved over time, starting from simple methods like gradient descent
(GD). GD helps models learn by making small adjustments to parameters to minimize the error
between predictions and actual values. This was first used in neural networks in the 1980s and
remains important today. However, standard GD has issues, such as slow learning and getting
stuck in areas where it cannot improve. This led to the development of new methods, such as
Adam, RMSprop, and AdaGrad, which improve GD by adjusting the learning rate based on past
performance. These newer methods make learning faster and more stable, especially in deep
learning models with many parameters.

Second-order methods, like Newton’s method, can provide faster learning by using more
information about the problem, but they are often too computationally demanding for large ML
tasks. For problems where traditional methods struggle, alternative approaches like genetic
algorithms and particle swarm optimization (which do not rely on gradients) have proven useful,
especially in complex or non-smooth problems. Combining different methods, called "hybrid"
optimization, is also becoming popular, as it can give the advantages of multiple techniques.

3. OBJECTIVES

1. To compare different optimization techniques and group them by type (e.g., gradient-
based, second-order, non-gradient).
2. To study how these methods perform in terms of accuracy, speed, and stability in
different ML tasks, such as image classification and sequence prediction.
3. To look at how adaptive and second-order techniques help with specific challenges like
avoiding areas where progress stops.
4. To see how each optimization technique can be applied to large datasets and complex
models like deep neural networks.
5. To understand the pros and cons of each method, especially when used in real-world ML
applications.
4. METHODOLOGY
The study will include both theoretical research and practical experiments. In the theoretical part,
we’ll study how each method works, including their strengths and weaknesses. In the practical
part, we will test these methods on common datasets like MNIST (for handwritten digit
recognition) and CIFAR-10 (for image classification). Python libraries like TensorFlow and
PyTorch will help with these implementations. We will compare each technique based on speed,
accuracy, and how well they minimize the error. We will also examine their performance on
different types of models to see which techniques work best with different tasks.

5. EXPECTED OUTCOMES
The study should provide clear insights into the pros and cons of various optimization methods
in ML. It will highlight which techniques work best for specific challenges, such as preventing
overfitting, improving speed, or handling large datasets. The findings may also point to the
benefits of hybrid methods for certain types of ML problems.

6. TENTATIVE TIMEFRAME

1. Literature Review – 2 months

2. Algorithm Implementation – 3 months
3. Experimental Analysis – 6 months
4. Result Interpretation & Thesis Write-Up – 4 months

7. REFERENCES

1. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by

back-propagating errors. Nature, 323(6088), 533-536.
2. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv
preprint arXiv:1412.6980.
3. Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online
learning and stochastic optimization. Journal of Machine Learning Research, 12, 2121-
2159.
4. Boyd, S., & Vandenberghe, L. (2004). Convex Optimization. Cambridge University
Press.
5. Nocedal, J., & Wright, S. J. (2006). Numerical Optimization. Springer Science &
Business Media.
6. Hinton, G. (2012). Lecture 6e rmsprop: Divide the gradient by a running average of its
recent magnitude. Coursera Lecture Slides on Neural Networks for Machine Learning