Scribd
Upload
36 views

Lecture 16 - Hyperparameter Tuning

Machine Learning

Uploaded by

ilham.hasib
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
36 views

Lecture 16 - Hyperparameter Tuning

Machine Learning

Uploaded by

ilham.hasib
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 10

16 Hyperparameter

Tuning
- Dr. Sifat Momen (SfM1)
Learning goals
• After this presentation, you should be able to
• Appreciate the need for hyperparameter optimization
• Understand the differences between manual and automated optimization
• Understand the difference between random search and grid search
techniques
• Apply hyperparameter optimization techniques

08/06/2024 Slides by Dr. Sifat Momen 2


Jupyter Notebook
• Please note that there is an associated Jupyter notebook with this
presentation
• Please use both in parallel for optimal understanding

08/06/2024 Slides by Dr. Sifat Momen 3


Hyperparameters
• Choosing the correct hyperparameters for machine learning or deep
learning models is one of the best ways to improve model
performance
• Model parameters: These are the parameters that are estimated by
the model from the given data. For example the coefficients in linear
regression model.
• Model hyperparameters: These are the parameters that cannot be
estimated by the model from the given data. These parameters are
used to estimate the model parameters. For example, the learning
rate in deep neural networks.

08/06/2024 Slides by Dr. Sifat Momen 4


Hyperparameters
• What are some of the hyperparameters in
• KNN
• Decision Tree
• Random Forest

08/06/2024 Slides by Dr. Sifat Momen 5


Hyperparameter Tuning
• Hyperparameter tuning (or hyperparameter optimization) is the
process of determining the right combination of hyperparameters
that maximizes the model performance. It works by running multiple
trials in a single training process. Each trial is a complete execution of
your training application with values for your chosen
hyperparameters, set within the limits you specify. This process once
finished will give you the set of hyperparameter values that are best
suited for the model to give optimal results.

08/06/2024 Slides by Dr. Sifat Momen 6


How to do hyperparameter tuning?
• Manual Hyperparameter tuning
• Automated Hyperparameter tuning

08/06/2024 Slides by Dr. Sifat Momen 7


Manual Hyperparameter Tuning
• Manual hyperparameter tuning involves experimenting with different sets of
hyperparameters manually i.e. each trial with a set of hyperparameters will be
performed by you. This technique will require a robust experiment tracker
(such as MLflow).
• Advantages of manual hyperparameter optimization:
• Tuning hyperparameters manually means more control over the process.
• If you are researching or studying tuning and how it affects the network weights then
doing it manually would make sense.
• Disadvantages of manual hyperparameter optimization:
• Manual tuning is a tedious process since there can be many trials and keeping track can
prove costly and time-consuming.
• This isn’t a very practical approach when there are a lot of hyperparameters to consider.

08/06/2024 Slides by Dr. Sifat Momen 8


Automated Hyperparameter Tuning
• Automated hyperparameter tuning utilizes already existing algorithms
to automate the process. The steps you follow are:
• First, specify a set of hyperparameters and limits to those hyperparameters’
values (note: every algorithm requires this set to be a specific data structure,
e.g. dictionaries are common while working with algorithms).
• Then the algorithm does the heavy lifting for you. It runs those trials and
fetches you the best set of hyperparameters that will give optimal results.

08/06/2024 Slides by Dr. Sifat Momen 9


Popular Automated Hyperparameter
Optimization Methods
• Random Search
• In the random search method, we create a grid of possible values for
hyperparameters. Each iteration tries a random combination of
hyperparameters from this grid, records the performance, and lastly returns
the combination of hyperparameters that provided the best performance.
• Grid Search
• In the grid search method, we create a grid of possible values for
hyperparameters. Each iteration tries a combination of hyperparameters in a
specific order. It fits the model on each and every combination of
hyperparameters possible and records the model performance. Finally, it
returns the best model with the best hyperparameters.

08/06/2024 Slides by Dr. Sifat Momen 10

You might also like