Scribd
Upload
5 views

Lec8 (1)

The document discusses the concepts of bias and variance in machine learning, highlighting their impact on model accuracy and prediction errors. It explains the types of errors, their causes, and strategies to mitigate high bias and high variance, emphasizing the importance of the Bias-Variance trade-off in model development. The goal is to achieve a balance that allows the model to generalize well to unseen data while capturing the essential patterns in the training data.

Uploaded by

ku777965
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
5 views

Lec8 (1)

The document discusses the concepts of bias and variance in machine learning, highlighting their impact on model accuracy and prediction errors. It explains the types of errors, their causes, and strategies to mitigate high bias and high variance, emphasizing the importance of the Bias-Variance trade-off in model development. The goal is to achieve a balance that allows the model to generalize well to unseen data while capturing the essential patterns in the training data.

Uploaded by

ku777965
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

Dr.

Supriyo Mandal,
Ph.D. (IIT Patna)
Postdoc (ZBW, University of Kiel,
Germany)

Course code: CS31002 (L-T-P-Cr: 3-1-0-4) Course Name: Machine Learning


Credits: 4
source: https://www.youtube.com/watch?v=EuBBz3bI-aA&ab_channel=StatQuestwithJoshStarmer
vIn machine learning, an error is a measure of how accurately an algorithm can make predictions for the
previously unknown dataset.
vOn the basis of these errors, the machine learning model is selected that can perform best on the particular dataset.
There are mainly two types of errors in machine learning, which are:
vReducible errors: These errors can be reduced to improve the model accuracy. Such errors can further be
classified into bias and Variance.
vIrreducible errors: These errors will always be present in the model.
vWhile making predictions, a difference occurs between prediction values made by the model and actual
values, and this difference is known as bias errors or Errors due to bias.
vIt can be defined as an inability of machine learning algorithms such as Linear Regression to capture the
true relationship between the data points.
vEach algorithm begins with some amount of bias because bias occurs from assumptions in the model, which
makes the target function simple to learn. A model has either:
vLow Bias: A low bias model will make fewer assumptions about the form of the target function.
vHigh Bias: A model with a high bias makes more assumptions, and the model becomes unable to capture
the important features of our dataset. A high bias model also cannot perform well on new data.
vGenerally, a linear algorithm has a high bias, as it makes them learn fast. The simpler the algorithm, the
higher the bias it has likely to be introduced. Whereas a nonlinear algorithm often has low bias.
vSome examples of machine learning algorithms with low bias are Decision Trees, k-Nearest Neighbours and
Support Vector Machines.
vAt the same time, an algorithm with high bias is Linear Regression, Linear Discriminant Analysis and
Logistic Regression.
vHigh bias mainly occurs due to a much simple model.

vBelow are some ways to reduce the high bias:


vIncrease the input features as the model is underfitted.
vDecrease the regularization term.
vUse more complex models, such as including some polynomial features
vThe variance specify the amount of variation in the prediction if the different training data
was used.
vIdeally, a model should not vary too much from one training dataset to another, which means the
algorithm should be good in understanding the hidden mapping between inputs and output variables.
vVariance errors are either of low variance or high variance.
vLow variance means there is a small variation in the prediction of the target function with changes
in the training data set.
vAt the same time, High variance shows a large variation in the prediction of the target function with
changes in the training dataset.
vA model that shows high variance learns a lot and perform well with the training dataset, and does not
generalize well with the unseen dataset.
vAs a result, such a model gives good results with the training dataset but shows high error rates on the test
dataset.
vWith high variance, the model learns too much from the dataset, it leads to overfitting of the model.
vA model with high variance has the below problems:
• A high variance model leads to overfitting.
• Increase model complexities.
• Usually, nonlinear algorithms have a lot of flexibility to fit the model, have high variance.

vSome examples of machine learning algorithms with low variance are, Linear Regression, Logistic
Regression, and Linear discriminant analysis.
vAt the same time, algorithms with high variance are decision tree, Support Vector Machine, and K-
nearest neighbours.
• Reduce the input features or number of parameters as a model is overfitted.
• Do not use a much complex model.
• Increase the training data.
• Increase the Regularization term.
vThere are four possible combinations of bias and variances:-
• Low-Bias, Low-Variance: The combination of low bias and low variance shows an ideal machine learning
model. However, it is not possible practically.
• Low-Bias, High-Variance: With low bias and high variance, model predictions are inconsistent and
accurate on average. This case occurs when the model learns with a large number of parameters and
hence leads to an overfitting
• High-Bias, Low-Variance: With High bias and low variance, predictions are consistent but inaccurate on
average. This case occurs when a model does not learn well with the training dataset or uses few numbers
of the parameter. It leads to underfitting problems in the model.
• High-Bias, High-Variance: With high bias and high variance, predictions are inconsistent and also
inaccurate on average.
vHigh variance can be identified if the model has:
vLow training error and high test error.

vHigh Bias can be identified if the model has:


vHigh training error and the test error is almost similar to training error.
vWhile building the machine learning model, it is really important to take care of bias and variance in order to
avoid overfitting and underfitting in the model.
vIf the model is very simple with fewer parameters, it may have low variance and high bias.
vWhereas, if the model has a large number of parameters, it will have high variance and low bias.
vSo, it is required to make a balance between bias and variance errors, and this balance between the bias
error and variance error is known as the Bias-Variance trade-off.
vFor an accurate prediction of the model, algorithms need a low variance and low bias. But this is not
possible because bias and variance are related to each other:
• If we decrease the variance, it will increase the bias.
• If we decrease the bias, it will increase the variance.

vBias-Variance trade-off is a central issue in supervised learning.


vIdeally, we need a model that accurately captures the regularities in training data and simultaneously
generalizes well with the unseen dataset.
vUnfortunately, doing this is not possible simultaneously. Because a high variance algorithm may perform well
with training data, but it may lead to overfitting to noisy data.
vWhereas, high bias algorithm generates a much simple model that may not even capture important regularities
in the data. So, we need to find a sweet spot between bias and variance to make an optimal model.
vHence, the Bias-Variance trade-off is about finding the sweet spot to make a balance between bias and
variance errors.

You might also like