Scribd
Upload
144 views

Bagging and Boosting

Bagging and boosting are ensemble techniques that combine multiple decision tree classifiers to improve performance. Bagging generates additional training data sets by sampling the original data with replacement and trains an individual model on each sample. Boosting runs a weak learner on reweighted training data multiple times and combines the results through voting. Examples of boosting algorithms include AdaBoost, which uses decision stumps, and gradient boosting, which uses gradient descent.

Uploaded by

Ana-Cosmina Popescu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
144 views

Bagging and Boosting

Bagging and boosting are ensemble techniques that combine multiple decision tree classifiers to improve performance. Bagging generates additional training data sets by sampling the original data with replacement and trains an individual model on each sample. Boosting runs a weak learner on reweighted training data multiple times and combines the results through voting. Examples of boosting algorithms include AdaBoost, which uses decision stumps, and gradient boosting, which uses gradient descent.

Uploaded by

Ana-Cosmina Popescu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

Bagging and Boosting

Popescu Ana-Cosmina, AI, 2019


Bagging and Boosting

They both are ensemble techniques = combine several decision tree classifiers
for a better performance.
Bagging and Boosting

They both are ensemble techniques = combine several decision tree classifiers
for a better performance.

Ensemble techniques:
- reduce variance and bias;
- increase the robustness;
- especially useful for unstable classifiers.
Bagging

Stands for “Bootstrap Aggregation”.

… What is Bootstrapping?
Bootstrapping
● Random sampling with replacement;
● Understand the bias and variance of the dataset.
Bagging

Stands for “Bootstrap Aggregation”;

● The application of the Bootstrap


procedure to a high-variance ML
algorithm (typically decision
trees);
● Each model runs independently;
● Aggregation = average/voting.
Bagging Decision Trees

1. Sample, with replacement,


n training examples from
the dataset.
2. Train a decision tree on
the n samples.
3. Repeat t times, for some t.
Extension: Random Forests
= bagging decision trees + “feature bagging”
Every time a new decision node is made, only a subset of p features is used.
Boosting

Given a weak learner, run it multiple times on (reweighted) training data, then
let the learned classifiers vote.
Boosting - visual example
Extension: Gradient Descent
= gradient descent + boosting

GD = a first-order iterative
optimization algorithm for finding
the minimum of a function.
Extension: Gradient Boosting
In Practice: AdaBoost
Stands for “Adaptive Boosting”;

● AdaBoost trees are usually just a node + 2 leaves = a stump.


● Stumps are “weak learners” (1 variable/decision)
AdaBoost (2)

Forest of Trees: Forest of Stumps:


AdaBoost (3)

Decision rule: H(x) = sign(∑wkhk(x)), k ← 1,K.


To sum up...
Thank you for your attention!
Bibliography

1. https://becominghuman.ai/ensemble-learning-bagging-and-boosting-d20f38be9b1e
2. https://towardsdatascience.com/decision-tree-ensembles-bagging-and-boosting-266a8ba60fd9
3. https://victorzhou.com/blog/intro-to-random-forests/
4. http://bccvl.org.au/algorithms-exposed-random-forest/
5. https://campus.datacamp.com/courses/machine-learning-with-tree-based-models-in-python/boost
ing?ex=5
6. http://datahacker.rs/gradient-descent-python/
7. https://www.datacamp.com/community/tutorials/adaboost-classifier-python
8. https://www.educba.com/adaboost-algorithm/
9. https://sefiks.com/2018/10/04/a-step-by-step-gradient-boosting-decision-tree-example/

You might also like