Random Forest Classifier in Machine Learning | Palin Analytics
Download as PPTX, PDF3 likes10,454 views
Random forest is a supervised learning ensemble algorithm that combines multiple decision trees to improve classification accuracy through the 'bagging' method. It references random subsets of both training data and features to enhance diversity and robustness in model predictions. This algorithm is efficient, handles large datasets with numerous variables, and provides insights into feature importance, making it a strong choice for quick model development.
Random Forest Classifier in Machine Learning | Palin Analytics
- 1. Random Forest Classifier Classification Technique
- 2. Overview Random Forest is a supervised learning ensemble algorithm. Ensemble algorithms are those which combine more than one algorithms of same or different kind for classifying objects. The ‘forest’ that Random Forest Classifier builds, is an ensemble of Decision Trees, most of the time trained with the ‘bagging’ method. The general idea of the bagging method is that a combination of learning models increases the overall result. Random forest classifier creates a set of decision trees from randomly selected subset of training set. It then aggregates the votes from different decision trees to decide the final class of the test object. Random Forest adds additional randomness to the model, while growing the trees. Instead of searching for the most important feature while splitting a node, it searches for the best feature among a random subset of features. This results in a wide diversity that generally results in a better model.
- 3. Explanation Say, we have 1000 observations in the complete population with 10 variables. Random forest tries to build multiple CART model with different sample and different initial variables. For instance, it will take a random sample of 100 observation and 5 randomly chosen initial variables to build a CART model. It will repeat the process (say) 10 times and then make a final prediction on each observation. Final prediction is a function of each prediction. This final prediction can simply be the mean of each prediction.
- 4. Each tree in a forest is grown as follows: • If the number of cases in the training set is N, sample n cases at random (but with replacement) from the original data. This sample will be the training set for growing the tree. • If there are M input variables, a number m < M is specified such that at each node, m variables are selected at random out of the M and the best split on these m is used to split the node. The value of m is held constant during the forest growing. • Each tree is grown to the largest extent possible. There is no pruning.
- 5. Forest Error rate depends on two things: • The correlation between any two trees in the forest. Increasing the correlation increases the forest error rate. • The strength of each individual tree in the forest. A tree with a low error rate is a strong classifier. Increasing the strength of the individual trees decreases the forest error rate. Reducing m reduces both the correlation and the strength. Increasing it increases both. Somewhere in between is an "optimal" range of m (usually quite wide). Using the OOB error rate (explained in later slides) an optimal value of m can quickly be found. This is the only adjustable parameter to which random forests is somewhat sensitive.
- 6. Features • It is unexcelled in accuracy among current algorithms. • It runs efficiently on large data bases. • It can handle thousands of input variables without variable deletion. • It gives estimates of what variables are important in the classification. • It generates an internal unbiased estimate of the generalization error as the forest building progresses. • It has an effective method for estimating missing data. It maintains accuracy even when a large proportion of the data are missing. • It has methods for balancing error in class population unbalanced data sets. • Generated forests can be saved for future use on other data. • Prototypes are computed that give information about the relation between the variables and the classification. • The capabilities of the above can be extended to unlabeled data, leading to unsupervised clustering, data views and outlier detection. • It offers an experimental method for detecting variable interactions.
- 7. Out-Of-Bag (OOB) When the training set for the current tree is drawn by sampling with replacement, about one-third of the observations are left out of the sample. This OOB (out-of-bag) data is used to get a running unbiased estimate of the classification error as trees are added to the forest. It is also used to get estimates of variable importance. Each tree is constructed using a different bootstrap sample from the original data. About one-third of the cases are left out of the bootstrap sample and not used in the construction of the kth tree.
- 8. Out-Of-Bag (OOB) Error Estimate Each tree is constructed using a different bootstrap sample from the original data. About one-third of the cases are left out of the bootstrap sample and not used in the construction of the kth tree. Put each case left out in the construction of the kth tree down the kth tree to get a classification. In this way, a test set classification is obtained for each case in about one-third of the trees. At the end of the run, take j to be the class that got most of the votes every time case n was OOB. The proportion of times that j is not equal to the true class of n averaged over all cases is the OOB error estimate. This has proven to be unbiased in many tests.
- 9. Overfitting Random Forest does not overfit. You can run as many trees as you want. It is fast.
- 10. Summary Random Forest is a great algorithm to train early in the model development process, to see how it performs and it’s hard to build a “bad” Random Forest, because of its simplicity. This algorithm is also a great choice, if you need to develop a model in a short period of time. On top of that, it provides a pretty good indicator of the importance it assigns to your features. Random Forests are also very hard to beat in terms of performance. Of course you can probably always find a model that can perform better, like a neural network, but these usually take much more time in the development. And on top of that, they can handle a lot of different feature types, like binary, categorical and numerical.