Random Forest belongs to the ensemble methods and is based on decision trees. Many small decision trees are averaged to build a powerful overall model.
Bagging
The concept of Random Forest is based on the method of bagging. Bagging belongs to the so-called ensemble methods. Ensemble methods have in common that they combine many weak learners to create a strong, predictive model (strong learners). Bagging stands for bootstrap aggregating. In order to counteract the high variance of a single decision tree and to increase the accuracy of the prediction, many training data would ideally be drawn from a population. A separate model is estimated for each training data set. The average of all models is then calculated to obtain an overall model.
Bootstrapping
Since such a procedure cannot be implemented from a practical point of view (due to lack of mulitple training data sets), bootstrapping is used to simulate different training data sets. In bootstrapping, a new data set with the same number of observations is generated from the original data set by drawing with replacement. This has the consequence that observations from the original data set can occur several times in a bootstrap sample.
In the context of classification decision trees, B bootstrap samples (= training data sets) are drawn by bootstrapping. A decision tree is estimated for each bootstrap sample. To receive an overall model, models are averaged by a majority vote. A new unseen observation is classified by all B decision trees, using as final decision the class that was estimated by majority vote.
By using bagging, all estimated decision trees tend to have a high correlation. On average two thirds of all observations are used per bootstrap sample. This in turn means that a feature that has a strong influence will appear in the majority of the decision trees (possibly even in the first split). Thus, the decision trees correlate strongly during bagging. The approach of Random Forest addresses this weakness. The goal is to obtain B decision trees that are as uncorrelated as possible. Similar to bagging, B bootstrap training samples are generated.
However, for each split in each tree only a random subset of m features from all p features is considered. The split only considers the features from the random subset. If an extremely strong feature is now present, it will on average only be considered in (p-m)/p cases (James et al. 2013). If m is set equal to p, then bagging and random forest do not differ. The B estimated trees are therefore now very different from each other, so that the correlation between them decreases.
import pandas as pd
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score, recall_score
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
randomforest = RandomForestClassifier(n_estimators = 1000, random_state = 42)
cv_scores = cross_val_score(randomforest, X_train, y_train, cv = 3, scoring = ‘recall’)
print(cv_scores)
print(“Average 3-Fold CV recall score: {}”.format(np.mean(cv_scores)))
randomforest.fit(X_train, y_train)
y_pred = randomforest.predict(X_test)
y_pred_proba = randomforest.predict_proba(X_test)[:,1]
The code snippet is written in the programming language Python and is based on the module scikit-learn.
For a hands-on tutorial on how to build a random forest model, visit TowardsDataScience.
