Cross Validation

Cross-Validation is a procedure for the validation of models from data mining and machine learning.

The input data is prepared in Data Preparation according to current machine learning standards (pre-processing). Normally, the data is now split into training and test data set. Then the model is trained on the training data and evaluated on the test data. However, often an unfavorable split in training and test data can seriously lead to a miscalculated model. For example, the distribution of the target variable (target class) in both training and test data may be unequal, or the distribution of certain features may differ greatly. However, this is usually not a problem if the data sets are large enough. But when is the data set large enough?

In k-fold cross validation, the data is split into training and test data in k iterations. Accordingly, we obtain k model evaluations. In a 5-fold-Cross-Validation (see picture) the data is split 5 times. In a 5-fold validation, the data is split k = 5 times into 1/k = 1/5 = 20% training data and 80% test data. Likewise, with k = 10 in 1/k = 1/10 = 10% training data set and 90% test data set etc. The split is done systematically by index and not randomly. In the first iteration, the first 1/k % of the data is used as training data, in the second iteration the next 1/k % of the data is used, and so on (see picture).

We have a record of 1000 rows. For model evaluation we use accuracy (True Positive + True Negative / number of total rows). In the following this is fictitious and serves as an illustration. We apply a 5-fold cross validation.

1. Iteration:
   • Test data: rows 1-200
   • Training data: rows 201-1000
   • True Negative: 745
   • True Positive: 85
   • Accuracy: (745+85)/1000 = 83
2. Iteration:
   • Test data: rows 201-400
   • Training data: rows 1-200 & 401-1000
   • True Negative: 764
   • True Positive: 76
   • Accuracy: (764+76)/1000 = 84
3. Iteration:
   • Test data: rows 401-600
   • Training data: rows 1-400 & 601-1000
   • True Negative: 789
   • True Positive: 21
   • Accuracy: (745+85)/1000 = 80%.
4. Iteration:
   • Test data: rows 601-800
   • Training data: rows 1-600 & 801-1000
   • True Negative: 755
   • True Positive: 85
   • Accuracy: (745+85)/1000 = 84
5. Iteration:
   • Test data: rows 801-1000
   • Training data: rows 1-800
   • True Negative: 758
   • True Positive: 52

We obtain an average accuracy of (83%+84%+80%+84%+81%)/5 = 82.4%.

Thus, if we now got an accuracy of 89% in a random split, this indicates overfitting and a randomly unfavorable split of training and test data set.

Advantages of k-fold cross-validation at a glance

• Identification of overfitting
• Testing the robustness of the model
• Significant model performance on small data sets
• Significant model performance on data sets with balance problems
• Use for tuning of models

The final model parameters are determined in a usual split into training and test data set.

Looking for a hands-on tutorial in Python on how to use cross-validation? Then check out TowardsDataScience!