Model Tuning

An Optimization Problem

Model tuning is the systematic modification of model parameters to identify the most performant model. Model tuning presents itself as an optimization problem. Depending on the model and target variable, various performance indicators can be selected as optimization targets, such as:

• Continuous target variable
  • Root Mean Squared Error (RMSE)
  • R-Squared
  • Loss function
  • Mean Absolute Percentage Error (MAPE)
• Discrete target variable
  • Accuracy
  • Sensitivity
  • Specificity
  • Precision
  • F1-Score
  • Kappa
  • Area under Curve (AUC)

To counteract the trade-off between model performance and overfitting, cross-validation is recommended when tuning models. The aim is to fine-tune the model parameters in order to optimize the selected performance metric. The hyperparameters differ from method to method. Some are listed below as examples (not exhaustive), depending on the method:

• Regression
  • Included variables
  • Interaction effects
  • Transformed variables
  • L1-Regularization
  • L2-Regularization
• Decision tree
  • Depth of tree
  • Split criteria
  • Minimal number of observations in end nodes
• Random Forest
  • Number of trees
  • Number of features considered per split
  • Depth of tree
  • Split criteria
  • Sample size of bootstrapping
• Gradient Boosting
  • Number of trees
  • Learning rate
  • Sample size of bootstrapping
  • Number of end nodes
• Neural Nets
  • Learning rate
  • Number of hidden layers
  • Activitation function
  • Batch size

Often a so-called grid search is used for hyperparameter tuning. The hyperparameter values are selected by ratio, empirical evidence or by trial and error. If different values are taken for the different hyperparameters, this results in a large number of different model specifications in combination. This matrix of combinatorial specifications of hyperparameters is called Grid or Hypergrid. The goal is to find out from these many specifications which one performs best according to a performance metric. An exemplary grid can be found below for a random forest.

Exemplary Grid

• Number of trees: 500, 1000
• Features considered per split: Square-root from features, +1, -1
• Depth of tree: 4,5
• Split criteria: Gini, Information Gain

This results in 2x3x2x2 = 24 model specifications to be tested. It becomes obvious that by multiplying the specifications very large grids can be created quickly. Since each row of a grid represents a model estimation, long and intensive computations can occur depending on the complexity of the model, the number of features and the number of observations. Therefore there is a obvious trade-off between the size of the grid and the computational costs. Our above fictitious grid would now be applied with cross-validation. The model performance is measured and then the average performance is compared. The specification with the best model performance determines the parameters in the final model. This could exemplary look like the following:

• Number of trees: 1000
• Features considered per split: Square-root from features +1
• Depth of tree: 4
• Split criteria: Information Gain

Looking for a hands-on tutorial in Python? Then check out TowardsDataScience.