Cross-validation (statistics)

Cross-validation (statistics) is a statistical resampling technique used to evaluate the performance and generalization ability of a predictive model by repeatedly dividing available data into training and validation subsets. It is widely used in Machine learning for model evaluation, model selection, and hyperparameter tuning.

In cross-validation, different portions of the dataset are used for training and validation so that model performance can be assessed on observations that were not used to fit the model in each iteration. This can provide a more reliable estimate of out-of-sample performance than evaluating a model on a single fixed validation split.

Purpose

The principal purposes of cross-validation are to:

estimate model performance on unseen data;
assess model generalization;
compare competing models;
select model hyperparameters;
identify potential Overfitting;
make efficient use of limited datasets.

Basic procedure

A typical cross-validation procedure involves:

Dividing the available dataset into multiple subsets.
Training the model using a portion of the subsets.
Evaluating the model using the remaining subset.
Repeating the process with different training and validation subsets.
Aggregating the resulting performance measurements.

The resulting performance estimate is generally based on the average performance across the validation iterations.

K-fold cross-validation

K-fold cross-validation is one of the most commonly used forms of cross-validation.

The dataset is divided into k approximately equal-sized folds. The model is trained k times. During each iteration, one fold is used for validation while the remaining k − 1 folds are used for training.

The procedure can be represented as:

$Training folds = k - 1$

and:

$Validation folds = 1$

Each observation is therefore used for validation once and for training approximately k − 1 times.

If the performance measure for fold i is M_i, the mean cross-validation performance can be calculated as:

${\bar{M}}_{C V} = \frac{1}{k} \sum_{i = 1}^{k} M_{i}$

The variation among the fold-specific results can also be examined to assess the stability of the estimate.

Choice of k

Common choices include 5-fold and 10-fold cross-validation.

A smaller value of k generally requires fewer model-training operations but may produce a less stable estimate of performance. A larger value uses more observations for training in each iteration but increases computational cost.

The appropriate choice depends on dataset size, computational resources, and the objectives of the analysis.

Stratified cross-validation

Stratified cross-validation is commonly used for classification problems where the distribution of target classes should remain approximately consistent across folds.

Each fold is constructed so that its class proportions are similar to those of the overall dataset.

This can be particularly useful when the classes are imbalanced.

Leave-one-out cross-validation

Leave-one-out cross-validation (LOOCV) is a special case of k-fold cross-validation in which:

$k = n$

where n is the number of observations.

The model is trained using n − 1 observations and validated using the remaining observation. This process is repeated until every observation has been used as the validation observation.

LOOCV can make efficient use of small datasets but may require substantially more model-fitting operations than k-fold cross-validation.

Repeated cross-validation

Repeated cross-validation performs the cross-validation procedure multiple times using different partitions of the dataset.

Repeating the process can provide a more stable estimate of model performance, particularly when the results are sensitive to the particular partitioning of the data.

Time-series cross-validation

Standard random cross-validation is generally inappropriate for many time-series problems because observations may have a temporal order.

Time-series cross-validation preserves the chronological structure of the data.

For example, a model may be trained on earlier observations and evaluated on subsequent observations, with the training period expanding or moving forward over successive iterations.

This approach helps avoid using future observations to predict the past.

Cross-validation and overfitting

Cross-validation can help identify and control Overfitting by evaluating a model on observations that were not used to fit the model during each validation iteration.

A model that performs extremely well on its training data but substantially worse during cross-validation may have poor generalization ability.

Cross-validation does not itself guarantee that overfitting has been eliminated.

Cross-validation and model selection

Cross-validation can be used to compare different models or model configurations.

For example, several models can be evaluated using the same folds, and their average validation performance can be compared.

The model or configuration with the preferred cross-validation performance may then be selected for final training.

Cross-validation and hyperparameter tuning

Cross-validation is commonly used for hyperparameter tuning.

Parameters such as regularization strength, tree depth, learning rate, or model complexity can be evaluated using cross-validation.

The selected hyperparameters are then typically used to train a final model using the appropriate training data.

Nested cross-validation

Nested cross-validation uses two levels of cross-validation.

The inner loop is used for model or hyperparameter selection, while the outer loop is used to estimate the performance of the resulting selection procedure.

This approach can reduce optimistic bias when model selection and performance estimation are performed on the same data.

Data leakage

Cross-validation can produce misleading results if information from the validation data is inadvertently used during model training or preprocessing.

Examples include performing feature scaling, feature selection, or imputation using the complete dataset before cross-validation.

To avoid such data leakage, data-dependent preprocessing should generally be performed separately within each training fold.

Cross-validation and the test set

A final test dataset may be kept separate from the cross-validation process.

Cross-validation can be used for model development, comparison, and hyperparameter selection, while the held-out test set can provide an independent final estimate of performance.

Using the test set repeatedly during model development can reduce its usefulness as an unbiased evaluation set.

Advantages

Cross-validation can:

make efficient use of limited data;
provide multiple estimates of model performance;
reduce dependence on a single train-validation split;
assist with model selection;
assist with hyperparameter tuning;
provide information about performance variability.

Limitations

Cross-validation can be computationally expensive because the model must be trained multiple times.

Its estimates can also be affected by the way folds are constructed and may be unreliable when the validation procedure does not reflect the conditions under which the model will ultimately be deployed.

For time-dependent, grouped, hierarchical, or otherwise dependent observations, standard random k-fold cross-validation may produce misleading estimates.

Applications

Cross-validation is widely used in:

Machine learning;
statistical modelling;
regression;
classification;
predictive analytics;
Model selection;
hyperparameter optimization;
Deep learning;
scientific modelling.

Importance

Cross-validation is an important method for estimating how predictive models are likely to perform on unseen data. By repeatedly separating model fitting from validation, it provides a systematic framework for assessing generalization, comparing models, and selecting model configurations while making efficient use of available data.