View source for Cross-validation (statistics)

'''[[Cross-validation (statistics)]]''' is a statistical resampling technique used to evaluate the performance and generalization ability of a predictive model by repeatedly dividing available data into training and validation subsets. It is widely used in [[Machine learning]] for model evaluation, model selection, and [[Hyperparameter optimization|hyperparameter tuning]].

In cross-validation, different portions of the dataset are used for training and validation so that model performance can be assessed on observations that were not used to fit the model in each iteration. This can provide a more reliable estimate of out-of-sample performance than evaluating a model on a single fixed validation split.

== Purpose ==
The principal purposes of cross-validation are to:

* estimate model performance on unseen data;
* assess model generalization;
* compare competing models;
* select model hyperparameters;
* identify potential [[Overfitting]];
* make efficient use of limited datasets.

== Basic procedure ==
A typical cross-validation procedure involves:

# Dividing the available dataset into multiple subsets.
# Training the model using a portion of the subsets.
# Evaluating the model using the remaining subset.
# Repeating the process with different training and validation subsets.
# Aggregating the resulting performance measurements.

The resulting performance estimate is generally based on the average performance across the validation iterations.

== K-fold cross-validation ==
'''K-fold cross-validation''' is one of the most commonly used forms of cross-validation.

The dataset is divided into '''k''' approximately equal-sized folds. The model is trained '''k''' times. During each iteration, one fold is used for validation while the remaining '''k − 1''' folds are used for training.

The procedure can be represented as:

<math>
\text{Training folds}=k-1
</math>

and:

<math>
\text{Validation folds}=1
</math>

Each observation is therefore used for validation once and for training approximately '''k − 1''' times.

If the performance measure for fold '''i''' is '''M<sub>i</sub>''', the mean cross-validation performance can be calculated as:

<math>
\bar{M}_{CV}
=
\frac{1}{k}
\sum_{i=1}^{k}M_i
</math>

The variation among the fold-specific results can also be examined to assess the stability of the estimate.

== Choice of k ==
Common choices include 5-fold and 10-fold cross-validation.

A smaller value of '''k''' generally requires fewer model-training operations but may produce a less stable estimate of performance. A larger value uses more observations for training in each iteration but increases computational cost.

The appropriate choice depends on dataset size, computational resources, and the objectives of the analysis.

== Stratified cross-validation ==
'''Stratified cross-validation''' is commonly used for classification problems where the distribution of target classes should remain approximately consistent across folds.

Each fold is constructed so that its class proportions are similar to those of the overall dataset.

This can be particularly useful when the classes are imbalanced.

== Leave-one-out cross-validation ==
'''Leave-one-out cross-validation''' (LOOCV) is a special case of k-fold cross-validation in which:

<math>
k=n
</math>

where '''n''' is the number of observations.

The model is trained using '''n − 1''' observations and validated using the remaining observation. This process is repeated until every observation has been used as the validation observation.

LOOCV can make efficient use of small datasets but may require substantially more model-fitting operations than k-fold cross-validation.

== Repeated cross-validation ==
'''Repeated cross-validation''' performs the cross-validation procedure multiple times using different partitions of the dataset.

Repeating the process can provide a more stable estimate of model performance, particularly when the results are sensitive to the particular partitioning of the data.

== Time-series cross-validation ==
Standard random cross-validation is generally inappropriate for many [[Time series|time-series]] problems because observations may have a temporal order.

'''Time-series cross-validation''' preserves the chronological structure of the data.

For example, a model may be trained on earlier observations and evaluated on subsequent observations, with the training period expanding or moving forward over successive iterations.

This approach helps avoid using future observations to predict the past.

== Cross-validation and overfitting ==
Cross-validation can help identify and control [[Overfitting]] by evaluating a model on observations that were not used to fit the model during each validation iteration.

A model that performs extremely well on its training data but substantially worse during cross-validation may have poor generalization ability.

Cross-validation does not itself guarantee that overfitting has been eliminated.

== Cross-validation and model selection ==
Cross-validation can be used to compare different models or model configurations.

For example, several models can be evaluated using the same folds, and their average validation performance can be compared.

The model or configuration with the preferred cross-validation performance may then be selected for final training.

== Cross-validation and hyperparameter tuning ==
Cross-validation is commonly used for [[Hyperparameter optimization|hyperparameter tuning]].

Parameters such as regularization strength, tree depth, learning rate, or model complexity can be evaluated using cross-validation.

The selected hyperparameters are then typically used to train a final model using the appropriate training data.

== Nested cross-validation ==
'''Nested cross-validation''' uses two levels of cross-validation.

The inner loop is used for model or hyperparameter selection, while the outer loop is used to estimate the performance of the resulting selection procedure.

This approach can reduce optimistic bias when model selection and performance estimation are performed on the same data.

== Data leakage ==
Cross-validation can produce misleading results if information from the validation data is inadvertently used during model training or preprocessing.

Examples include performing feature scaling, feature selection, or imputation using the complete dataset before cross-validation.

To avoid such [[data leakage]], data-dependent preprocessing should generally be performed separately within each training fold.

== Cross-validation and the test set ==
A final test dataset may be kept separate from the cross-validation process.

Cross-validation can be used for model development, comparison, and hyperparameter selection, while the held-out test set can provide an independent final estimate of performance.

Using the test set repeatedly during model development can reduce its usefulness as an unbiased evaluation set.

== Advantages ==
Cross-validation can:

* make efficient use of limited data;
* provide multiple estimates of model performance;
* reduce dependence on a single train-validation split;
* assist with model selection;
* assist with hyperparameter tuning;
* provide information about performance variability.

== Limitations ==
Cross-validation can be computationally expensive because the model must be trained multiple times.

Its estimates can also be affected by the way folds are constructed and may be unreliable when the validation procedure does not reflect the conditions under which the model will ultimately be deployed.

For time-dependent, grouped, hierarchical, or otherwise dependent observations, standard random k-fold cross-validation may produce misleading estimates.

== Applications ==
Cross-validation is widely used in:

* [[Machine learning]];
* statistical modelling;
* regression;
* classification;
* predictive analytics;
* [[Model selection]];
* hyperparameter optimization;
* [[Deep learning]];
* scientific modelling.

== Importance ==
Cross-validation is an important method for estimating how predictive models are likely to perform on unseen data. By repeatedly separating model fitting from validation, it provides a systematic framework for assessing generalization, comparing models, and selecting model configurations while making efficient use of available data.

== See also ==

* [[Machine learning]]
* [[Overfitting]]
* [[Underfitting]]
* [[Model selection]]
* [[Hyperparameter optimization]]
* [[Training, validation, and test sets]]
* [[Data leakage]]
* [[Statistical learning theory]]
* [[Bootstrap (statistics)]]
* [[Resampling (statistics)]]