Overfitting

Overfitting is a phenomenon in Machine learning in which a model learns the training data too closely, including patterns, noise, or random variations that do not generalize to new or previously unseen data. An overfitted model may achieve very low error on its training dataset while performing substantially worse on validation or test data.

Overfitting is commonly associated with models that have excessive complexity relative to the amount or quality of available training data. It can be addressed through techniques such as regularization, cross-validation, early stopping, data augmentation, model simplification, and increasing the amount or diversity of training data.

Characteristics

An overfitted model commonly exhibits:

very strong performance on training data;
weaker performance on validation or test data;
a substantial difference between training and generalization error;
sensitivity to noise or unusual observations;
excessive model complexity relative to the available data.

The defining characteristic is poor generalization rather than simply high model complexity.

Example

Suppose a model is trained to predict house prices using property characteristics.

A model that learns meaningful relationships between property size, location, and price may perform well on both training and previously unseen data.

An excessively complex model may instead memorize unusual characteristics of individual properties in the training dataset. It may achieve very low training error but produce inaccurate predictions for properties that were not included in the training data.

This behavior is an example of overfitting.

Training and test performance

Overfitting can be identified by comparing model performance on training and unseen data.

A typical pattern is:

$Training error ≪ Test error$

As model complexity increases, training error may continue to decrease while test error initially decreases and subsequently increases.

This divergence indicates that the model may be fitting noise or idiosyncratic patterns in the training data.

Relationship with underfitting

Underfitting occurs when a model is too simple to capture important patterns in the data.

Overfitting and underfitting represent different forms of poor generalization:

Underfitting — the model fails to learn sufficient structure from the data.
Overfitting — the model learns the training data too specifically and fails to generalize.

An effective model seeks an appropriate balance between model complexity and generalization.

Bias-variance tradeoff

Overfitting is commonly discussed in relation to the Bias–variance tradeoff.

Models with high variance may be highly sensitive to the particular training dataset and therefore more likely to overfit.

A simplified decomposition of expected prediction error can be expressed as:

$Expected error = {Bias}^{2} + Variance + Irreducible error$

The objective is not necessarily to minimize model complexity, but to achieve a level of complexity that provides good generalization.

Causes

Overfitting may arise from several factors, including:

excessive model complexity;
insufficient training data;
noisy training data;
highly correlated or irrelevant features;
excessive training;
inadequate regularization;
data leakage;
insufficiently representative training data.

The relative importance of these factors depends on the model and dataset.

Regularization

Regularization is one of the principal techniques used to reduce overfitting.

A regularized objective may be represented as:

$J (θ) = L (θ) + λ R (θ)$

where:

L(θ) = original loss function;
R(θ) = regularization term;
λ = regularization strength.

The additional penalty discourages certain complex model configurations.

Common regularization techniques include:

L1 regularization;
L2 regularization;
weight decay;
dropout.

Cross-validation

Cross-validation evaluates a model using multiple training and validation partitions.

It provides a more robust estimate of how a model is likely to perform on unseen data and can assist in selecting model complexity and hyperparameters.

Early stopping

Early stopping is a regularization technique in which training is stopped when performance on validation data stops improving.

It is commonly used in neural-network training to prevent continued optimization of the training loss from causing deterioration in validation performance.

Data augmentation

Data augmentation creates additional training examples by applying appropriate transformations to existing data.

It can increase the effective diversity of a training dataset and reduce a model's tendency to memorize particular training examples.

Model simplification

Reducing model complexity can also reduce overfitting.

Possible approaches include:

reducing the number of model parameters;
removing irrelevant features;
reducing network depth or width;
restricting the complexity of decision rules;
selecting a simpler model architecture.

Increasing training data

Increasing the amount and diversity of representative training data can reduce overfitting by providing the model with more examples of the underlying relationships it is expected to learn.

Additional data is particularly useful when the original dataset is small relative to model complexity.

Overfitting in neural networks

Overfitting is an important consideration in artificial neural networks and Deep learning.

Large neural networks can contain millions or billions of trainable parameters. Without appropriate training procedures and regularization, such models may learn patterns that do not generalize to the intended data distribution.

Techniques such as dropout, weight decay, data augmentation, and early stopping may be used to improve generalization.

Overfitting in machine learning

Overfitting can occur across many types of machine-learning models, including:

linear and nonlinear regression;
decision trees;
ensemble methods;
support vector machines;
neural networks;
deep-learning models.

The mechanisms and appropriate mitigation techniques vary according to the model and application.

Detection

Overfitting can be assessed using separate training, validation, and test datasets.

A common diagnostic is to compare training and validation performance as model complexity or training duration changes.

For example, if training loss continues to decrease while validation loss begins to increase, the model may be overfitting the training data.

Applications and implications

Overfitting is particularly important in applications where models must perform reliably on previously unseen observations, including:

predictive modelling;
financial forecasting;
medical research;
computer vision;
natural language processing;
fraud detection;
scientific modelling;
recommendation systems.

Limitations of mitigation techniques

No single technique guarantees that overfitting will be eliminated.

Regularization may cause underfitting if applied excessively, while additional training data may not improve generalization if it is unrepresentative or contains similar biases and noise.

Model evaluation must therefore consider the data-generating environment and the intended deployment conditions.

Importance

Overfitting is a fundamental challenge in machine learning because the objective of a predictive model is generally to perform well on data beyond the observations used for training. Understanding and controlling overfitting is therefore essential for developing models that generalize reliably to new data.