Regularization (mathematics)

Regularization (mathematics) is a mathematical technique used to modify an optimization problem by introducing additional constraints or penalty terms to discourage undesirable solutions. In Machine learning, regularization is commonly used to reduce Overfitting by penalizing model complexity and encouraging solutions that generalize better to previously unseen data.

A regularized optimization objective can generally be represented as:

$J(\theta )=L(\theta )+\lambda R(\theta )$

where:

• J(θ) = regularized objective function;
• L(θ) = original loss or objective function;
• R(θ) = regularization term;
• λ = regularization parameter;
• θ = model parameters.

The regularization term imposes an additional cost on certain parameter values or model characteristics. The parameter λ controls the strength of the regularization.

Purpose

The principal purposes of regularization are to:

• reduce overfitting;
• control model complexity;
• improve generalization;
• stabilize optimization;
• discourage excessively large or complex parameter values;
• incorporate prior constraints into an optimization problem.

Regularization in machine learning

In Machine learning, a model may fit its training data extremely closely while performing poorly on new data. This phenomenon is known as Overfitting.

Regularization modifies the training objective so that the model is penalized for undesirable complexity.

A general regularized objective can be written as:

${\theta }^{\ast }=\arg \underset{\theta }{min}[L(\theta )+\lambda R(\theta )]$

The model therefore seeks a balance between fitting the training data and satisfying the regularization constraint.

L1 regularization

L1 regularization adds a penalty based on the absolute values of model parameters:

$R(\theta )=\sum _{j=1}^p|{\theta }_j|$

The resulting objective is:

$J(\theta )=L(\theta )+\lambda \sum _{j=1}^p|{\theta }_j|$

L1 regularization can encourage some model parameters to become exactly zero. It is therefore commonly associated with sparse model representations and feature selection.

L2 regularization

L2 regularization adds a penalty based on the squared values of model parameters:

$R(\theta )=\sum _{j=1}^p{\theta }_j^2$

The resulting objective is:

$J(\theta )=L(\theta )+\lambda \sum _{j=1}^p{\theta }_j^2$

L2 regularization generally discourages large parameter values without necessarily forcing individual parameters to zero.

In linear and neural-network models, L2 regularization is often associated with weight decay.

Elastic net regularization

Elastic net combines L1 and L2 regularization.

A simplified form is:

$J(\theta )=L(\theta )+{\lambda }_1\sum _{j=1}^p|{\theta }_j|+{\lambda }_2\sum _{j=1}^p{\theta }_j^2$

This approach combines the sparsity-inducing properties of L1 regularization with the parameter-shrinking properties of L2 regularization.

Regularization parameter

The regularization parameter controls the strength of the penalty.

For the objective:

$J(\theta )=L(\theta )+\lambda R(\theta )$

a larger λ places greater emphasis on the regularization term, while a smaller λ places greater emphasis on fitting the training data.

An excessively large regularization parameter may cause Underfitting, while insufficient regularization may leave the model vulnerable to overfitting.

Regularization and optimization

Regularization changes the objective function that an optimization algorithm attempts to minimize.

For example, Gradient descent can optimize:

$J(\theta )=L(\theta )+\lambda R(\theta )$

rather than the unregularized loss alone.

The resulting parameter updates therefore reflect both prediction error and the regularization penalty.

Regularization and neural networks

Regularization is widely used in artificial neural networks and Deep learning.

Common techniques include:

• L1 regularization;
• L2 regularization;
• weight decay;
• dropout;
• early stopping;
• data augmentation.

These techniques differ mathematically, but they are generally intended to reduce overfitting or improve the generalization of the model.

Dropout

Dropout is a neural-network regularization technique in which selected units are temporarily omitted during training according to a specified probability.

The technique reduces reliance on particular network pathways and can improve generalization in some models.

Early stopping

Early stopping is a regularization technique in which training is stopped when performance on a validation dataset stops improving or begins to deteriorate.

It limits the extent to which a model can adapt to the training data and may therefore reduce overfitting.

Data augmentation

Data augmentation generates additional training examples through transformations of existing data.

Examples include image transformations, text transformations, or other domain-specific modifications.

By increasing the variety of training examples, data augmentation can improve model generalization.

Example

Suppose a model has an original loss function:

$L(\theta )$

An L2-regularized objective can be written as:

$J(\theta )=L(\theta )+\lambda \sum _{j=1}^p{\theta }_j^2$

The optimization algorithm therefore attempts to find parameter values that both fit the training data and avoid unnecessarily large parameter magnitudes.

Relationship with loss function

A Loss function measures the discrepancy between a model's predictions and the desired outcomes.

Regularization adds an additional penalty to the loss or objective, producing a regularized objective function.

Thus:

$\text{Regularized objective}=\text{Loss}+\text{Regularization penalty}$

Relationship with gradient descent

Gradient descent can minimize a regularized objective by calculating gradients that include both the loss and regularization terms.

For:

$J(\theta )=L(\theta )+\lambda R(\theta )$

the gradient is:

${\mathrm{\nabla }}_{\theta }J(\theta )={\mathrm{\nabla }}_{\theta }L(\theta )+\lambda {\mathrm{\nabla }}_{\theta }R(\theta )$

The optimization process therefore accounts for both prediction error and the regularization penalty.

Relationship with overfitting

Overfitting occurs when a model captures patterns specific to its training data that do not generalize well to new observations.

Regularization can reduce overfitting by constraining model complexity or discouraging parameter configurations that produce unnecessarily complex models.

Applications

Regularization is widely used in:

• Machine learning;
• Deep learning;
• artificial neural networks;
• regression;
• classification;
• statistical modelling;
• signal processing;
• inverse problems;
• numerical optimization.

Advantages

Regularization can improve model generalization, reduce overfitting, stabilize parameter estimates, and constrain overly complex solutions.

It also provides a mathematical framework for incorporating preferences or constraints into optimization problems.

Limitations

Excessive regularization can cause underfitting and reduce a model's ability to capture meaningful patterns in the data.

The appropriate regularization method and strength depend on the model, dataset, objective, and application.

Importance

Regularization is a fundamental technique in mathematical optimization and machine learning because it provides a systematic way to control model complexity and constrain solutions. In modern machine learning, regularization works alongside loss functions and optimization algorithms to improve generalization and reduce the risk of overfitting.

See also

• Machine learning
• Deep learning
• Loss function
• Gradient descent
• Overfitting
• Underfitting
• Artificial neural network
• Weight decay
• Dropout (neural networks)
• Elastic net regularization
• Lasso (statistics)
• Ridge regression