Hyperparameter optimization

Hyperparameter optimization is the process of selecting values for the hyperparameters of a Machine learning model in order to improve its performance according to a specified evaluation criterion. Unlike model parameters, which are learned from training data, hyperparameters are generally specified before or during the training process and control aspects of model architecture, training, or regularization.

Hyperparameter optimization is commonly performed using cross-validation or a separate validation dataset. It is widely used in Machine learning, Deep learning, statistical modelling, and artificial intelligence.

Hyperparameters

Hyperparameters are configuration values that influence how a machine-learning model is trained or structured.

Examples include:

• learning rate;
• regularization strength;
• number of training iterations;
• batch size;
• number of hidden layers;
• number of neurons in a neural network;
• tree depth;
• number of trees in an ensemble;
• dropout rate.

Hyperparameters differ from model parameters because model parameters are typically learned from the training data, whereas hyperparameters are selected through the model-development process.

Purpose

The principal objectives of hyperparameter optimization are to:

• improve predictive performance;
• improve model generalization;
• reduce Overfitting;
• identify suitable model complexity;
• optimize training efficiency;
• determine appropriate regularization;
• compare different model configurations.

Basic process

A typical hyperparameter optimization process consists of:

1. Defining the hyperparameters to be optimized.
2. Specifying possible values or ranges for those hyperparameters.
3. Selecting an evaluation procedure.
4. Training models using different hyperparameter configurations.
5. Evaluating each configuration.
6. Selecting the configuration that provides the preferred performance.
7. Training the final model using the selected configuration.

The evaluation procedure commonly uses cross-validation.

Search space

The search space defines the possible values that can be considered during hyperparameter optimization.

For example, a learning rate may be searched over a range such as:

$\eta \in \{10^{-1},10^{-2},10^{-3},10^{-4}\}$

A continuous or discrete range can be specified depending on the hyperparameter.

The choice of search space can substantially influence the resulting optimization process.

Grid search

Grid search evaluates a predefined set of combinations of hyperparameter values.

For example, if two hyperparameters each have three possible values, grid search evaluates:

$3\times 3=9$

configurations.

Grid search is straightforward and systematic but can become computationally expensive as the number of hyperparameters or candidate values increases.

Random search

Random search selects hyperparameter combinations randomly from the specified search space.

Unlike grid search, random search does not allocate equal numbers of evaluations to every possible value along every dimension.

Random search can be more efficient than grid search when only a subset of hyperparameters has a substantial effect on model performance.

Bayesian optimization

Bayesian optimization uses a probabilistic model of the relationship between hyperparameter configurations and their observed performance.

The optimization procedure uses previous evaluations to determine which configurations are promising and should be evaluated next.

This approach can reduce the number of expensive model-training evaluations required when individual evaluations are computationally costly.

Evolutionary optimization

Evolutionary optimization applies concepts inspired by biological evolution to search for effective hyperparameter configurations.

Candidate configurations may undergo selection, mutation, and recombination to produce new configurations.

Evolutionary methods can be useful for complex search spaces but may require substantial computational resources.

Successive halving

Successive halving allocates limited computational resources to many candidate configurations initially and progressively eliminates poorly performing configurations.

More resources are then allocated to the remaining candidates.

This approach can reduce the computational cost of evaluating large numbers of configurations.

Hyperband

Hyperband is a resource-allocation strategy that combines random sampling with successive-halving procedures.

It evaluates multiple configurations using different resource budgets and concentrates computational resources on promising configurations.

Cross-validation

Cross-validation is commonly used to evaluate hyperparameter configurations.

For each candidate configuration, the model can be trained and evaluated across multiple folds.

The mean validation performance can be calculated as:

${\overline{M}}_{CV}=\frac{1}{k}\sum _{i=1}^k{M}_i$

where:

• k = number of folds;
• M_i = performance measure obtained from fold i.

The configuration with the preferred cross-validation performance can then be selected.

Nested cross-validation

Nested cross-validation can be used when an unbiased estimate of the complete model-selection process is required.

An inner cross-validation loop is used for hyperparameter optimization, while an outer loop is used to estimate generalization performance.

This separation helps prevent the hyperparameter-selection process from being evaluated on the same data used to make the selection.

Hyperparameter optimization and overfitting

Hyperparameter optimization can itself lead to overfitting if a validation dataset is repeatedly used to select configurations.

A model-development process may therefore inadvertently become adapted to the validation data.

Separating model selection from final evaluation can reduce this risk.

Objective function

Hyperparameter optimization can be represented as an optimization problem.

Let λ represent a vector of hyperparameters and let M(λ) represent the model's validation performance.

The objective can be expressed as:

${\lambda }^{\ast }=\arg \underset{\lambda }{min}M(\lambda )$

for a metric where lower values are preferred.

For metrics where higher values are preferred, the objective may instead be expressed as:

${\lambda }^{\ast }=\arg \underset{\lambda }{max}M(\lambda )$

Examples

For a neural network, hyperparameter optimization may evaluate:

• learning rate;
• batch size;
• number of hidden layers;
• number of neurons;
• dropout rate;
• weight-decay coefficient.

For a decision tree, the search may include:

• maximum tree depth;
• minimum samples required for a split;
• minimum samples per leaf.

For an ensemble model, hyperparameters may include the number of trees and other model-specific configuration values.

Relationship with model parameters

Hyperparameters and model parameters serve different functions.

Model parameters are generally learned from training data. Examples include neural-network weights and biases.

Hyperparameters are selected outside the ordinary parameter-learning process. Examples include learning rate, regularization strength, tree depth, and batch size.

The distinction is important because hyperparameter optimization determines the configuration under which model parameters are subsequently learned.

Relationship with cross-validation

Cross-validation provides a method for estimating how a particular hyperparameter configuration performs on unseen data.

Hyperparameter optimization uses these performance estimates to compare candidate configurations.

The two processes therefore serve complementary purposes:

• Cross-validation evaluates a configuration.
• Hyperparameter optimization searches for a preferred configuration.

Relationship with regularization

Regularization introduces a penalty or constraint intended to control model complexity.

The strength of regularization is often itself a hyperparameter.

For example, an L2-regularized objective may be written as:

$J(\theta )=L(\theta )+\lambda R(\theta )$

where λ can be optimized as a hyperparameter.

Applications

Hyperparameter optimization is widely used in:

• Machine learning;
• Deep learning;
• artificial neural networks;
• Natural language processing;
• Computer vision;
• predictive modelling;
• statistical modelling;
• Generative artificial intelligence;
• large language models.

Advantages

Hyperparameter optimization provides a systematic method for selecting model configurations and can improve predictive performance, generalization, and training efficiency.

Automated search methods can also reduce the need to manually select configuration values.

Limitations

Hyperparameter optimization can be computationally expensive, particularly for models that require substantial time or resources to train.

Other limitations include:

• sensitivity to the search space;
• risk of overfitting to validation data;
• difficulty optimizing very large configuration spaces;
• dependence on the chosen evaluation metric;
• substantial computational requirements for large models.

An optimized configuration may also perform poorly when deployment conditions differ from the data used during model development.

Importance

Hyperparameter optimization is an important component of modern machine-learning model development. By systematically searching for effective model configurations and evaluating them using appropriate validation procedures, it can improve model performance while providing a structured alternative to manual parameter selection.

See also

• Machine learning
• Cross-validation (statistics)
• Overfitting
• Regularization (mathematics)
• Model selection
• Grid search
• Random search
• Bayesian optimization
• Deep learning
• Gradient descent
• Loss function