Gradient descent

Gradient descent is an iterative optimization algorithm used to minimize a loss function by adjusting the parameters of a mathematical model in the direction that reduces the loss. It is one of the fundamental optimization methods used in Machine learning and Deep learning, particularly for training artificial neural networks.

Gradient descent uses the gradient of the loss function to determine the direction of greatest local increase in the loss. The parameters are then updated in the opposite direction, with the magnitude of the update controlled by the learning rate.

For a model parameter θ, a basic gradient descent update is:

$θ_{t + 1} = θ_{t} - η \nabla_{θ} L (θ_{t})$

where:

θ_t = parameter value at iteration t;
L(θ) = loss function;
∇_θL(θ_t) = gradient of the loss with respect to the parameters;
η = learning rate;
θ_t+1 = updated parameter value.

Purpose

The principal purpose of gradient descent is to find parameter values that minimize an objective or loss function.

In machine learning, the loss function measures how closely a model's predictions correspond to the desired outcomes. Gradient descent iteratively modifies the model parameters to reduce this loss.

Process

A basic gradient descent procedure consists of:

Initialize the model parameters.
Calculate the model's output for the available data.
Calculate the loss.
Calculate the gradient of the loss with respect to the model parameters.
Update the parameters in the direction opposite to the gradient.
Repeat the process until a specified stopping condition is reached.

The process can be represented as:

$θ_{t + 1} = θ_{t} - η \nabla_{θ} L (θ_{t})$

Gradient

The gradient is a vector containing the partial derivatives of a function with respect to its parameters.

For a function of two variables:

$L (x, y)$

the gradient is:

$\nabla L = [\begin{matrix} \frac{\partial L}{\partial x} \\ \frac{\partial L}{\partial y} \end{matrix}]$

The gradient points in the direction of the greatest local increase of the function. Gradient descent moves in the opposite direction to reduce the function value.

Learning rate

The learning rate determines the size of each parameter update.

A simplified update for a parameter θ is:

$θ_{t + 1} = θ_{t} - η \frac{\partial L}{\partial θ}$

If the learning rate is too large, optimization may become unstable or fail to converge. If it is too small, training may require an excessive number of iterations.

Selecting an appropriate learning rate is therefore an important part of model optimization.

Batch gradient descent

Batch gradient descent calculates the gradient using the entire training dataset before performing a parameter update.

For a dataset containing m observations:

$\nabla L (θ) = \frac{1}{m} \sum_{i = 1}^{m} \nabla L_{i} (θ)$

This approach produces a relatively stable gradient estimate but may require substantial computational resources for large datasets.

Stochastic gradient descent

Stochastic gradient descent (SGD) updates model parameters using one training example at a time.

For an individual observation:

$θ_{t + 1} = θ_{t} - η \nabla L_{i} (θ_{t})$

SGD can perform updates more frequently than batch gradient descent but introduces greater variability into the optimization process.

Mini-batch gradient descent

Mini-batch gradient descent calculates gradients using a small subset of the training dataset.

For a mini-batch B:

$\nabla L_{B} (θ) = \frac{1}{| B |} \sum_{i \in B} \nabla L_{i} (θ)$

Mini-batch training is widely used in modern deep learning because it provides a practical balance between computational efficiency and gradient stability.

Optimization methods

Several optimization algorithms extend or modify basic gradient descent.

Common examples include:

Stochastic gradient descent;
momentum;
Adam;
RMSProp;
AdaGrad.

These methods may adjust the learning rate, incorporate information from previous gradients, or apply other techniques to improve optimization.

Gradient descent in neural networks

In an Artificial neural network, the model contains potentially large numbers of trainable weights and biases.

Backpropagation calculates the gradients of the loss function with respect to these parameters. Gradient descent or another optimization algorithm then uses those gradients to update the parameters.

The process can therefore be summarized as:

$Forward pass \to Loss calculation \to Backpropagation \to Parameter update$

Gradient descent and backpropagation

Backpropagation and gradient descent perform different but complementary functions.

Backpropagation efficiently calculates the gradients of the loss function through the layers of a neural network, while gradient descent uses those gradients to update the network's parameters.

The combination is fundamental to the training of many neural networks.

Local minima and saddle points

Gradient descent can encounter optimization challenges such as:

local minima;
saddle points;
flat regions;
steep regions;
poorly conditioned loss surfaces.

In high-dimensional neural networks, saddle points and flat regions may be more relevant optimization challenges than simple local minima.

Different optimization methods and training techniques are used to improve convergence.

Applications

Gradient descent is widely used in:

Advantages

Gradient descent provides a relatively simple and computationally efficient method for optimizing models with large numbers of parameters.

Its mathematical formulation allows it to be applied to a broad range of differentiable optimization problems and makes it particularly suitable for training large neural networks.

Limitations

Gradient descent may require many iterations to reach a satisfactory solution and can be sensitive to the choice of learning rate and parameter initialization.

Other limitations include:

slow convergence in some loss landscapes;
sensitivity to poorly scaled parameters;
difficulty with flat or highly curved regions;
potential convergence to suboptimal solutions;
computational requirements for large datasets.

Modern optimizers and learning-rate schedules are often used to address some of these limitations.

Importance

Gradient descent is a foundational optimization method in modern machine learning and deep learning. By using gradients to iteratively reduce an objective function, it provides the mechanism through which the parameters of many statistical and neural-network models are optimized. Together with backpropagation, it forms a central part of the training process for contemporary artificial intelligence systems.