Chain rule

Chain rule is a fundamental rule of differential calculus used to calculate the derivative of a composite function. It expresses how the rate of change of an outer function with respect to an inner function combines with the rate of change of the inner function with respect to the independent variable.

If a function is defined as:

$y=f(g(x))$

then the derivative of y with respect to x is:

$\frac{dy}{dx}=\frac{dy}{dg}\frac{dg}{dx}$

Equivalently, if:

$y=f(u)\text{and}u=g(x)$

then:

$\frac{dy}{dx}=f^{\prime }(u)g^{\prime }(x)$

The chain rule is widely used in Calculus, Optimization, Machine learning, and Deep learning. It provides the mathematical basis for Backpropagation, which uses repeated applications of the chain rule to calculate gradients through the layers of an Artificial neural network.

Purpose

The principal purpose of the chain rule is to determine derivatives of functions whose inputs or outputs depend on other functions.

It is particularly useful when a mathematical expression contains nested functions, such as:

$y=f(g(x))$

Basic formula

For a composite function:

$h(x)=f(g(x))$

the derivative is:

$h^{\prime }(x)=f^{\prime }(g(x))g^{\prime }(x)$

The derivative of the outer function is evaluated at the inner function, and the result is multiplied by the derivative of the inner function.

Example

Consider:

$y=(x^2+1{)}^3$

The outer function is:

$f(u)=u^3$

and the inner function is:

$u=x^2+1$

Their derivatives are:

$\frac{dy}{du}=3u^2$

and:

$\frac{du}{dx}=2x$

Applying the chain rule:

$\frac{dy}{dx}=\frac{dy}{du}\frac{du}{dx}$

Therefore:

$\frac{dy}{dx}=3(x^2+1{)}^2(2x)$

or:

$\frac{dy}{dx}=6x(x^2+1{)}^2$

Multivariable form

The chain rule can also be applied to functions involving multiple variables.

If:

$z=f(x,y)$

where both x and y depend on t, then:

$\frac{dz}{dt}=\frac{\partial z}{\partial x}\frac{dx}{dt}+\frac{\partial z}{\partial y}\frac{dy}{dt}$

This form is important in multivariable calculus and optimization.

Chain rule in neural networks

The chain rule is fundamental to the training of artificial neural networks.

A neural network consists of multiple mathematical transformations arranged in sequence. The output of one layer becomes the input of another layer.

For a sequence of functions:

$y=f(g(h(x)))$

the derivative can be calculated as:

$\frac{dy}{dx}=\frac{dy}{df}\frac{df}{dg}\frac{dg}{dh}\frac{dh}{dx}$

More generally, the derivative of a composite function is obtained by multiplying the derivatives of the successive functions.

Relationship with backpropagation

Backpropagation applies the chain rule repeatedly to calculate the gradient of a neural network's Loss function with respect to its parameters.

For a parameter w, the gradient may be expressed as:

$\frac{\partial L}{\partial w}=\frac{\partial L}{\partial y}\frac{\partial y}{\partial w}$

In a multilayer network, additional intermediate derivatives are included:

$\frac{\partial L}{\partial w}=\frac{\partial L}{\partial y}\frac{\partial y}{\partial z}\frac{\partial z}{\partial w}$

Backpropagation uses these chained derivatives to efficiently calculate gradients across many layers.

Applications

The chain rule is widely used in:

• Calculus;
• Differential calculus;
• Optimization;
• Machine learning;
• Deep learning;
• Backpropagation;
• artificial neural networks;
• physics;
• engineering;
• economics.

Importance

The chain rule is one of the fundamental rules of differential calculus. In machine learning and deep learning, it provides the mathematical foundation for calculating gradients through complex computational systems and is therefore essential to the operation of backpropagation and the training of modern neural networks.

See also

• Calculus
• Differential calculus
• Derivative
• Partial derivative
• Gradient
• Backpropagation
• Gradient descent
• Artificial neural network
• Machine learning
• Deep learning