View source for Ordinary least squares

'''[[Ordinary least squares]]''' ('''OLS''') is a method for estimating the parameters of a [[Linear regression|linear regression]] model by minimizing the sum of the squared differences between observed and fitted values.

For a simple linear regression model:

<math>
Y_i
=
\beta_0
+
\beta_1X_i
+
\epsilon_i
</math>

the fitted value is:

<math>
\hat{Y}_i
=
\hat{\beta}_0
+
\hat{\beta}_1X_i
</math>

and the residual is:

<math>
e_i
=
Y_i-\hat{Y}_i
</math>

The OLS estimator chooses the coefficients that minimize the sum of squared residuals:

<math>
\min_{\beta_0,\beta_1}
\sum_{i=1}^{n}
(Y_i-\beta_0-\beta_1X_i)^2
</math>

==Objective function==

The OLS objective function is commonly called the '''residual sum of squares''' (RSS) or '''sum of squared errors''' (SSE):

<math>
\mathrm{RSS}
=
\sum_{i=1}^{n}
(Y_i-\hat{Y}_i)^2
</math>

The estimated coefficients are therefore:

<math>
(\hat{\beta}_0,\hat{\beta}_1)
=
\underset{\beta_0,\beta_1}{\operatorname{arg\,min}}
\sum_{i=1}^{n}
(Y_i-\beta_0-\beta_1X_i)^2
</math>

The use of squared residuals prevents positive and negative residuals from cancelling each other and places greater weight on larger deviations.

==Simple linear regression==

For simple linear regression, the OLS estimate of the slope coefficient is:

<math>
\hat{\beta}_1
=
\frac{
\sum_{i=1}^{n}
(X_i-\bar{X})(Y_i-\bar{Y})
}{
\sum_{i=1}^{n}
(X_i-\bar{X})^2
}
</math>

The intercept is:

<math>
\hat{\beta}_0
=
\bar{Y}
-
\hat{\beta}_1\bar{X}
</math>

where:

* <math>\bar{X}</math> = sample mean of the explanatory variable;
* <math>\bar{Y}</math> = sample mean of the dependent variable.

The fitted regression equation is therefore:

<math>
\hat{Y}_i
=
\hat{\beta}_0
+
\hat{\beta}_1X_i
</math>

==Geometric interpretation==

OLS selects the line that minimizes the squared vertical distance between the observed data points and their fitted values.

The residual for each observation is:

<math>
e_i
=
Y_i-\hat{Y}_i
</math>

and the objective is:

<math>
\sum_{i=1}^{n}e_i^2
\rightarrow
\min
</math>

This produces the line of best fit under the least-squares criterion.

==Multiple linear regression==

OLS can be extended to models containing multiple explanatory variables:

<math>
Y_i
=
\beta_0
+
\beta_1X_{1,i}
+
\beta_2X_{2,i}
+
\cdots
+
\beta_kX_{k,i}
+
\epsilon_i
</math>

The fitted model is:

<math>
\hat{Y}_i
=
\hat{\beta}_0
+
\hat{\beta}_1X_{1,i}
+
\hat{\beta}_2X_{2,i}
+
\cdots
+
\hat{\beta}_kX_{k,i}
</math>

OLS estimates the coefficients by minimizing:

<math>
\min_{\boldsymbol{\beta}}
\sum_{i=1}^{n}
(Y_i-\mathbf{x}_i^{\mathsf{T}}\boldsymbol{\beta})^2
</math>

where <math>\boldsymbol{\beta}</math> is the vector of regression coefficients.

==Matrix formulation==

The multiple regression model can be written in matrix form as:

<math>
\mathbf{y}
=
\mathbf{X}\boldsymbol{\beta}
+
\boldsymbol{\epsilon}
</math>

The OLS estimator is:

<math>
\hat{\boldsymbol{\beta}}
=
(\mathbf{X}^{\mathsf{T}}\mathbf{X})^{-1}
\mathbf{X}^{\mathsf{T}}\mathbf{y}
</math>

provided that <math>\mathbf{X}^{\mathsf{T}}\mathbf{X}</math> is invertible.

The fitted values are:

<math>
\hat{\mathbf{y}}
=
\mathbf{X}\hat{\boldsymbol{\beta}}
</math>

and the residual vector is:

<math>
\mathbf{e}
=
\mathbf{y}-\hat{\mathbf{y}}
</math>

==Normal equations==

The OLS estimates satisfy the normal equations:

<math>
\mathbf{X}^{\mathsf{T}}
(\mathbf{y}-\mathbf{X}\hat{\boldsymbol{\beta}})
=
\mathbf{0}
</math>

Equivalently:

<math>
\mathbf{X}^{\mathsf{T}}\mathbf{X}
\hat{\boldsymbol{\beta}}
=
\mathbf{X}^{\mathsf{T}}\mathbf{y}
</math>

These equations arise from setting the derivatives of the sum of squared residuals with respect to the regression coefficients equal to zero.

==OLS in finance==

OLS is widely used in financial analysis to estimate relationships between asset returns and market or factor returns.

A simplified market model is:

<math>
R_i
=
\alpha_i
+
\beta_iR_m
+
\epsilon_i
</math>

where:

* <math>R_i</math> = return of asset <math>i</math>;
* <math>\alpha_i</math> = regression intercept;
* <math>\beta_i</math> = sensitivity of the asset to market returns;
* <math>R_m</math> = market return;
* <math>\epsilon_i</math> = residual return.

OLS can be used to estimate:

<math>
(\hat{\alpha}_i,\hat{\beta}_i)
=
\underset{\alpha_i,\beta_i}{\operatorname{arg\,min}}
\sum_{t=1}^{n}
(R_{i,t}-\alpha_i-\beta_iR_{m,t})^2
</math>

The estimated beta is:

<math>
\hat{\beta}_i
=
\frac{
\operatorname{Cov}(R_i,R_m)
}{
\operatorname{Var}(R_m)
}
</math>

==Factor models==

OLS can also be used to estimate multifactor models.

A simplified factor model is:

<math>
R_i-R_f
=
\alpha_i
+
\beta_{i1}F_1
+
\beta_{i2}F_2
+
\cdots
+
\beta_{ik}F_k
+
\epsilon_i
</math>

where:

* <math>R_i-R_f</math> = asset's excess return;
* <math>F_k</math> = factor <math>k</math>;
* <math>\beta_{ik}</math> = estimated exposure to factor <math>k</math>;
* <math>\alpha_i</math> = intercept;
* <math>\epsilon_i</math> = residual return.

OLS estimates the factor exposures by minimizing the squared residuals.

==Statistical properties==

Under the classical linear regression assumptions, OLS has important statistical properties.

The [[Gauss–Markov theorem]] states that, under its applicable assumptions, OLS is the '''best linear unbiased estimator''' (BLUE) of the regression coefficients.

In particular, the OLS estimator is:

* linear in the observed dependent variable;
* unbiased under the relevant assumptions;
* minimum variance among linear unbiased estimators.

These properties depend on the assumptions of the regression model.

==Assumptions==

Common assumptions associated with classical OLS inference include:

* linearity of the conditional mean;
* appropriate model specification;
* no perfect multicollinearity among explanatory variables;
* zero conditional mean of the error term;
* appropriate assumptions concerning the dependence structure of observations;
* homoskedasticity for conventional OLS standard errors;
* appropriate conditions for statistical inference.

The exact assumptions required depend on whether the objective is coefficient estimation, unbiasedness, efficiency, hypothesis testing, or prediction.

==Heteroskedasticity==

'''Heteroskedasticity''' occurs when the variance of the regression error differs across observations or levels of the explanatory variables.

It can be represented as:

<math>
\operatorname{Var}(\epsilon_i\mid X_i)
\neq
\sigma^2
</math>

OLS coefficient estimates can remain unbiased under appropriate exogeneity conditions, but conventional standard errors may be incorrect.

[[Heteroskedasticity-consistent standard errors]] can be used to obtain inference that is robust to certain forms of heteroskedasticity.

==Multicollinearity==

'''Multicollinearity''' occurs when explanatory variables in a multiple regression are highly linearly related.

Perfect multicollinearity prevents the standard OLS coefficient formula from being uniquely defined because:

<math>
\mathbf{X}^{\mathsf{T}}\mathbf{X}
</math>

is not invertible.

High but imperfect multicollinearity does not prevent estimation but can increase the standard errors of individual coefficient estimates and make their interpretation more difficult.

==OLS and coefficient of determination==

OLS regression is commonly evaluated using the [[Coefficient of determination|coefficient of determination]]:

<math>
R^2
=
1-
\frac{
\mathrm{RSS}
}{
\mathrm{TSS}
}
</math>

where:

<math>
\mathrm{TSS}
=
\sum_{i=1}^{n}
(Y_i-\bar{Y})^2
</math>

is the total sum of squares.

For a regression containing an intercept, OLS minimizes RSS and decomposes the total variation into explained and residual components.

==Limitations==

OLS has several limitations:

* results can be sensitive to outliers;
* omitted variables can bias coefficient estimates;
* multicollinearity can make individual coefficients imprecise;
* heteroskedasticity can invalidate conventional standard errors;
* autocorrelation can affect inference in time-series applications;
* linear models may not adequately represent nonlinear relationships;
* statistical association does not establish causation.

In financial applications, OLS estimates can also be affected by structural changes, non-stationarity, changing volatility, and changing relationships between financial variables.

==See also==

* [[Linear regression]]
* [[Multiple linear regression]]
* [[Regression analysis]]
* [[Gauss–Markov theorem]]
* [[Coefficient of determination]]
* [[Pearson correlation coefficient]]
* [[Covariance]]
* [[Variance]]
* [[Beta (finance)]]
* [[Alpha (finance)]]
* [[Factor model]]
* [[Capital asset pricing model]]
* [[Arbitrage pricing theory]]
* [[Econometrics]]
* [[Quantitative finance]]

[[Category:Regression analysis]]
[[Category:Statistics]]
[[Category:Econometrics]]
[[Category:Financial mathematics]]
[[Category:Quantitative finance]]