Ordinary least squares

Ordinary least squares (OLS) is a method for estimating the parameters of a linear regression model by minimizing the sum of the squared differences between observed and fitted values.

For a simple linear regression model:

$Y_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i$

the fitted value is:

${\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{X}_i$

and the residual is:

$e_i=Y_i-{\hat{Y}}_i$

The OLS estimator chooses the coefficients that minimize the sum of squared residuals:

$\underset{{\beta }_0,{\beta }_1}{min}\sum _{i=1}^n(Y_i-{\beta }_0-{\beta }_1{X}_i{)}^2$

Objective function

The OLS objective function is commonly called the residual sum of squares (RSS) or sum of squared errors (SSE):

$\mathrm{R}\mathrm{S}\mathrm{S}=\sum _{i=1}^n(Y_i-{\hat{Y}}_i{)}^2$

The estimated coefficients are therefore:

$({\hat{\beta }}_0,{\hat{\beta }}_1)=\underset{{\beta }_0,{\beta }_1}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{i=1}^n(Y_i-{\beta }_0-{\beta }_1{X}_i{)}^2$

The use of squared residuals prevents positive and negative residuals from cancelling each other and places greater weight on larger deviations.

Simple linear regression

For simple linear regression, the OLS estimate of the slope coefficient is:

${\hat{\beta }}_1=\frac{\sum _{i=1}^n(X_i-\overline{X})(Y_i-\overline{Y})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}$

The intercept is:

${\hat{\beta }}_0=\overline{Y}-{\hat{\beta }}_1\overline{X}$

where:

• $\overline{X}$ = sample mean of the explanatory variable;
• $\overline{Y}$ = sample mean of the dependent variable.

The fitted regression equation is therefore:

${\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{X}_i$

Geometric interpretation

OLS selects the line that minimizes the squared vertical distance between the observed data points and their fitted values.

The residual for each observation is:

$e_i=Y_i-{\hat{Y}}_i$

and the objective is:

$\sum _{i=1}^n{e}_i^2\to min$

This produces the line of best fit under the least-squares criterion.

Multiple linear regression

OLS can be extended to models containing multiple explanatory variables:

$Y_i={\beta }_0+{\beta }_1{X}_{1,i}+{\beta }_2{X}_{2,i}+\cdots +{\beta }_k{X}_{k,i}+{ϵ}_i$

The fitted model is:

${\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{X}_{1,i}+{\hat{\beta }}_2{X}_{2,i}+\cdots +{\hat{\beta }}_k{X}_{k,i}$

OLS estimates the coefficients by minimizing:

$\underset{𝜷}{min}\sum _{i=1}^n(Y_i-{𝐱}_i^{𝖳}𝜷{)}^2$

where $𝜷$ is the vector of regression coefficients.

Matrix formulation

The multiple regression model can be written in matrix form as:

$𝐲=𝐗𝜷+𝝐$

The OLS estimator is:

$\widehat{𝜷}=({𝐗}^{𝖳}𝐗{)}^{-1}{𝐗}^{𝖳}𝐲$

provided that ${𝐗}^{𝖳}𝐗$ is invertible.

The fitted values are:

$\widehat{𝐲}=𝐗\widehat{𝜷}$

and the residual vector is:

$𝐞=𝐲-\widehat{𝐲}$

Normal equations

The OLS estimates satisfy the normal equations:

${𝐗}^{𝖳}(𝐲-𝐗\widehat{𝜷})=\mathrm{𝟎}$

Equivalently:

${𝐗}^{𝖳}𝐗\widehat{𝜷}={𝐗}^{𝖳}𝐲$

These equations arise from setting the derivatives of the sum of squared residuals with respect to the regression coefficients equal to zero.

OLS in finance

OLS is widely used in financial analysis to estimate relationships between asset returns and market or factor returns.

A simplified market model is:

$R_i={\alpha }_i+{\beta }_i{R}_m+{ϵ}_i$

where:

• $R_i$ = return of asset $i$;
• ${\alpha }_i$ = regression intercept;
• ${\beta }_i$ = sensitivity of the asset to market returns;
• $R_m$ = market return;
• ${ϵ}_i$ = residual return.

OLS can be used to estimate:

$({\hat{\alpha }}_i,{\hat{\beta }}_i)=\underset{{\alpha }_i,{\beta }_i}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{t=1}^n(R_{i,t}-{\alpha }_i-{\beta }_i{R}_{m,t}{)}^2$

The estimated beta is:

${\hat{\beta }}_i=\frac{\mathrm{Cov}(R_i,R_m)}{\mathrm{Var}(R_m)}$

Factor models

OLS can also be used to estimate multifactor models.

A simplified factor model is:

$R_i-R_f={\alpha }_i+{\beta }_{i1}{F}_1+{\beta }_{i2}{F}_2+\cdots +{\beta }_{ik}{F}_k+{ϵ}_i$

where:

• $R_i-R_f$ = asset's excess return;
• $F_k$ = factor $k$;
• ${\beta }_{ik}$ = estimated exposure to factor $k$;
• ${\alpha }_i$ = intercept;
• ${ϵ}_i$ = residual return.

OLS estimates the factor exposures by minimizing the squared residuals.

Statistical properties

Under the classical linear regression assumptions, OLS has important statistical properties.

The Gauss–Markov theorem states that, under its applicable assumptions, OLS is the best linear unbiased estimator (BLUE) of the regression coefficients.

In particular, the OLS estimator is:

• linear in the observed dependent variable;
• unbiased under the relevant assumptions;
• minimum variance among linear unbiased estimators.

These properties depend on the assumptions of the regression model.

Assumptions

Common assumptions associated with classical OLS inference include:

• linearity of the conditional mean;
• appropriate model specification;
• no perfect multicollinearity among explanatory variables;
• zero conditional mean of the error term;
• appropriate assumptions concerning the dependence structure of observations;
• homoskedasticity for conventional OLS standard errors;
• appropriate conditions for statistical inference.

The exact assumptions required depend on whether the objective is coefficient estimation, unbiasedness, efficiency, hypothesis testing, or prediction.

Heteroskedasticity

Heteroskedasticity occurs when the variance of the regression error differs across observations or levels of the explanatory variables.

It can be represented as:

$\mathrm{Var}({ϵ}_i\mid X_i)\ne {\sigma }^2$

OLS coefficient estimates can remain unbiased under appropriate exogeneity conditions, but conventional standard errors may be incorrect.

Heteroskedasticity-consistent standard errors can be used to obtain inference that is robust to certain forms of heteroskedasticity.

Multicollinearity

Multicollinearity occurs when explanatory variables in a multiple regression are highly linearly related.

Perfect multicollinearity prevents the standard OLS coefficient formula from being uniquely defined because:

${𝐗}^{𝖳}𝐗$

is not invertible.

High but imperfect multicollinearity does not prevent estimation but can increase the standard errors of individual coefficient estimates and make their interpretation more difficult.

OLS and coefficient of determination

OLS regression is commonly evaluated using the coefficient of determination:

$R^2=1-\frac{\mathrm{R}\mathrm{S}\mathrm{S}}{\mathrm{T}\mathrm{S}\mathrm{S}}$

where:

$\mathrm{T}\mathrm{S}\mathrm{S}=\sum _{i=1}^n(Y_i-\overline{Y}{)}^2$

is the total sum of squares.

For a regression containing an intercept, OLS minimizes RSS and decomposes the total variation into explained and residual components.

Limitations

OLS has several limitations:

• results can be sensitive to outliers;
• omitted variables can bias coefficient estimates;
• multicollinearity can make individual coefficients imprecise;
• heteroskedasticity can invalidate conventional standard errors;
• autocorrelation can affect inference in time-series applications;
• linear models may not adequately represent nonlinear relationships;
• statistical association does not establish causation.

In financial applications, OLS estimates can also be affected by structural changes, non-stationarity, changing volatility, and changing relationships between financial variables.

See also

• Linear regression
• Multiple linear regression
• Regression analysis
• Gauss–Markov theorem
• Coefficient of determination
• Pearson correlation coefficient
• Covariance
• Variance
• Beta (finance)
• Alpha (finance)
• Factor model
• Capital asset pricing model
• Arbitrage pricing theory
• Econometrics
• Quantitative finance