Linear regression

Linear regression is a statistical method used to model the relationship between a dependent variable and one or more explanatory variables by estimating a linear function.

In the simplest case, linear regression relates one dependent variable $Y$ to one explanatory variable $X$.

A simple linear regression model is:

$Y_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i$

where:

• $Y_i$ = observed value of the dependent variable;
• $X_i$ = observed value of the explanatory variable;
• ${\beta }_0$ = intercept;
• ${\beta }_1$ = slope coefficient;
• ${ϵ}_i$ = error term;
• $i$ = observation index.

The fitted regression equation is:

${\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{X}_i$

contentReference[oaicite:0]{index=0}

Ordinary least squares

The most common method for estimating the coefficients of a linear regression is ordinary least squares (OLS).

OLS chooses the coefficients that minimize the sum of squared residuals:

$\underset{{\beta }_0,{\beta }_1}{min}\sum _{i=1}^n(Y_i-{\beta }_0-{\beta }_1{X}_i{)}^2$

The residual for observation $i$ is:

$e_i=Y_i-{\hat{Y}}_i$

The objective of OLS is therefore to minimize:

$\sum _{i=1}^n{e}_i^2$

Slope coefficient

For simple linear regression, the estimated slope coefficient can be calculated as:

${\hat{\beta }}_1=\frac{\sum _{i=1}^n(X_i-\overline{X})(Y_i-\overline{Y})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}$

where:

• $\overline{X}$ = sample mean of $X$;
• $\overline{Y}$ = sample mean of $Y$.

The slope measures the estimated change in $Y$ associated with a one-unit change in $X$.

Intercept

The estimated intercept is:

${\hat{\beta }}_0=\overline{Y}-{\hat{\beta }}_1\overline{X}$

The intercept represents the fitted value of $Y$ when $X=0$, although its practical interpretation depends on the context and whether $X=0$ is meaningful.

Regression in finance

Linear regression is widely used in finance to examine relationships between asset returns and market returns, economic variables, or other risk factors.

A simplified market model is:

$R_i={\alpha }_i+{\beta }_i{R}_m+{ϵ}_i$

where:

• $R_i$ = return of asset $i$;
• ${\alpha }_i$ = intercept or regression alpha;
• ${\beta }_i$ = sensitivity of the asset to market returns;
• $R_m$ = market return;
• ${ϵ}_i$ = residual return.

The fitted equation is:

${\hat{R}}_i={\hat{\alpha }}_i+{\hat{\beta }}_i{R}_m$

In this formulation, beta represents the slope of the regression line and measures the estimated sensitivity of the asset's returns to changes in the market return.

Beta estimation

For a regression of an asset's returns against market returns, beta can be expressed as:

${\hat{\beta }}_i=\frac{\mathrm{Cov}(R_i,R_m)}{\mathrm{Var}(R_m)}$

Using correlation, this can also be written as:

${\hat{\beta }}_i={\rho }_{i,m}\frac{{\sigma }_i}{{\sigma }_m}$

where:

• ${\rho }_{i,m}$ = correlation between asset and market returns;
• ${\sigma }_i$ = standard deviation of asset returns;
• ${\sigma }_m$ = standard deviation of market returns.

Coefficient of determination

The coefficient of determination, commonly denoted as $R^2$, measures the proportion of variation in the dependent variable explained by the fitted regression model.

A common expression is:

$R^2=1-\frac{\sum _{i=1}^n(Y_i-{\hat{Y}}_i{)}^2}{\sum _{i=1}^n(Y_i-\overline{Y}{)}^2}$

where:

• the numerator represents the residual sum of squares;
• the denominator represents the total sum of squares.

For a simple linear regression with an intercept:

$R^2=r_{XY}^2$

where $r_{XY}$ is the Pearson correlation coefficient.

Multiple linear regression

Multiple linear regression extends the model to two or more explanatory variables.

A simplified model is:

$Y_i={\beta }_0+{\beta }_1{X}_{1,i}+{\beta }_2{X}_{2,i}+\cdots +{\beta }_k{X}_{k,i}+{ϵ}_i$

The fitted model is:

${\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{X}_{1,i}+{\hat{\beta }}_2{X}_{2,i}+\cdots +{\hat{\beta }}_k{X}_{k,i}$

Multiple regression allows the relationship between the dependent variable and one explanatory variable to be estimated while controlling for the other included variables.

Factor models

Linear regression is commonly used to estimate factor models in finance.

A simplified multifactor model is:

$R_i-R_f={\alpha }_i+{\beta }_{i1}{F}_1+{\beta }_{i2}{F}_2+\cdots +{\beta }_{ik}{F}_k+{ϵ}_i$

where:

• $R_i-R_f$ = asset's excess return;
• $F_k$ = factor $k$;
• ${\beta }_{ik}$ = asset's exposure to factor $k$;
• ${\alpha }_i$ = intercept;
• ${ϵ}_i$ = residual return.

Such models can be used to estimate an asset's exposure to market, size, value, momentum, quality, and other factors.

Residuals

A regression residual is the difference between the observed value and its fitted value:

$e_i=Y_i-{\hat{Y}}_i$

The residual represents the portion of the observed outcome that is not explained by the fitted regression equation.

In financial models, residual returns can represent asset-specific or idiosyncratic components after accounting for the included explanatory factors.

Statistical significance

Regression coefficients can be evaluated using statistical tests.

For a coefficient ${\beta }_j$, a common null hypothesis is:

$H_0:{\beta }_j=0$

against an alternative hypothesis such as:

$H_1:{\beta }_j\ne 0$

A t-statistic can be represented as:

$t=\frac{{\hat{\beta }}_j-{\beta }_{j,0}}{\mathrm{SE}({\hat{\beta }}_j)}$

where:

• ${\hat{\beta }}_j$ = estimated coefficient;
• ${\beta }_{j,0}$ = hypothesized coefficient value;
• $\mathrm{SE}({\hat{\beta }}_j)$ = standard error of the estimated coefficient.

Statistical significance does not by itself establish economic significance or causation.

Assumptions

Classical OLS inference commonly relies on assumptions concerning the regression model and its error terms.

Depending on the application, these may include:

• linearity of the conditional mean;
• appropriate specification of the model;
• independence or appropriate dependence structure of observations;
• absence of problematic perfect multicollinearity;
• constant error variance for conventional homoskedastic inference;
• appropriate conditions for statistical inference.

Normality of errors is not required for OLS coefficient estimation itself, although it can be relevant for certain small-sample inferential procedures.

Limitations

Linear regression has several limitations:

• it assumes a linear conditional relationship;
• results can be sensitive to outliers;
• omitted variables can affect coefficient estimates;
• multicollinearity can make individual coefficients difficult to interpret;
• correlation does not establish causation;
• model estimates can change when the sample period changes;
• financial relationships may be nonlinear or unstable over time.

In financial markets, regression results should therefore be interpreted in the context of the model specification, sample period, data quality, and underlying economic assumptions.

See also

• Ordinary least squares
• Regression analysis
• Multiple linear regression
• Correlation
• Pearson correlation coefficient
• Covariance
• Coefficient of determination
• Beta (finance)
• Alpha (finance)
• Factor model
• Factor investing
• Capital asset pricing model
• Arbitrage pricing theory
• Quantitative finance