Multiple linear regression

Multiple linear regression is a statistical method used to model the relationship between a dependent variable and two or more explanatory variables. It extends simple linear regression by allowing multiple explanatory variables to be included in the same model.

A general multiple linear regression model is:

$Y_i={\beta }_0+{\beta }_1{X}_{1,i}+{\beta }_2{X}_{2,i}+\cdots +{\beta }_k{X}_{k,i}+{ϵ}_i$

where:

• $Y_i$ = dependent variable for observation $i$;
• $X_{j,i}$ = value of explanatory variable $j$ for observation $i$;
• ${\beta }_0$ = intercept;
• ${\beta }_j$ = coefficient associated with explanatory variable $j$;
• ${ϵ}_i$ = error term;
• $k$ = number of explanatory variables;
• $i$ = observation index.

The fitted regression equation is:

${\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{X}_{1,i}+{\hat{\beta }}_2{X}_{2,i}+\cdots +{\hat{\beta }}_k{X}_{k,i}$

Ordinary least squares

Multiple linear regression is commonly estimated using ordinary least squares (OLS).

OLS chooses the coefficient estimates that minimize the sum of squared residuals:

$\underset{{\beta }_0,{\beta }_1,\dots ,{\beta }_k}{min}\sum _{i=1}^n{(Y_i-{\beta }_0-\sum _{j=1}^k{\beta }_j{X}_{j,i})}^2$

The residual for observation $i$ is:

$e_i=Y_i-{\hat{Y}}_i$

The residual sum of squares is:

$\mathrm{R}\mathrm{S}\mathrm{S}=\sum _{i=1}^n{e}_i^2$

Matrix formulation

The multiple linear regression model can be written in matrix form as:

$𝐲=𝐗𝜷+𝝐$

where:

• $𝐲$ = vector of dependent-variable observations;
• $𝐗$ = design matrix of explanatory variables;
• $𝜷$ = vector of regression coefficients;
• $𝝐$ = vector of error terms.

Under the standard OLS formulation, the coefficient estimator is:

$\widehat{𝜷}=({𝐗}^{𝖳}𝐗{)}^{-1}{𝐗}^{𝖳}𝐲$

provided that ${𝐗}^{𝖳}𝐗$ is invertible.

Interpretation of coefficients

In multiple linear regression, a coefficient represents the estimated change in the dependent variable associated with a one-unit change in the corresponding explanatory variable, holding the other included explanatory variables constant.

For example:

$Y={\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2+ϵ$

Here, ${\beta }_1$ represents the estimated change in $Y$ associated with a one-unit increase in $X_1$, conditional on $X_2$ remaining unchanged.

Similarly, ${\beta }_2$ measures the corresponding conditional association between $X_2$ and $Y$.

This interpretation depends on the model specification and does not by itself establish causation.

Intercept

The intercept is the expected or fitted value of the dependent variable when all explanatory variables equal zero:

${\beta }_0=E[Y\mid X_1=0,X_2=0,\dots ,X_k=0]$

under the conditional-mean interpretation of the model.

The practical meaning of the intercept depends on whether zero is a meaningful value for all explanatory variables.

Coefficient of determination

The coefficient of determination, denoted by $R^2$, measures the proportion of sample variation in the dependent variable accounted for by the fitted regression model relative to an intercept-only model.

It is calculated as:

$R^2=1-\frac{\mathrm{R}\mathrm{S}\mathrm{S}}{\mathrm{T}\mathrm{S}\mathrm{S}}$

where:

$\mathrm{T}\mathrm{S}\mathrm{S}=\sum _{i=1}^n(Y_i-\overline{Y}{)}^2$

is the total sum of squares.

Equivalently:

$R^2=\frac{\mathrm{E}\mathrm{S}\mathrm{S}}{\mathrm{T}\mathrm{S}\mathrm{S}}$

where $\mathrm{E}\mathrm{S}\mathrm{S}$ is the explained sum of squares.

Unlike simple linear regression, the coefficient of determination in multiple regression is not generally equal to the square of the correlation between any single explanatory variable and the dependent variable.

Adjusted R-squared

Adding explanatory variables to a regression cannot decrease ordinary $R^2$, even when the additional variables provide little useful information.

The adjusted R-squared accounts for the number of explanatory variables:

${\overline{R}}^2=1-(1-R^2)\frac{n-1}{n-k-1}$

where:

• ${\overline{R}}^2$ = adjusted coefficient of determination;
• $R^2$ = coefficient of determination;
• $n$ = number of observations;
• $k$ = number of explanatory variables.

Adjusted $R^2$ can decrease when an additional explanatory variable does not provide sufficient improvement in model fit.

Statistical inference

Multiple linear regression can be used to test hypotheses about individual coefficients.

For an explanatory variable $X_j$, a common null hypothesis is:

$H_0:{\beta }_j=0$

A corresponding t-statistic is:

$t=\frac{{\hat{\beta }}_j-{\beta }_{j,0}}{\mathrm{SE}({\hat{\beta }}_j)}$

where:

• ${\hat{\beta }}_j$ = estimated coefficient;
• ${\beta }_{j,0}$ = hypothesized coefficient value;
• $\mathrm{SE}({\hat{\beta }}_j)$ = standard error of the estimated coefficient.

A joint hypothesis concerning several coefficients can be evaluated using an F-test.

For example:

$H_0:{\beta }_1={\beta }_2=\cdots ={\beta }_q=0$

The F-test evaluates whether the specified group of explanatory variables provides statistically significant explanatory power within the model.

Multicollinearity

Multicollinearity occurs when explanatory variables are highly linearly related.

For example, if:

$X_2\approx a+b{X}_1$

for constants $a$ and $b$, the two explanatory variables contain substantial overlapping information.

Perfect multicollinearity prevents unique estimation of the standard OLS coefficients because the design matrix does not have full column rank.

High but imperfect multicollinearity does not prevent estimation but can:

• increase coefficient standard errors;
• make individual coefficients unstable;
• make coefficient interpretation more difficult;
• cause coefficients to change substantially when variables are added or removed.

Heteroskedasticity

Heteroskedasticity occurs when the conditional variance of the regression errors is not constant:

$\mathrm{Var}({ϵ}_i\mid 𝐗)\ne {\sigma }^2$

OLS coefficient estimates can remain unbiased under appropriate exogeneity conditions, but conventional standard errors may be unreliable.

Heteroskedasticity-consistent standard errors can be used for inference that is robust to certain forms of non-constant error variance.

Autocorrelation

In time-series and other ordered data, regression errors may be correlated across observations.

A simplified representation is:

$\mathrm{Cov}({ϵ}_t,{ϵ}_{t-k})\ne 0$

for some non-zero lag $k$.

Serial correlation can affect the efficiency of OLS estimates and the validity of conventional standard errors, depending on the model and assumptions.

Regression in finance

Multiple linear regression is widely used in finance to model asset returns as a function of multiple market or economic variables.

A simplified financial model is:

$R_i={\alpha }_i+{\beta }_{i1}{F}_1+{\beta }_{i2}{F}_2+\cdots +{\beta }_{ik}{F}_k+{ϵ}_i$

where:

• $R_i$ = return of asset $i$;
• ${\alpha }_i$ = regression intercept;
• $F_k$ = factor $k$;
• ${\beta }_{ik}$ = asset's exposure to factor $k$;
• ${ϵ}_i$ = residual return.

If excess returns are modeled, the equation may be written as:

$R_i-R_f={\alpha }_i+{\beta }_{i1}{F}_1+{\beta }_{i2}{F}_2+\cdots +{\beta }_{ik}{F}_k+{ϵ}_i$

Such models can estimate an asset's exposure to multiple systematic risk factors.

Market and factor models

A simple market model is:

$R_i={\alpha }_i+{\beta }_i{R}_m+{ϵ}_i$

Multiple regression extends this framework by including additional explanatory variables.

For example, a multifactor model may include market, size, value, momentum, and profitability factors:

$R_i-R_f={\alpha }_i+{\beta }_{iM}M+{\beta }_{iS}S+{\beta }_{iV}V+{\beta }_{iMOM}MOM+{\beta }_{iQ}Q+{ϵ}_i$

where each beta represents the estimated exposure of the asset to the corresponding factor.

Prediction

Once the coefficients have been estimated, the model can be used to calculate fitted values or generate predictions.

For a new observation with explanatory-variable vector ${𝐱}_0$:

${\hat{Y}}_0={𝐱}_0^{𝖳}\widehat{𝜷}$

Prediction intervals can incorporate both uncertainty in the estimated regression function and uncertainty associated with the individual outcome.

Advantages

Multiple linear regression can:

• model several explanatory variables simultaneously;
• estimate conditional relationships;
• control statistically for included explanatory variables;
• quantify the marginal association of explanatory variables;
• generate fitted values and predictions;
• provide statistical tests and confidence intervals;
• support financial factor and risk models.

Limitations

Multiple linear regression has several limitations:

• statistical association does not establish causation;
• omitted variables can bias coefficient estimates;
• multicollinearity can make individual coefficients imprecise;
• outliers can materially influence estimates;
• nonlinear relationships may not be adequately represented;
• model results depend on the selected explanatory variables;
• relationships can change over time;
• high $R^2$ does not necessarily indicate a correctly specified model.

In financial applications, regression estimates can also be affected by changing market regimes, structural breaks, non-stationarity, and time-varying volatility.

See also

• Linear regression
• Regression analysis
• Ordinary least squares
• Coefficient of determination
• Adjusted R-squared
• Multicollinearity
• Heteroskedasticity
• F-test
• Pearson correlation coefficient
• Covariance
• Beta (finance)
• Factor model
• Capital asset pricing model
• Arbitrage pricing theory
• Quantitative finance