Regression analysis

Regression analysis is a set of statistical methods used to examine the relationship between a dependent variable and one or more explanatory variables. It is used to estimate relationships, quantify associations, test hypotheses, and generate predictions.

In its simplest form, regression analysis models a dependent variable $Y$ as a function of an explanatory variable $X$:

$Y_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i$

where:

• $Y_i$ = dependent variable;
• $X_i$ = explanatory variable;
• ${\beta }_0$ = intercept;
• ${\beta }_1$ = slope coefficient;
• ${ϵ}_i$ = error term;
• $i$ = observation index.

Types of regression

Regression analysis includes a range of statistical techniques. Common forms include:

• Linear regression;
• Multiple linear regression;
• Polynomial regression;
• Logistic regression;
• Ridge regression;
• Lasso regression;
• Quantile regression;
• Nonlinear regression;
• Time series regression.

The appropriate method depends on the nature of the dependent variable, explanatory variables, data structure, and purpose of the analysis.

Linear regression

In linear regression, the conditional mean of the dependent variable is modeled as a linear function of explanatory variables.

A simple linear regression model is:

$Y_i={\beta }_0+{\beta }_1{X}_i+{ϵ}_i$

The fitted relationship is:

${\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{X}_i$

where ${\hat{Y}}_i$ is the predicted or fitted value.

Ordinary least squares

Ordinary least squares (OLS) is a commonly used method for estimating the coefficients of a linear regression model.

OLS chooses the coefficients that minimize the sum of squared residuals:

$\underset{𝜷}{min}\sum _{i=1}^n(Y_i-{𝐱}_i^{𝖳}𝜷{)}^2$

For simple linear regression, this becomes:

$\underset{{\beta }_0,{\beta }_1}{min}\sum _{i=1}^n(Y_i-{\beta }_0-{\beta }_1{X}_i{)}^2$

The residual for observation $i$ is:

$e_i=Y_i-{\hat{Y}}_i$

and the residual sum of squares is:

$\mathrm{R}\mathrm{S}\mathrm{S}=\sum _{i=1}^n{e}_i^2$

Multiple regression

Multiple linear regression extends the model to multiple explanatory variables:

$Y_i={\beta }_0+{\beta }_1{X}_{1,i}+{\beta }_2{X}_{2,i}+\cdots +{\beta }_k{X}_{k,i}+{ϵ}_i$

The coefficient ${\beta }_j$ represents the estimated association between $X_j$ and $Y$, conditional on the other explanatory variables included in the model.

In matrix form:

$𝐲=𝐗𝜷+𝝐$

Under the standard OLS formulation, the coefficient estimator is:

$\widehat{𝜷}=({𝐗}^{𝖳}𝐗{)}^{-1}{𝐗}^{𝖳}𝐲$

provided that ${𝐗}^{𝖳}𝐗$ is invertible.

Regression coefficients

Regression coefficients describe the estimated relationship between explanatory variables and the dependent variable.

For a simple linear regression:

${\hat{\beta }}_1=\frac{\sum _{i=1}^n(X_i-\overline{X})(Y_i-\overline{Y})}{\sum _{i=1}^n(X_i-\overline{X}{)}^2}$

The intercept is:

${\hat{\beta }}_0=\overline{Y}-{\hat{\beta }}_1\overline{X}$

The slope coefficient indicates the estimated change in $Y$ associated with a one-unit change in $X$.

Residuals and fitted values

A fitted value is the value predicted by the estimated regression model:

${\hat{Y}}_i=f(X_i,\widehat{𝜷})$

A residual is the difference between the observed and fitted values:

$e_i=Y_i-{\hat{Y}}_i$

Residual analysis is commonly used to evaluate whether the regression model adequately represents important features of the data.

Patterns in residuals can indicate problems such as nonlinear relationships, heteroskedasticity, omitted variables, or other forms of model misspecification.

Coefficient of determination

The coefficient of determination, commonly denoted by $R^2$, measures the proportion of sample variation in the dependent variable accounted for by the fitted regression model relative to an intercept-only model.

For a regression with an intercept:

$R^2=1-\frac{\sum _{i=1}^n(Y_i-{\hat{Y}}_i{)}^2}{\sum _{i=1}^n(Y_i-\overline{Y}{)}^2}$

The numerator is the residual sum of squares, while the denominator is the total sum of squares.

In a simple linear regression with an intercept:

$R^2=r_{XY}^2$

where $r_{XY}$ is the Pearson correlation coefficient.

Statistical inference

Regression analysis can be used to estimate uncertainty around regression coefficients and test hypotheses concerning those coefficients.

For a coefficient ${\beta }_j$, a common null hypothesis is:

$H_0:{\beta }_j=0$

A corresponding t-statistic can be written as:

$t=\frac{{\hat{\beta }}_j-{\beta }_{j,0}}{\mathrm{SE}({\hat{\beta }}_j)}$

where:

• ${\hat{\beta }}_j$ = estimated coefficient;
• ${\beta }_{j,0}$ = hypothesized coefficient value;
• $\mathrm{SE}({\hat{\beta }}_j)$ = standard error of the estimated coefficient.

Confidence intervals can also be constructed around regression coefficients.

Regression assumptions

The assumptions required for regression analysis depend on the model and the type of inference being performed.

For classical linear regression, commonly considered assumptions include:

• linearity of the conditional mean;
• appropriate model specification;
• no perfect multicollinearity;
• zero conditional mean of the error term;
• appropriate dependence structure of observations;
• constant error variance for conventional homoskedastic inference.

Normality of errors is not required for OLS coefficient estimation itself, although it can be relevant to certain small-sample inferential procedures.

Heteroskedasticity

Heteroskedasticity occurs when the conditional variance of the error term is not constant:

$\mathrm{Var}({ϵ}_i\mid X_i)\ne {\sigma }^2$

OLS coefficient estimates can remain unbiased under appropriate exogeneity conditions, but conventional standard errors may be unreliable.

Heteroskedasticity-consistent standard errors can be used for inference that is robust to certain forms of non-constant error variance.

Autocorrelation

In time-series and other ordered data, regression errors may be correlated across observations.

A simplified representation is:

$\mathrm{Cov}({ϵ}_t,{ϵ}_{t-k})\ne 0$

for some non-zero lag $k$.

Serial correlation can affect the efficiency of OLS estimates and the validity of conventional standard errors, depending on the model and assumptions.

Multicollinearity

Multicollinearity occurs when explanatory variables in a multiple regression are strongly linearly related.

Perfect multicollinearity prevents unique estimation of the standard OLS coefficients.

High but imperfect multicollinearity can increase the standard errors of individual coefficient estimates and make the interpretation of individual coefficients more difficult.

Regression in finance

Regression analysis is widely used in finance to examine relationships between asset returns and market returns, economic variables, or risk factors.

A simplified market model is:

$R_i={\alpha }_i+{\beta }_i{R}_m+{ϵ}_i$

where:

• $R_i$ = return of asset $i$;
• ${\alpha }_i$ = regression intercept;
• ${\beta }_i$ = sensitivity of the asset to market returns;
• $R_m$ = market return;
• ${ϵ}_i$ = residual return.

The beta coefficient can be estimated as:

${\hat{\beta }}_i=\frac{\mathrm{Cov}(R_i,R_m)}{\mathrm{Var}(R_m)}$

The regression intercept can be interpreted within the specified model as the expected asset return when the market return is zero.

Factor models

Regression analysis is commonly used to estimate factor models.

A simplified multifactor model is:

$R_i-R_f={\alpha }_i+{\beta }_{i1}{F}_1+{\beta }_{i2}{F}_2+\cdots +{\beta }_{ik}{F}_k+{ϵ}_i$

where:

• $R_i-R_f$ = asset's excess return;
• $F_k$ = factor $k$;
• ${\beta }_{ik}$ = asset's exposure to factor $k$;
• ${\alpha }_i$ = regression intercept;
• ${ϵ}_i$ = residual return.

Such models can be used to estimate exposures to market, size, value, momentum, quality, and other factors.

Prediction and forecasting

Regression models can be used to generate fitted values and predictions.

A fitted value is:

${\hat{Y}}_i={𝐱}_i^{𝖳}\widehat{𝜷}$

For a new observation with explanatory-variable vector ${𝐱}_0$, a model-based prediction is:

${\hat{Y}}_0={𝐱}_0^{𝖳}\widehat{𝜷}$

Prediction intervals may be wider than confidence intervals for the estimated mean because predictions also incorporate uncertainty associated with the individual outcome.

Correlation and regression

Regression and correlation are related but measure different concepts.

The Pearson correlation coefficient measures the strength and direction of a linear association:

${\rho }_{XY}=\frac{\mathrm{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}$

Regression additionally specifies a dependent variable and estimates the conditional relationship between that variable and one or more explanatory variables.

Therefore:

$\text{Correlation}\to \text{Strength and Direction of Linear Association}$

$\text{Regression}\to \text{Estimated Conditional Relationship}$

Neither correlation nor regression, by itself, establishes causation.

Limitations

Regression analysis has several limitations:

• statistical association does not establish causation;
• results depend on model specification;
• omitted variables can affect estimates;
• outliers can materially influence results;
• multicollinearity can make individual coefficients imprecise;
• nonlinear relationships may not be adequately represented by a linear model;
• relationships can change over time;
• statistical significance does not necessarily imply economic significance.

In financial applications, regression relationships may also be affected by changing market regimes, structural breaks, non-stationarity, and time-varying volatility.

See also

• Linear regression
• Multiple linear regression
• Ordinary least squares
• Coefficient of determination
• Pearson correlation coefficient
• Covariance
• Variance
• Beta (finance)
• Alpha (finance)
• Factor model
• Econometrics
• Time series
• Quantitative finance
• Statistical model