Introduction to Regression Procedures

Linear Regression Models

In matrix notation, a linear model is written as

\[ \bY = \bX \bbeta + \bepsilon \]

where $\bX $ is the $(n \times k)$ design matrix (rows are observations and columns are the regressors), $\bbeta $ is the $(k \times 1)$ vector of unknown parameters, and $\bepsilon $ is the $(n \times 1)$ vector of unobservable model errors. The first column of $\bX $ is usually a vector of 1s and is used to estimate the intercept term.

The statistical theory of linear models is based on strict classical assumptions. Ideally, you measure the response after controlling all factors in an experimentally determined environment. If you cannot control the factors experimentally, some tests must be interpreted as being conditional on the observed values of the regressors.

Other assumptions are as follows:

  • The form of the model is correct (all important explanatory variables have been included). This assumption is reflected mathematically in the assumption of a zero mean of the model errors, $\mr{E}[\bepsilon ] = \mb{0}$.

  • Regressor variables are measured without error.

  • The expected value of the errors is 0.

  • The variance of the error (and thus the dependent variable) for the ith observation is $\sigma ^2/w_ i$, where $w_ i$ is a known weight factor. Usually, $w_ i=1$ for all i and thus $\sigma ^2$ is the common, constant variance.

  • The errors are uncorrelated across observations.

When hypotheses are tested, or when confidence and prediction intervals are computed, an additional assumption is made that the errors are normally distributed.