Analysis of Variance :: SAS/STAT(R) 12.3 User's Guide

Analysis of Variance

The identity

$\bY = \bX \tilde{\bbeta } + \left(\bY -\bX \tilde{\bbeta }\right)$

holds for all vectors $\tilde{\bbeta }$ , but only for the least squares solution is the residual $(\bY -\bX \widehat{\bbeta })$ orthogonal to the predicted value $\bX \widehat{\bbeta }$ . Because of this orthogonality, the additive identity holds not only for the vectors themselves, but also for their lengths (Pythagorean theorem):

$||\bY ||^2 = ||\bX \widehat{\bbeta }||^2 + ||(\bY -\bX \widehat{\bbeta })||^2$

Note that $\bX \widehat{\bbeta } = \bX \left(\bX ’\bX \right)^{-1}\bX ’\bY$ = $\bH \bY$ and note that $\bY - \bX \widehat{\bbeta } = (\bI - \bH )\bY = \bM \bY$ . The matrices $\mb {H}$ and $\bM = \bI -\bH$ play an important role in the theory of linear models and in statistical computations. Both are projection matrices—that is, they are symmetric and idempotent. (An idempotent matrix $\bA$ is a square matrix that satisfies $\bA \bA = \bA$ . The eigenvalues of an idempotent matrix take on the values 1 and 0 only.) The matrix $\mb {H}$ projects onto the subspace of that is spanned by the columns of $\bX$ . The matrix $\bM$ projects onto the orthogonal complement of that space. Because of these properties you have $\mb {H}’=\mb {H}$ , $\mb {HH}=\mb {H}$ , $\mb {M}’=\mb {M}$ , $\mb {MM}=\mb {M}$ , $\mb {HM}=\mb {0}$ .

The Pythagorean relationship now can be written in terms of $\bH$ and $\bM$ as follows:

$||\bY ||^2 = \bY ’\bY = ||\mb {HY}||^2 + ||\mb {MY}||^2 = \bY ’\bH ’\bH \bY + \bY ’\bM ’\bM \bY = \bY ’\mb {HY} + \bY ’\mb {MY}$

If $\bX ’\bX$ is deficient in rank and a generalized inverse is used to solve the normal equations, then you work instead with the projection matrices $\bH =\bX \left(\bX ’\bX \right)^{-}\bX ’$ . Note that if $\bG$ is a generalized inverse of $\bX ’\bX$ , then $\mb {XGX}’$ , and hence also $\bH$ and $\bM$ , are invariant to the choice of $\bG$ .

The matrix $\bH$ is sometimes referred to as the “hat” matrix because when you premultiply the vector of observations with $\mb {H}$ , you produce the fitted values, which are commonly denoted by placing a “hat” over the $\bY$ vector, $\widehat{\bY } = \mb {HY}$ .

The term $\bY ’\bY$ is the uncorrected total sum of squares ( $\mr {SST}$ ) of the linear model, $\bY ’\mb {MY}$ is the error (residual) sum of squares ( $\mr {SSR}$ ), and $\bY ’\mb {HY}$ is the uncorrected model sum of squares. This leads to the analysis of variance table shown in Table 3.2.

Table 3.2: Analysis of Variance with Uncorrected Sums of Squares

Source	df	Sum of Squares
Model	$\mr {rank}(\bX )$	$\mr {SSM} = \bY ’\bH \bY = \widehat{\bbeta }’\bX ’\bY$
Residual	$n-\mr {rank}(\bX )$	$\mr {SSR} = \bY ’\bM \bY = \bY ’\bY - \widehat{\bbeta }\bX ’\bY =$
		$\sum _{i=1}^ n\left(Y_ i - \widehat{Y}_ i\right)^2$
Uncorr. Total	n	$\mr {SST} = \bY ’\bY = \sum _{i=1}^ n Y_ i^2$

When the model contains an intercept term, then the analysis of variance is usually corrected for the mean, as shown in Table 3.3.

Table 3.3: Analysis of Variance with Corrected Sums of Squares

Source	df	Sum of Squares
Model	$\mr {rank}(\bX )-1$	$\mr {SSM}_ c = \widehat{\bbeta }’\bX ’\bY - n\overline{Y}^2 = \sum _{i=1}^ n\left(\widehat{Y}_ i - \overline{Y}_ i\right)^2$
Residual	$n-\mr {rank}(\bX )$	$\mr {SSR} = \bY ’\bM \bY = \bY ’\bY - \widehat{\bbeta }\bX ’\bY =$
		$\sum _{i=1}^ n\left(Y_ i - \widehat{Y}_ i\right)^2$
Corrected Total		$\mr {SST}_ c = \bY ’\bY - n\overline{Y}^2 = \sum _{i=1}^ n\left(Y_ i - \overline{Y}\right)^2$

The coefficient of determination, also called the R-square statistic, measures the proportion of the total variation explained by the linear model. In models with intercept, it is defined as the ratio

$R^2 = 1 - \frac{ \mr {SSR} }{ \mr {SST}_ c } = 1 - \frac{ \sum _{i=1}^ n \left(Y_ i - \widehat{Y}_ i\right)^2 }{ \sum _{i=1}^ n \left(Y_ i - \overline{Y} \right)^2 }$

In models without intercept, the R-square statistic is a ratio of the uncorrected sums of squares

$R^2 = 1 - \frac{ \mr {SSR} }{ \mr {SST} } = 1 - \frac{ \sum _{i=1}^ n \left(Y_ i - \widehat{Y}_ i\right)^2 }{ \sum _{i=1}^ n Y_ i^2 }$