The UCM Procedure

Statistics of Fit

This section explains the goodness-of-fit statistics reported to measure how well the specified model fits the data.

First the various statistics of fit that are computed using the prediction errors, $y_{t} - \hat{y}_{t}$, are considered. In these formulas, n is the number of nonmissing prediction errors and k is the number of fitted parameters in the model. Moreover, the sum of squared errors, ${\mi{SSE} = \sum {( y_{t} - \hat{y}_{t} )^{2}}}$, and the total sum of squares for the series corrected for the mean, $SST = {\sum {( y_{t} - {\overline y} )^{2}}}$, where ${{\overline y}}$ is the series mean, and the sums are over all the nonmissing prediction errors.

Mean Squared Error The mean squared prediction error, ${\mi{MSE} = \frac{1}{n} \mi{SSE} }$

Root Mean Squared Error The root mean square error, RMSE = ${\sqrt {\mi{MSE} }}$

Mean Absolute Percent Error The mean absolute percent prediction error, MAPE = ${\frac{100}{n} \sum _{t=1}^{n}{{|( y_{t} - \hat{y}_{t} ) / y_{t} |}}}$.The summation ignores observations where $y_ t = 0$.

R-square The R-square statistic, ${ \mi{R}^{2}=1-\mi{SSE} / \mi{SST} }$.If the model fits the series badly, the model error sum of squares, SSE, might be larger than SST and the R-square statistic will be negative.

Adjusted R-square The adjusted R-square statistic, ${1 - (\frac{n-1}{n-k}) (1- \mi{R} ^{2}) }$

Amemiya’s Adjusted R-square Amemiya’s adjusted R-square, ${1 - (\frac{n+k}{n-k}) (1 - \mi{R} ^{2}) }$

Random Walk R-square The random walk R-square statistic (Harvey’s R-square statistic that uses the random walk model for comparison), ${1 - (\frac{n-1}{n}) \mi{SSE} / \mi{RWSSE} }$, where ${\mi{RWSSE} = \sum _{t=2}^{n}{( y_{t} - y_{t-1} - {\mu } )^{2}}}$, and ${{\mu } = \frac{1}{n-1} \sum _{t=2}^{n}{( y_{t} - y_{t-1} )}}$

Maximum Percent Error The largest percent prediction error, ${100 ~ \mr{max} ( ( y_{t} - \hat{y}_{t} ) / y_{t} )}$. In this computation the observations where $y_ t = 0$ are ignored.

The likelihood-based fit statistics are reported separately (see the section The UCMs as State Space Models). They include the full log likelihood ($ L_{\infty }$), the diffuse part of the log likelihood, the normalized residual sum of squares, and several information criteria: AIC, AICC, HQIC, BIC, and CAIC. Let q denote the number of estimated parameters, n be the number of nonmissing measurements in the estimation span, and d be the number of diffuse elements in the initial state vector that are successfully initialized during the Kalman filtering process. Moreover, let $n^* = (n-d)$. The reported information criteria, all in smaller-is-better form, are described in Table 34.4:

Table 34.4: Information Criteria

Criterion

Formula

Reference

AIC

$-2 L_{\infty } + 2q$

Akaike (1974)

AICC

$-2 L_{\infty } + 2q n^*/(n^*-q-1)$

Hurvich and Tsai (1989)

   

Burnham and Anderson (1998)

HQIC

$-2 L_{\infty } + 2q \log \log ( n^* )$

Hannan and Quinn (1979)

BIC

$-2 L_{\infty } + q \log ( n^* )$

Schwarz (1978)

CAIC

$-2 L_{\infty } + q(\log ( n^* ) + 1)$

Bozdogan (1987)