The Yule-Walker Method

Let represent the vector of autoregressive parameters,

and let the variance matrix of the error vector be ,

If the vector of autoregressive parameters is known, the matrix can be computed from the autoregressive parameters. is then . Given , the efficient estimates of regression parameters can be computed using generalized least squares (GLS). The GLS estimates then yield the unbiased estimate of the variance ,

The Yule-Walker method alternates estimation of using generalized least squares with estimation of using the Yule-Walker equations applied to the sample autocorrelation function. The YW method starts by forming the OLS estimate of . Next, is estimated from the sample autocorrelation function of the OLS residuals by using the Yule-Walker equations. Then is estimated from the estimate of , and is estimated from and the OLS estimate of . The autocorrelation corrected estimates of the regression parameters are then computed by GLS, using the estimated matrix. These are the Yule-Walker estimates.

If the ITER option is specified, the Yule-Walker residuals are used to form a new sample autocorrelation function, the new autocorrelation function is used to form a new estimate of and , and the GLS estimates are recomputed using the new variance matrix. This alternation of estimates continues until either the maximum change in the estimate between iterations is less than the value specified by the CONVERGE= option or the maximum number of allowed iterations is reached. This produces the iterated Yule-Walker estimates. Iteration of the estimates may not yield much improvement.

The Yule-Walker equations, solved to obtain and a preliminary estimate of , are

Here , where is the lag i sample autocorrelation. The matrix is the Toeplitz matrix whose i,jth element is . If you specify a subset model, then only the rows and columns of and corresponding to the subset of lags specified are used.

If the BACKSTEP option is specified, for purposes of significance testing, the matrix is treated as a sum-of-squares-and-crossproducts matrix arising from a simple regression with observations, where k is the number of estimated parameters.

The Unconditional Least Squares and Maximum Likelihood Methods

Define the transformed error, , as

where .

The unconditional sum of squares for the model, , is

The ULS estimates are computed by minimizing with respect to the parameters and .

The full log likelihood function for the autoregressive error model is

where denotes determinant of . For the ML method, the likelihood function is maximized by minimizing an equivalent sum-of-squares function.

Maximizing l with respect to (and concentrating out of the likelihood) and dropping the constant term produces the concentrated log likelihood function

Rewriting the variable term within the logarithm gives

PROC AUTOREG computes the ML estimates by minimizing the objective function .

The maximum likelihood estimates may not exist for some data sets (Anderson and Mentz; 1980). This is the case for very regular data sets, such as an exact linear trend.

Computational Methods

Data Transformation and the Kalman Filter

The calculation of from for the general AR model is complicated, and the size of depends on the number of observations. Instead of actually calculating and performing GLS in the usual way, in practice a Kalman filter algorithm is used to transform the data and compute the GLS results through a recursive process.

In all of the estimation methods, the original data are transformed by the inverse of the Cholesky root of . Let denote the Cholesky root of — that is, with lower triangular. For an AR model, is a band diagonal matrix with m anomalous rows at the beginning and the autoregressive parameters along the remaining rows. Thus, if there are no missing values, after the first observations the data are transformed as

The transformation is carried out using a Kalman filter, and the lower triangular matrix is never directly computed. The Kalman filter algorithm, as it applies here, is described in Harvey and Phillips (1979) and Jones (1980). Although is not computed explicitly, for ease of presentation the remaining discussion is in terms of . If there are missing values, then the submatrix of consisting of the rows and columns with nonmissing values is used to generate the transformations.

Gauss-Newton Algorithms

The ULS and ML estimates employ a Gauss-Newton algorithm to minimize the sum of squares and maximize the log likelihood, respectively. The relevant optimization is performed simultaneously for both the regression and AR parameters. The OLS estimates of and the Yule-Walker estimates of are used as starting values for these methods.

The Gauss-Newton algorithm requires the derivatives of or with respect to the parameters. The derivatives with respect to the parameter vector are

These derivatives are computed by the transformation described previously. The derivatives with respect to are computed by differentiating the Kalman filter recurrences and the equations for the initial conditions.

Variance Estimates and Standard Errors

For the Yule-Walker method, the estimate of the error variance, , is the error sum of squares from the last application of GLS, divided by the error degrees of freedom (number of observations N minus the number of free parameters).

The variance-covariance matrix for the components of is taken as for the Yule-Walker method. For the ULS and ML methods, the variance-covariance matrix of the parameter estimates is computed as . For the ULS method, is the matrix of derivatives of with respect to the parameters. For the ML method, is the matrix of derivatives of divided by . The estimate of the variance-covariance matrix of assuming that is known is .

Park and Mitchell (1980) investigated the small sample performance of the standard error estimates obtained from some of these methods. In particular, simulating an AR(1) model for the noise term, they found that the standard errors calculated using GLS with an estimated autoregressive parameter underestimated the true standard errors. These estimates of standard errors are the ones calculated by PROC AUTOREG with the Yule-Walker method.

The estimates of the standard errors calculated with the ULS or ML method take into account the joint estimation of the AR and the regression parameters and may give more accurate standard-error values than the YW method. At the same values of the autoregressive parameters, the ULS and ML standard errors will always be larger than those computed from Yule-Walker. However, simulations of the models used by Park and Mitchell (1980) suggest that the ULS and ML standard error estimates can also be underestimates. Caution is advised, especially when the estimated autocorrelation is high and the sample size is small.

High autocorrelation in the residuals is a symptom of lack of fit. An autoregressive error model should not be used as a nostrum for models that simply do not fit. It is often the case that time series variables tend to move as a random walk. This means that an AR(1) process with a parameter near one absorbs a great deal of the variation. See Example 8.3 later in this chapter, which fits a linear trend to a sine wave.

For ULS or ML estimation, the joint variance-covariance matrix of all the regression and autoregression parameters is computed. For the Yule-Walker method, the variance-covariance matrix is computed only for the regression parameters.

Lagged Dependent Variables

The Yule-Walker estimation method is not directly appropriate for estimating models that include lagged dependent variables among the regressors. Therefore, the maximum likelihood method is the default when the LAGDEP or LAGDEP= option is specified in the MODEL statement. However, when lagged dependent variables are used, the maximum likelihood estimator is not exact maximum likelihood but is conditional on the first few values of the dependent variable.