The Model Selection Criterion

Return to the Develop Models window (Figure 40.9) and notice the Root Mean Square Error button at the right side of the table banner. This is the model selection criterion—the statistic used by the system to select the best fitting model. So far in this example you have fit two models and have left the default criterion, root mean square error (RMSE), in effect. Because the Airline Model has the smaller value of this criterion and because smaller values of the RMSE indicate better fit, the system has chosen this model as the forecasting model, indicated by the check box in the Forecast Model column.

The statistics available as model selection criteria are a subset of the statistics available for informational purposes. To access the entire set, select Options from the menu bar, and then select Statistics of Fit. The Statistics of Fit Selection window appears, as shown in Figure 40.12.

Figure 40.12 Statistics of Fit

By default, five of the more well known statistics are selected. You can select and deselect statistics by clicking the check boxes in the left column. For this exercise, select All, and notice that all the check boxes become checked. Select the OK button to close the window. Now if you choose Statistics of Fit in the Model Viewer window, all of the statistics will be shown for the selected model.

To change the model selection criterion, click the Root Mean Square Error button or select Options from the menu bar and then select Model Selection Criterion. Notice that most of the statistics of fit are shown, but those which are not relevant to model selection, such as number of observations, are not shown. Select Schwarz Bayesian Information Criterion and click OK. Since this statistic puts a high penalty on models with larger numbers of parameters, the ARIMA(0,1,0)(0,1,0)s model comes out with the better fit.

Notice that changing the selection criterion does not automatically select the model that is best according to that criterion. You can always choose the model you want to use for forecasts by selecting its check box in the Forecast Model column.

Now bring up the Model Selection Criterion window again and select Akaike Information Criterion. This statistic puts a lesser penalty on number of parameters, and the Airline Model comes out as the better fitting model.