Working with the Assessment Window
About the Assessment Window
For a decision tree, the Assessment window plots lift, ROC, and misclassification rates. Use the Assessment window to determine how well the model fits the data.
Lift is the ratio of the percent of captured responses within each percentile bin to the average percent of responses for the model. Similarly, cumulative lift is calculated by using all of the data up to and including the current percentile
bin.
A receiver operating characteristic (ROC) chart displays the ability of a model to avoid false positive and false negative classifications. A false positive classification means that an observation has been identified as an event when it is actually a nonevent (also referred to as a Type I error). A false negative classification means that
an observation has been identified as a nonevent when it is actually an event (also
referred to as a Type II error).
The misclassification plot displays how many observations were correctly and incorrectly classified for each value of the response variable. When the response variable is not binary, the logistic regression model considers all levels that are not events as equal. A significant number of misclassifications could indicate that your model does not fit the data.
Use the Assessment Window
Lift
The default lift chart displays the cumulative lift of the model. To view the noncumulative lift, click
on the vertical axis, and select Lift.
on the vertical axis, and select Lift.
For comparison, the lift chart plots a best model based on complete knowledge of the
input data.
ROC
The specificity of a model is the true negative rate. To derive the false positive rate, subtract the specificity from 1. The false positive rate, labeled 1
– Specificity, is the X axis of the ROC chart. The sensitivity of a model is the true positive rate. This is the Y axis of the ROC chart. Therefore, the ROC chart plots how the true positive rate changes
as the false positive rate changes.
A good ROC chart has a very steep initial slope and levels off quickly. That is, for
each misclassification of an observation, significantly more observations are correctly
classified. For a perfect model, one with no false positives and no false negatives, the ROC chart would start at (0,0), continue vertically to (0,1), and then horizontally
to (1,1). In this instance, the model would correctly classify every observation before
a single misclassification could occur.
The ROC chart includes two lines to help you interpret the ROC chart. The first line
is a baseline model that has a slope of 1. This line mimics a model that correctly classifies observations
at the same rate it incorrectly classifies them. An ideal ROC chart maximizes the
distance between the baseline model and the ROC chart. A model that classifies more observations incorrectly than
correctly would fall below the baseline model. The second line is a vertical line
at the false positive rate where the difference between the Kolmogorov-Smirnov values
for the ROC chart and
baseline models is maximized.
Misclassification
The misclassification plot displays how many observations were correctly and incorrectly
classified. A significant number of misclassifications might indicate that the model
does not fit the data.
When the ratio of events to non-events in your data is relatively large, the misclassification plot might show a large number
of true positives and false positives. In this case, your model predicts most observations
as events and is correct more often than not.
Assessment
When the number of Response
bins is set to more than 10, the Assessment window plots the predicted average and observed average values. Use this plot to
determine how well the model fits the data.
Copyright © SAS Institute Inc. All Rights Reserved.