Working with the Assessment Window
About the Assessment Window
For a decision tree, the Assessment window plots lift, ROC, and misclassification rates. Use the Assessment window
to determine how well the model fits the data.
Lift is the ratio of the percent of captured responses within each percentile bin to the average percent of responses for the model. Similarly, cumulative lift is calculated by using all of the data up to and including the current percentile
bin.
A receiver operating characteristic (ROC) chart displays the ability of a model to avoid false positive and false negative classifications. A false positive classification means that an observation has been identified as an event when it is actually a nonevent (also referred to as a Type I error). A false negative classification means that
an observation has been identified as a nonevent when it is actually an event (also
referred to as a Type II error).
The misclassification plot displays how many observations were correctly and incorrectly classified for each
value of the response variable. When the response variable is not binary, the logistic regression model considers all levels that are not events as equal. A significant number of misclassifications
could indicate that your model does not fit the data.
Use the Assessment Window
Lift
The default lift chart displays the cumulative lift of the model. To view the noncumulative lift, click
on the vertical axis, and select Lift.
on the vertical axis, and select Lift.
For comparison, the lift chart plots a best model based on complete knowledge of the
input data.
ROC
The specificity of a model is the true negative rate. To derive the false positive rate, subtract the specificity from 1. The false positive rate, labeled 1
– Specificity, is the X axis of the ROC chart. The sensitivity of a model is the true positive rate. This is the Y axis of the ROC chart. Therefore, the ROC chart plots how the true
positive rate changes as the false positive rate changes.
A good ROC chart has a very steep initial slope and levels off quickly. That is, for
each misclassification of an observation, significantly more observations are correctly
classified. For a perfect model, one
with no false positives and no false negatives, the ROC chart would start at (0,0),
continue vertically to (0,1), and then horizontally to (1,1). In this instance, the
model would correctly classify every observation before a single misclassification
could occur.
The ROC chart includes two lines to help you interpret the ROC chart. The first line
is a baseline model that has a slope of 1. This line mimics a model that correctly classifies observations
at the same rate it incorrectly classifies them. An ideal ROC chart maximizes the
distance between the baseline model and the ROC chart. A model that classifies more observations incorrectly than
correctly would fall below the baseline model. The second line is a vertical line
at the false positive rate where the difference between the Kolmogorov-Smirnov values
for the ROC chart and
baseline models is maximized.
Misclassification
The misclassification plot displays how many observations were correctly and incorrectly
classified. A significant
number of misclassifications might indicate that the model does not fit the data.
When the ratio of events to non-events in your data is relatively large, the misclassification
plot might show a large number of true positives and false positives. In this case,
your
model predicts most observations as events and is correct more often than not.
Assessment
When the number of Response bins is set to more than 10, the Assessment window plots the predicted average and observed average values. Use this plot to
determine how well the model fits the data.
The Assessment window
bins the data based on the values specified in the Assessment properties. At each bin, you can hold the mouse over one or both of the lines to
display a tooltip.
Copyright © SAS Institute Inc. All Rights Reserved.
Last updated: January 8, 2019