Sample 25018: Plot ROC curve with labelled points for a binary-response model
 |  |  |  |  |  |
Plot ROC curve with labelled points for a binary-response model
| Contents: | Purpose / History / Requirements / Usage / Details / Limitations / See Also / References |
NOTE: Beginning in SAS 9.1, you can use ODS graphics to create a plot of the ROC curve, but points cannot be labeled. Beginning in SAS 9.2, you can use the PLOTS=ROC(ID=OBS|PROB) option to label points with their observation number or predicted probability. See the LOGISTIC documentation.
- PURPOSE:
- Produce a plot showing the Receiver Operating Characteristic (ROC) curve associated with a fitted binary-response model. This is a plot of the sensitivity against 1-specificity values associated with the observations' predicted event probabilities.
- HISTORY:
-
Version Update Notes 1.1 - Added automatic check for new version.
- Added GRID= option to display a grid in the plot.
- Added version option to display current macro version.
- Added special variables for sensitivity, specificity, and observation number for use as ID= variables.
- Added some white space at both ends of each axis (controlled by ROFFSET=).
- Fixed observations with missing probabilities causing "no match" message.
- Added (0,0) point to plot to complete the ROC curve.
- For low-resolution plots, added second plot without labels to make it easier to see the shape of the curve.
- Fixed multiple observations with same cutoff values from causing "no match" message. The message should only occur if cutoffs don't match in the OUT= and OUTROC= data sets, presumably because the ROCEPS=0 wasn't used as suggested in the message.
- Removed the position option. Macro controls positioning to avoid overlaps.
- Labels are now angled to improve clarity. Labels are vertical at the right edge and almost horizontal at the left edge.
- To reduce label clutter, only top and bottom points in a vertical stack are labeled. Label of top point is above-left. Label of bottom point is below-right.
- If two points have specificity values that differ by less than the MINDIST= value, one of the points will not be labeled. Points more than the MINDIST= value apart are labeled.
1.0 Initial coding. - REQUIREMENTS:
- SAS/STAT software, Release 6.12 or later, is needed to create the required input data sets from PROC LOGISTIC. The macro requires only Base SAS software to generate low-resolution plots via PROC PLOT. High-resolution plots require PROC GPLOT in SAS/GRAPH software.
- USAGE:
-
Before using the %ROCPLOT macro, you must first run the LOGISTIC procedure to fit the desired model
using the OUTROC= and ROCEPS=0 options in the MODEL statement
and the OUT= and P= options in the OUTPUT statement.
Follow the instructions in the Downloads tab of this sample to save the %ROCPLOT macro definition. Replace the text within quotes in the following statement with the location of the %ROCPLOT macro definition file on your system. In your SAS program or in the SAS editor window, specify this statement to define the %ROCPLOT macro and make it available for use:
%inc "<location of your file containing the ROCPLOT macro>";
Following this statement, you may call the %ROCPLOT macro. See the Results tab for an example.
The following parameters are required when using the macro:
- out=data-set-name
- Name of the OUT= data set from PROC LOGISTIC.
- outroc=data-set-name
- Name of the OUTROC= data set from PROC LOGISTIC.
- p=variable
- Name of the variable specified in the P= option in the OUTPUT statement of PROC LOGISTIC. This variable contains the predicted probabilities.
- id=variable list
-
Names of the variables to be used to label the
points on the ROC curve. At least one variable must be specified.
If additional variables are specified, they must be separated by spaces.
Note that the point
labels may become too long to be fully displayed if you specify
many variables. You may specify any of
the predictor variables from the fitted model. You may also
specify any variables in the PROC LOGISTIC input (DATA=) data
set, the OUTPUT OUT= data set (such as
the predicted probabilities), or the OUTROC= data set (such as
the counts of false positives, _FALPOS_, or false negatives,
_FALNEG_). Additionally, you can specify any of these variables:
- _OBS_ , the observation number.
- _SENS_ , the sensitivity in 5.3 format (for example, 0.138).
- _SPEC_ , the sensitivity in 5.3 format.
The following are optional parameters for controlling the graphical appearance:
- plottype=high|low
- PLOTTYPE=HIGH (the default) produces a high- resolution ROC plot using PROC GPLOT. PLOTTYPE=LOW produces a low-resolution plot using PROC PLOT.
- font=font-name
- Font of label text. Default: swissb.
- grid=yes|no
- Specifies whether a set of gray grid lines, at 0.1 intervals, should be added to the plot. Default: no.
- size=value
- Height of the label text. Default: 3 (3% of the graphics output area).
- color=color-name
- Color of the label text. Default: black.
- plotchar=char
- Symbol for points on the ROC curve. Default: dot.
- roffset=value
- Specifies the amount of white space (in percent of graphic area) at both ends of both axes. Default: 4.
- round=value
- Specifies what predicted values (in the OUTPUT OUT= data set) and the cutoff values (in the OUTROC= data set) should be rounded to. After rounding, any observations with duplicate values are discarded. Increasing this value can be used to reduce the number of points in the plot. Default: 1e-8.
- mindist=value
- If two points have specificity values that differ by less than value, one of the points will not be labeled. Points more than value apart are labeled. Default: 0.02.
The version of the %ROCPLOT macro that you are using is displayed when you specify version (or any string) as the first argument. For example:
%ROCPLOT(version, out=logout, outroc=logor, p=p_hat, id=x1 x2) - DETAILS:
-
The receiver operating characteristic (ROC) curve is a diagnostic
tool for assessing the ability of a logistic model to discriminate
between events and nonevents. If the model discriminates perfectly,
the ROC curve passes through the point (0,1) and the area below the
curve is one. If the model has no discriminating ability, the curve
is a line from (0,0) to (1,1). Each point on the ROC curve provides
the sensitivity and specificity measures associated with a cutoff
on the probability scale which allows classification of each
observation as either a predicted event or a predicted nonevent.
It is important to use the ROCEPS=0 option in the MODEL statement of PROC LOGISTIC when you fit your model because this option allows all the unique predicted values to be output to the OUTROC= data set. Otherwise, the values may be rounded yielding fewer points on the ROC plot.
If several points have the same specificity value, only the top- and bottom-most points are labeled, and only if they are sufficiently separated from other labels (as controlled by the MINDIST= option).
If multiple predictor settings occur at same point, there is not indication of this in a high-resolution plot (PLOTTYPE=HIGH). However, low-resolution plots (PLOTTYPE=LOW) use characters which indicate the number of multiple settings at a point.
An ROC plot for a large data set may have many points with labels that overlap depending on the values of the SIZE= and MINDIST= options. In the worst cases, a high density of overlapping labels may form a big black blob. You should be able to reduce the number of labeled points and produce a reasonable plot using one or more of these:
- Increase MINDIST= to reduce the number of points that are labelled.
- Decrease the size of the labeling font using the SIZE= option. For example, SIZE=1.5 produces fairly small text which would reduce label overlap.
- Increase the ROUND= value to reduce the number of points in plot.
The _ROCPLOT data set generated by the macro contains all the information needed to produce the ROC plot. The _ID variable in this data set contains the label for the points on the plot.
The %ROCPLOT macro attempts to check for a later version of itself. If it is unable to do this (such as if there is no active internet connection available), the macro will issue the following message:
ROCPLOT: Unable to check for newer version
The computations performed by the macro are not affected by the appearance of this message.
- LIMITATIONS:
-
Limited error checking is done. Be sure that the specified P= and ID=
variables exist in the OUT= data set.
This message:
ROCPLOT: Some predicted values in OUT= did not match predicted values in OUTROC=. Verify that you used the ROCEPS=0 option in PROC LOGISTIC.may appear in the SAS Log if the list of distinct predicted probabilities in the OUTROC= data set is shortened because the ROCEPS=0 option was not used in PROC LOGISTIC, or if, for some other reason, the number of distinct predicted probabilities in the OUTPUT OUT= data set differs from the number of distinct cutoff values in the OUTROC= data set.
This message does not indicate a problem, but the message is issued to remind you to use the ROCEPS=0 option in case that was the cause.
Note that observations with different settings of the ID= variables may have the same predicted probability, but only one of the settings labels the point on the ROC curve associated with that predicted probability.
Also, two (or more) points on the ROC curve may have the same label because they have the same values of the ID= variables but different predicted probabilities. For example, this may happen if a subset of the predictors is specified in ID= to label the points, and points having the same values of the predictors in ID= differ on one or more of the predictors not in ID=.
- SEE ALSO:
-
The %ROC macro performs nonparametric comparison of areas under correlated ROC curves. It provides point and confidence interval estimates of each curve's area and of the pairwise differences among the areas. Tests of the pairwise differences are also given. Any contrast among the areas may be estimated and tested.
An additional example using the %ROCPLOT macro can be seen in %ROC macro examples.
- REFERENCES:
-
DeLong, E.R., DeLong, D.M. and Clarke-Pearson, D.L. (1988),
"Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach,"
Biometrics, 44, 837-845.
Hanley, J.A. and McNeil, B.J. (1982), "The meaning and use of the area under a receiver operating characteristic (ROC) curve," Radiology, 143, 29-36.
Hanley, J.A. and McNeil, B.J. (1983), "A method of comparing the areas under receiver operating characteristic curves derived from the same cases," Radiology, 148, 839-843.
Metz, C.E. (1978), "Basic principles of ROC analysis," Seminars in Nuclear Medicine, 8(4), 283-298.
An additional example in the Results tab of the ROC macro description shows a comparative plot of several ROC curves and a test of the equality of the areas below the curves.
- EXAMPLE 1: Single Predictor; ROC area displayed in title
- This example appears in the LOGISTIC documentation modeling the probability of disease as a function of age. Note that the c statistic reported by PROC LOGISTIC, also known as the concordance index, is the area under the ROC curve. The ODS OUTPUT statement and subsequent DATA step are used to obtain the area under the ROC curve and store it in a macro variable for use in the title of the ROC plot.
data Data1; input disease n age; datalines; 0 14 25 0 20 35 0 19 45 7 18 55 6 12 65 17 17 75 ; ods select parameterestimates association; proc logistic data=data1; model disease/n = age / outroc=roc1 roceps=0; output out=outp p=phat; ods output association=assoc; run; data _null_; set assoc; if label2='c' then call symput("area",cvalue2); run; - RESULTS:
-
Following are the results from PROC LOGISTIC. Note that the area under the ROC curve is 0.953.
The first call of the ROCPLOT macro plots the ROC curve labeling points by the corresponding value of AGE and displays a set of grid lines.
* Define the ROCPLOT macro ;
%inc "<location of your file containing the ROCPLOT macro>";
title "ROC plot for disease = age";
title2 "Approximate area under curve = &area";
%rocplot(outroc = roc1,
out = outp,
p = phat,
id = age,
grid = yes)
|
The second ROCPLOT call labels the points on the ROC curve with their associated sensitivity, specificity, and cutoff value.
data outp;
set outp;
pfmt=put(phat,6.4);
run;
%rocplot(outroc = roc1,
out = outp,
p = phat,
id = _sens_ _spec_ pfmt,
grid = yes)
|
- EXAMPLE 2: Multiple Predictors
- Using the remission data set from the LOGISTIC documentation, the following statements fit a logistic model with SMEAR and BLAST as predictors.
data Remission; input remiss cell smear infil li blast temp; datalines; 1 .8 .83 .66 1.9 1.1 .996 1 .9 .36 .32 1.4 .74 .992 0 .8 .88 .7 .8 .176 .982 0 1 .87 .87 .7 1.053 .986 1 .9 .75 .68 1.3 .519 .98 0 1 .65 .65 .6 .519 .982 1 .95 .97 .92 1 1.23 .992 0 .95 .87 .83 1.9 1.354 1.02 0 1 .45 .45 .8 .322 .999 0 .95 .36 .34 .5 0 1.038 0 .85 .39 .33 .7 .279 .988 0 .7 .76 .53 1.2 .146 .982 0 .8 .46 .37 .4 .38 1.006 0 .2 .39 .08 .8 .114 .99 0 1 .9 .9 1.1 1.037 .99 1 1 .84 .84 1.9 2.064 1.02 0 .65 .42 .27 .5 .114 1.014 0 1 .75 .75 1 1.322 1.004 0 .5 .44 .22 .6 .114 .99 1 1 .63 .63 1.1 1.072 .986 0 1 .33 .33 .4 .176 1.01 0 .9 .93 .84 .6 1.591 1.02 1 1 .58 .58 1 .531 1.002 0 .95 .32 .3 1.6 .886 .988 1 1 .6 .6 1.7 .964 .99 1 1 .69 .69 .9 .398 .986 0 1 .73 .73 .7 .398 .986 ; ods select parameterestimates association; proc logistic data=Remission; model remiss = smear blast / outroc=roc1 roceps=0; output out=outp p=p; run; - RESULTS:
- Following are the results from fitting the model in PROC LOGISTIC. The area under the ROC curve is 0.753.
|
The LOGISTIC Procedure
| ||||||||||||||||||||||||||||||||||||||||||||||||||
The ROC plot produced by the following ROCPLOT macro call labels the points with the values of the SMEAR and BLAST predictors corresponding to each cutoff.
* Define the ROCPLOT macro ;
%inc "<location of your file containing the ROCPLOT macro>";
title "ROC plot for remiss = smear blast";
title2 " ";
%rocplot(outroc = roc1,
out = outp,
p = p,
id = smear blast,
roffset = 10,
grid = yes)
Note that if the model has multiple predictors, it is possible for multiple settings of the predictors to have the same predicted probability. In such cases, the label used in the plot is just one of these settings.
|
- EXAMPLE 3: Controlling point labeling
- In his book, Logistic Regression Using The SAS System, Allison presents a data set of 316 people who were victims of residential fires. They were assessed at three times and their post-traumatic stress status and number of problems in various aspects of life were recorded. Using the data from the first assessment in time, the following statements model the probability of post-traumatic stress symptoms as a function of the number of problems and produce an ROC curve.
ods select parameterestimates association; proc logistic data=ptsd; where time=1; model ptsd(event="1") = problems / outroc=or roceps=0; output out=out p=p; run; - RESULTS:
- The results of fitting the model indicate that the number of problems is significantly associated with the probability of post-traumatic stress symptoms such that the probability increases as the number of problems increases. The area under the ROC curve is 0.694.
|
The LOGISTIC Procedure
| ||||||||||||||||||||||||||||||||||||||||||||
The following produces a plot of the ROC curve for the model with it's points labelled by the number of problems. Notice that labels for several closely-spaced points in the upper-right of the plot are omitted to avoid overlap.
title "Model PTSD = PROBLEMS";
%rocplot(out=out, outroc=or, p=p, id=problems)
|
Decreasing the MINDIST= value increases the number of labels in the plot, but this causes some overlapping of labels.
%rocplot(out=out, outroc=or, p=p, id=problems, mindist=.005)
|
By rounding the cutoff values to the nearest tenth, the number of points on the ROC plot is greatly reduced. The labeling of points may change due to the rounding. For instance, the point labelled problems=6.75 in the plot above is labelled 6.125 in the plot below. This is because rounding effectively forms groups of points and the first point in the group containing the 6.75 point is a point with problems=6.125. It is the label of this first point in the group that appears in the plot below.
%rocplot(out=out, outroc=or, p=p, id=problems, round=.1)
|
Right-click on the link below and select Save to save
the %ROCPLOT macro definition
to a file. It is recommended that you name the file
rocplot.sas.
| Type: | Sample |
| Topic: | SAS Reference ==> Procedures ==> PLOT SAS Reference ==> Procedures ==> LOGISTIC SAS Reference ==> Procedures ==> GPLOT Analytics ==> Categorical Data Analysis Analytics ==> Regression |
| Date Modified: | 2008-01-28 10:55:21 |
| Date Created: | 2005-01-13 15:03:46 |
Operating System and Release Information
| Product Family | Product | Host | SAS Release | |
| Starting | Ending | |||
| SAS System | Base SAS | All | 6.12 | n/a |
| SAS System | SAS/STAT | All | 6.12 | n/a |
| SAS System | SAS/GRAPH | All | 6.12 | n/a |