Providing software solutions since 1976

• Log in
• Create profile
• Welcome
  [Log out]
• Edit profile

• Print  |
• Feedback  |
• Share

  • E-mail

  
  
  

  E-mail This Page

  Your Name:*
  Your E-mail:*
  Recipient E-mail:*
  Send a copy to me
  Personal Message:
  
  

  * indicates a required field

Nonparametric comparison of areas under correlated ROC curves

PURPOSE:

Nonparametric comparison of areas under correlated ROC curves. Provides point and confidence interval estimates of each curve's area and of the pairwise differences among the areas. Tests of the pairwise differences are also given. Any contrast among the areas may be estimated and tested.

HISTORY:

Version	Update Notes
1.7	Fixed row labeling when CONTRAST= option is specified.
1.6	Removed computation of estimated correlations to avoid problems with use of intercept-only model. Fix to automatic check for new version.
1.5	Fixed subscript error in RNAME with 15 or more data sets and XBETA variables. Added automatic check for new version.
1.4	Corrected errors with default contrast and only one variable specified in VAR=. The single area is now estimated and tested.
1.3	Added confidence intervals for each area. Added ALPHA= parameter to control confidence level. Default contrasts are now all pairwise differences. Added table of confidence intervals and tests of each contrast row.
1.2	Put original author's code in macro form. Allow input of several data sets to compare several models which may each have multiple predictors. Default contrast set generated. Added DETAILS= parameter to minimize default output. Creates macro variable with overall p-value.

REQUIREMENTS:

Version 7 or later of Base SAS and SAS/IML.

USAGE:

Follow the instructions in the Downloads tab of this sample to save the %ROC macro definition. Replace the text within quotes in the following statement with the location of the %ROC macro definition file on your system. In your SAS program or in the SAS editor window, specify this statement to define the %ROC macro and make it available for use:

   %inc "<location of your file containing the ROC macro>";

Following this statement, you may call the %ROC macro. See the Results tab for examples.

You must first run the LOGISTIC procedure to fit each of the the models whose ROC curves are to be compared. Each model must be fit to exactly the same data set (with observations in the same order) and they must all be fit to the same response variable. The ROC macro requires that the response variable have only values of 0 and 1 even though this is not a restriction necessary for PROC LOGISTIC. Specify the EVENT="1" option in parentheses following the response variable name in the MODEL statement. Use the XBETA= or PRED= option in the OUTPUT statement in each run of LOGISTIC. The output data set names must be unique. Since the XBETA= or PRED= output variables are used to identify and compare the models' ROC curves, the names must be unique and it is recommended that you select a name for each variable that adequately identifies the fitted model.

The following macro parameters can be specified. The DATA, VAR=, and RESPONSE= parameters are required.

DATA=

Specify the names, separated by spaces, of the OUT= data sets from PROC LOGISTIC to be analyzed. Each data set should contain a variable created by the XBETA= or PRED= option. This parameter is required.

VAR=

The names of the XBETA= or PRED= variables, one from each data set. No name may occur more than once. Separate variable names in the list with spaces. This parameter is required.

RESPONSE=

The name of the response variable which has values of 0 or 1 only. This parameter is required.

CONTRAST=%str(hypothesis matrix)

Specifies the desired hypothesis matrix, L, for testing the null hypothesis L*R=0, where R is the vector or ROC curve areas and the ROC curve areas are assumed to be in the order given in the VAR= variable. The hypothesis matrix, L, must be surrounded by %str(). An L matrix with more than one row may be specified by using commas to separate the rows of coefficients. If omitted, a default contrast testing the equality of all ROC curve areas jointly and pairwise is used.

ALPHA=α

Specifies the level of significance α for 100(1-α)% confidence intervals. The value α must be between 0 and 1. The default value is 0.05 which results in 95% intervals.

DETAILS=YES | NO

Requests additional output. Default is no. Any other value will produce the additional output.

The version of the %ROC macro that you are using is displayed when you specify version (or any string) as the first argument. For example:

    %roc(version, data=a b, var=xba xbb, response=y)

DETAILS:

Details of the methodology are given in DeLong, et. al. (1988). The method is closely related to the jackknife. The method of components is described in chapter 3 of Puri and Sen (1971).

Note that ROC curves from models fit to two or more independent groups of observations are not dependent and therefore cannot be compared using these methods. The SEE ALSO section below refers to an appropriate method for comparing independent curves.

The ROC macro assigns a name to each table it creates. You can use these names to reference the table when using the Output Delivery System (ODS) to select or exclude tables and to create output data sets. The table names are listed in the following table along with any macro parameters required to make the table available. When outputting a table to a data set, you can determine the names of variables in the data set by using PROC CONTENTS.

ODS table name	Description	%ROC macro option
T	Pairwise deletion Mann-Whitney Statistics	DETAILS=yes
V	Estimated Variance Matrix	DETAILS=yes
NX	X populations sample sizes	DETAILS=yes
NY	Y populations sample sizes	DETAILS=yes
L	Coefficients of Contrast	default
LT	Estimates of Contrast	default
LV	Variance Estimates of Contrast	DETAILS=yes
AREASTAB	ROC Area Std Error Confidence Limits	default
DIFFSTAB	Tests and 100(1-α)% Confidence Intervals for Contrast Rows	default
CTEST	Contrast Test Results	default

In addition to displayable output, the ROC macro creates a macro variable, pvalue, containing the p-value from the test of joint equality of ROC curve areas.

Macro Updates

The %ROC macro attempts to check for a later version of itself. If it is unable to do this (such as if there is no active internet connection available), the macro will issue the following message:

   ROC: Unable to check for newer version

The computations performed by the macro are not affected by the appearance of this message.

LIMITATIONS:

The ROC macro cannot do BY-group processing and cannot handle output from PROC LOGISTIC which used BY-group processing. Limited error checking is done. Be sure that the data sets and variables exist and are correctly spelled. Only response values of 0 and 1 are allowed.

MISSING VALUES:

Pairwise deletion of missing values is used prior to computing Mann-Whitney components.

SEE ALSO:

The %ROCPLOT macro can plot an ROC curve with labelled points.

See this note on comparing the areas under independent ROC curves.

REFERENCES:

E.R. DeLong, D.M. DeLong, and D.L. Clarke-Pearson (1988), "Comparing the Areas Under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach," Biometrics, 44, 837-845.

Puri M.L and Sen P.K. (1971), Nonparametric Methods in Multivariate Analysis, Wiley.

---

These sample files and code examples are provided by SAS Institute Inc. "as is" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Recipients acknowledge and agree that SAS Institute shall not be liable for any damages whatsoever arising out of their use of this material. In addition, SAS Institute will provide no support for the materials contained herein.

---

These sample files and code examples are provided by SAS Institute Inc. "as is" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Recipients acknowledge and agree that SAS Institute shall not be liable for any damages whatsoever arising out of their use of this material. In addition, SAS Institute will provide no support for the materials contained herein.

EXAMPLE 1: Comparing ROC curves of three diagnostics

From DeLong et. al. (1988). Three prognostic indices of post-operative success are compared.

Each ROC curve is plotted separately with labels and a final joint plot of all curves is produced. Statistical tests of the equality of the areas under all three curves and pairwise comparisons among the curves are given along with point and confidence interval estimates.

Before running this example, be sure to define both the %ROC and %ROCPLOT macros.

   /* Define the ROC macro */
   %include "<path to your copy of the ROC macro>";
   
   /* Define the ROCPLOT macro */
   %include "<path to your copy of the ROCPLOT macro>";
   
   data roc;
     input alb tp totscore popind;
     totscore = 10 - totscore;
     datalines;
   3.0 5.8 10 0
   3.2 6.3  5 1
   3.9 6.8  3 1
   2.8 4.8  6 0
   3.2 5.8  3 1
   0.9 4.0  5 0
   2.5 5.7  8 0
   1.6 5.6  5 1
   3.8 5.7  5 1
   3.7 6.7  6 1
    .   .   6 1
   3.2 5.4  4 1
   3.8 6.6  6 1
   4.1 6.6  5 1
   3.6 5.7  5 1
   4.3 7.0  4 1
   3.6 6.7  4 0
   2.3 4.4  6 1
   4.2 7.6  4 0
   4.0 6.6  6 0
   3.5 5.8  6 1
   3.8 6.8  7 1
   3.0 4.7  8 0
   4.5 7.4  5 1
   3.7 7.4  5 1
   3.1 6.6  6 1
   4.1 8.2  6 1
   4.3 7.0  5 1
   4.3 6.5  4 1
   3.2 5.1  5 1
   2.6 4.7  6 1
   3.3 6.8  6 0
   1.7 4.0  7 0
    .   .   6 1
   3.7 6.1  5 1
   3.3 6.3  7 1
   4.2 7.7  6 1
   3.5 6.2  5 1
   2.9 5.7  9 0
   2.1 4.8  7 1
    .   .   8 1
   2.8 6.2  8 0
    .   .   7 1
    .   .   7 1
   4.0 7.0  7 1
   3.3 5.7  6 1
   3.7 6.9  5 1
   2.0  .   7 1
   3.6 6.6  5 1
   ;
   
   /* Albumin */
     title "ROC plot for Albumin";
     proc logistic data=roc;
       model popind(event='1') = alb / outroc=or roceps=0;
       output out=albpred p=palb;
       ods output association=assoc;
       run;
   
     data _null_;
        set assoc;
        if label2='c' then call symput("area",cvalue2);
        run;
   
     title2 "Approximate area under curve = &area";
     %rocplot(out=albpred, outroc=or, p=palb, id=alb)
     title2;
   
     data joint;
       set _rocplot;
       length index $ 13;
       Index='Albumin';
       run;
   
   /* Total Protein */
     title "ROC plot for Total Protein";
     proc logistic data=roc;
       model popind(event='1') = tp / outroc=or roceps=0;
       output out=tppred p=ptp;
       ods output association=assoc;
       run;
   
     data _null_;
        set assoc;
        if label2='c' then call symput("area",cvalue2);
        run;
   
     title2 "Approximate area under curve = &area";
     %rocplot(out=tppred, outroc=or, p=ptp, id=tp)
     title2;
   
     data tp;
       set _rocplot;
       length index $ 13;
       Index='Total Protein';
       run;
   
     data joint;
       set joint tp;
       run;
   
   /* K-G score */
     title "ROC plot for K-G score";
     proc logistic data=roc;
       model popind(event='1') = totscore / outroc=or roceps=0;
       output out=totspred p=ptots;
       ods output association=assoc;
       run;
   
     data _null_;
        set assoc;
        if label2='c' then call symput("area",cvalue2);
        run;
   
     title2 "Approximate area under curve = &area";
     %rocplot(out=totspred, outroc=or, p=ptots, id=totscore)
     title2;
   
     data kg;
       set _rocplot;
       length index $ 13;
       Index='K-G Score';
       run;
   
     data joint;
       set joint kg;
       run;
   
   /* Compare areas under the ROC curves of the indices using the method presented
      in DeLong, et. al. (1988).
   */
     %roc(data=albpred tppred totspred,
          var=palb ptp ptots,
          response=popind)
   
   /* Plot all indices and p-value of overall test comparing areas */
     symbol1 i=join v=circle c=blue line=33;
     symbol2 i=join v=dot c=green line=1;
     symbol3 i=join v=triangle c=red line=4;
   
     proc gplot data=joint;
       title  "Post-Op Indices of surgical success";
       title2 "Test of H0: equal areas under curves -- p=&pvalue";
       footnote "From DeLong, et. al. (1988, Biometrics 44)";
       label index="Index";
       plot _sensit_ * _1mspec_ = Index /
          vaxis=0 to 1 by .1 haxis=0 to 1 by .1;
       run;
       quit;
     title;
     title2;
     footnote;

The analysis and plots can be done in one PROC LOGISTIC step using ROC and ROCCONTRAST statements.

     ods graphics on;
     proc logistic data=roc;
       model popind(event='1') = alb tp totscore / nofit;
       roc 'alb' alb;
       roc 'tp' tp;
       roc 'totscore' totscore;
       roccontrast / estimate=allpairs;
       run;

RESULTS:

The default set of contrasts compares the three indices. The large p-value (p=.4423) indicates no significant advantage of the scoring system. The results vary somewhat from the Delong et.al. (1988) paper because this program computes components based on pairwise deletion of missing observations, whereas the original analysis used pairwise deletion after the components were calculated. The current strategy is felt to be more conservative with respect to possible association of response with the probability of the response not being observed.

For brevity, the PROC LOGISTIC results are not shown. The following plots are generated by the %ROCPLOT macro:

• ROC plot for Albumin (alb)
• ROC plot for Total Protein (tp)
• ROC plot for K-G score (totscore)
• Comparative ROC plot

Following are the results of the %ROC macro.

The ROC Macro   ROC Curve Areas and 95% Confidence Intervals   ROC Area Std Error Confidence Limits palb 0.7188 0.0928 0.5368 0.9007 ptp 0.6478 0.1000 0.4518 0.8439 ptots 0.6869 0.1051 0.4809 0.8929 Contrast Coefficients   palb ptp ptots Row1 1 -1 0 Row2 1 0 -1 Row3 0 1 -1 Tests and 95% Confidence Intervals for Contrast Rows   Estimate Std Error Confidence Limits Chi-square Pr > ChiSq Row1 0.0709 0.0560 -0.0389 0.1807 1.6017 0.2057 Row2 0.0318 0.0980 -0.1603 0.2240 0.1053 0.7456 Row3 -0.0391 0.1069 -0.2485 0.1703 0.1338 0.7145 Contrast Test Results Chi-Square DF Pr > ChiSq 1.6314 2 0.4423

EXAMPLE 2: Comparing three competing models

The data are from an example in Logistic Regression Using the SAS System: Theory and Application by Paul Allison (book number 55770) in which students where asked if they would return a lost wallet and contents if found. The response, WALLET, indicates if the wallet and contents would be returned (WALLET=1) or kept (WALLET=0). Predictors are gender, whether in business school, parental punishment at various ages, and parental explanation for punishment.

Three models are fit and their ROC curve areas tested for equality. A significant finding is followed by pairwise comparisons.

   data wallet;
     input wallet male business punish explain @@;
     if wallet in (1,2) then wallet=0;
     if wallet=3 then wallet=1;
     datalines;
   2 0 0 2 0   3 0 0 1 1   3 1 0 1 1   2 0 1 1 1   3 1 0 1 1
   2 0 0 2 1   3 0 0 1 1   3 0 0 1 1   1 0 0 3 0   2 1 0 1 0
   3 0 0 1 1   3 0 0 1 1   3 0 1 1 1   1 0 1 2 0   3 0 1 2 1
   3 0 0 2 0   3 0 0 2 0   3 0 0 1 1   3 0 0 1 1   2 1 1 2 1
   1 1 0 1 1   3 1 1 1 1   3 0 0 1 1   3 0 0 1 1   3 0 0 1 1
   3 0 0 1 1   3 1 0 1 1   3 0 0 2 1   3 0 0 1 1   1 1 0 1 1
   3 0 0 1 1   3 0 0 1 1   2 1 1 1 0   3 1 1 1 1   3 1 1 3 1
   3 1 0 1 1   3 0 0 1 1   3 1 0 1 1   3 0 0 1 0   2 1 0 1 1
   3 1 0 1 1   3 0 0 1 0   3 0 0 1 1   3 0 0 1 1   1 0 0 1 0
   3 0 0 2 1   3 0 1 3 0   2 1 1 1 1   3 1 0 1 1   2 1 0 3 0
   2 0 1 1 1   3 1 0 2 0   3 0 0 1 1   3 1 1 1 1   3 0 0 1 1
   3 1 1 1 1   2 0 0 2 1   3 1 1 1 1   3 1 0 1 1   2 1 1 1 1
   3 1 0 1 1   3 0 0 1 1   3 0 0 1 1   2 1 1 1 1   3 0 0 1 1
   3 1 0 1 1   3 1 0 1 1   3 0 0 1 1   2 0 0 1 0   2 0 0 2 0
   3 0 0 1 1   3 1 0 1 1   3 0 0 1 1   3 0 0 1 1   2 1 0 1 1
   3 0 0 1 0   3 1 0 1 0   3 0 1 1 1   2 1 1 2 0   1 1 1 2 0
   3 0 0 2 1   3 1 1 1 0   3 0 0 2 1   3 1 0 1 0   2 1 0 2 0
   2 0 0 3 0   3 1 0 1 0   1 1 1 1 1   2 1 0 1 0   2 0 0 1 1
   1 1 1 3 0   2 1 1 3 1   1 1 0 2 0   3 0 0 2 1   1 0 1 3 0
   2 0 0 1 1   3 0 0 1 1   2 1 0 1 1   3 1 1 1 1   2 1 0 1 1
   2 1 0 2 0   3 0 0 3 1   1 1 1 1 1   1 0 0 3 0   2 1 0 1 1
   3 0 0 1 1   3 0 0 1 1   2 1 0 1 1   3 0 0 1 1   3 1 0 1 1
   3 1 0 1 1   3 0 0 1 0   1 1 1 3 0   3 1 1 1 1   3 0 0 1 1
   3 1 1 1 0   3 0 0 1 1   2 1 1 1 1   3 0 0 1 0   2 1 0 1 1
   3 1 0 1 1   1 0 1 1 1   1 1 1 3 1   3 0 1 1 0   1 1 0 2 0
   2 0 0 1 0   3 0 0 1 0   3 0 0 3 1   2 0 0 1 1   2 0 1 2 1
   2 1 0 3 0   3 1 0 1 1   3 0 0 1 1   3 1 0 1 1   3 0 0 2 0
   1 1 0 2 0   1 1 0 1 0   2 0 0 1 1   2 1 0 1 0   2 1 1 1 1
   3 1 0 2 0   3 1 0 3 1   3 0 0 1 1   3 0 0 1 0   3 0 1 2 1
   3 1 0 1 1   2 1 0 3 1   1 0 1 3 0   2 1 0 1 1   1 0 0 3 0
   3 0 0 1 1   1 1 1 2 1   3 0 0 1 1   3 1 0 1 1   2 1 1 1 0
   3 1 0 2 1   1 1 0 2 1   2 0 0 1 0   3 1 0 2 0   3 0 0 3 1
   1 1 0 1 1   3 0 0 1 1   3 0 0 1 1   3 1 1 2 1   2 1 0 2 1
   3 0 0 2 0   3 0 0 3 0   3 0 0 1 1   3 1 0 1 1   3 0 0 1 1
   3 1 0 1 0   2 1 0 1 1   2 1 0 1 1   2 1 0 1 0   3 1 0 3 1
   3 1 1 2 1   2 0 0 1 1   3 1 0 1 1   3 0 0 1 1   3 0 0 1 1
   3 0 0 2 0   2 1 0 1 1   3 0 0 1 1   3 1 0 1 1   3 1 1 1 1
   3 0 0 1 1   3 0 0 1 0   1 1 0 3 0   2 1 1 1 0   3 1 0 1 1
   3 1 0 1 1   3 1 1 1 1   2 1 0 1 1   3 0 0 1 1   2 1 0 1 1
   ;
   /* Fit three models to the Pr(return wallet) */
   proc logistic data=wallet;
     model wallet(event="1") = male business punish explain;
     output out=w1 xbeta=mbpe;
     run;
   
   proc logistic data=wallet;
     model wallet(event="1") = business punish explain;
     output out=w2 xbeta=bpe;
     run;
   
   proc logistic data=wallet;
     model wallet(event="1") = business punish ;
     output out=w3 xbeta=bp;
     run;
   
   /* Test equality of the three ROC curve areas. */
   %roc(data=w1 w2 w3,
        response=wallet,
        var=mbpe bpe bp)

The same analysis can be done in one PROC LOGISTIC step using ROC and ROCCONTRAST statements.

   proc logistic data=wallet;
     mbpe: model wallet(event="1") = male business punish explain;
     roc 'bpe' business punish explain;
     roc 'bp' business punish;
     roccontrast / estimate=allpairs;
     run;

RESULTS:

Following are the results from the %ROC macro. The overall p-value (p=.0281) indicates that the three areas differ. The default pairwise comparisons show that this difference is due to a significant difference between the first and third models.

The ROC Macro   ROC Curve Areas and 95% Confidence Intervals   ROC Area Std Error Confidence Limits mbpe 0.7412 0.0355 0.6717 0.8107 bpe 0.6873 0.0383 0.6122 0.7624 bp 0.6508 0.0379 0.5766 0.7250 Contrast Coefficients   mbpe bpe bp Row1 1 -1 0 Row2 1 0 -1 Row3 0 1 -1 Tests and 95% Confidence Intervals for Contrast Rows   Estimate Std Error Confidence Limits Chi-square Pr > ChiSq Row1 0.0539 0.0291 -0.0032 0.1110 3.4215 0.0644 Row2 0.0904 0.0340 0.0239 0.1570 7.0884 0.0078 Row3 0.0365 0.0276 -0.0175 0.0906 1.7530 0.1855 Contrast Test Results Chi-Square DF Pr > ChiSq 7.1417 2 0.0281

EXAMPLE 3: Testing the area under the ROC curve

You can test the null hypothesis that the area under an ROC curve is 0.5 by comparing the model of interest to an intercept-only model. The intercept-only model has area equal to 0.5, so the default contrast (1 -1) performs this test. These additional statements test the area of the last model in the previous example. The PROC LOGISTIC step fits the intercept-only model.

   proc logistic data=wallet;
     model wallet(event="1") =  ;
     output out=w4 xbeta=int;
     run;
   
   %roc(data=w3 w4,
        response=wallet,
        var=bp int)

The same analysis can be accomplished in PROC LOGISTIC using ROC and ROCCONTRAST statements as illustrated in this note.

RESULTS:

The test shows that the area under the ROC curve for this model is significantly different from 0.5 (p<0.0001).

The ROC Macro   ROC Curve Areas and 95% Confidence Intervals   ROC Area Std Error Confidence Limits bp 0.6508 0.0379 0.5766 0.7250 int 0.5000 0.0000 0.5000 0.5000 Contrast Coefficients   bp int Row1 1 -1 Tests and 95% Confidence Intervals for Contrast Rows   Estimate Std Error Confidence Limits Chi-square Pr > ChiSq Row1 0.1508 0.0379 0.0766 0.2250 15.8648 <.0001 Contrast Test Results Chi-Square DF Pr > ChiSq 15.8648 1 <.0001

Right-click on the link below and select Save to save the %ROC macro definition to a file. It is recommended that you name the file roc.sas.

Download and save roc.sas

Operating System and Release Information