SUPPORT / SAMPLES & SAS NOTES
 

Support

Usage Note 31821: Testing whether the area under the ROC curve differs from 0.5 (chance)

DetailsAboutRate It

You can test the null hypothesis that the area under an ROC curve (AUC) is 0.5 by comparing the model of interest to an intercept-only model. The intercept-only model has area equal to 0.5 and represents a model that performs no better than chance.

Beginning with SAS® 9.2, additional models can be specified in ROC statements and the ROCCONTRAST statement can be used to make comparisons among the models with respect to the areas under their ROC curves. An ROC statement containing no model specification fits the intercept-only model. An ROCCONTRAST statement with no contrast specification compares each additional model in ROC statements with the reference model in the MODEL statement.

These statements compare the areas under the ROC curves for the given model and the intercept-only model which is equivalent to testing that the area under the given model's ROC curve is 0.5. A plot of each model's ROC curve is also given when ODS graphics is enabled. Notice that the ROC curve for the intercept-only model is the diagonal line from (0,0) to (1,1) with area 0.5.

   data roc;
      input alb tp totscore popind @@;
      totscore = 10 - totscore;
      datalines;
   3.0 5.8 10 0   3.2 6.3  5 1   3.9 6.8  3 1   2.8 4.8  6 0   
   3.2 5.8  3 1   0.9 4.0  5 0   2.5 5.7  8 0   1.6 5.6  5 1   
   3.8 5.7  5 1   3.7 6.7  6 1   3.2 5.4  4 1   3.8 6.6  6 1   
   4.1 6.6  5 1   3.6 5.7  5 1   4.3 7.0  4 1   3.6 6.7  4 0   
   2.3 4.4  6 1   4.2 7.6  4 0   4.0 6.6  6 0   3.5 5.8  6 1   
   3.8 6.8  7 1   3.0 4.7  8 0   4.5 7.4  5 1   3.7 7.4  5 1   
   3.1 6.6  6 1   4.1 8.2  6 1   4.3 7.0  5 1   4.3 6.5  4 1   
   3.2 5.1  5 1   2.6 4.7  6 1   3.3 6.8  6 0   1.7 4.0  7 0   
   3.7 6.1  5 1   3.3 6.3  7 1   4.2 7.7  6 1   3.5 6.2  5 1   
   2.9 5.7  9 0   2.1 4.8  7 1   2.8 6.2  8 0   4.0 7.0  7 1   
   3.3 5.7  6 1   3.7 6.9  5 1   3.6 6.6  5 1   
   ;
   ods graphics on;
   ods html;
   proc logistic data=roc;
      model popind(event='0') = alb;
      roc; roccontrast;
      run;

The following tables provide the area under the ROC curve for the fitted model (0.7366), a 95% confidence interval for the area (0.5549,0.9182), and a test that the area equals 0.5 (p=0.0107).

Prior to SAS 9.2, the areas, confidence intervals, and test can be obtained using the ROC macro.

ROC Association Statistics
ROC Model Mann-Whitney Somers' D
(Gini)
Gamma Tau-a
Area Standard
Error
95% Wald
Confidence Limits
Model 0.7366 0.0927 0.5549 0.9182 0.4731 0.4809 0.1949
ROC1 0.5000 0 0.5000 0.5000 0 . 0

ROC Contrast Test Results
Contrast DF Chi-Square Pr > ChiSq
Reference = Model 1 6.5129 0.0107


Operating System and Release Information

Product FamilyProductSystemSAS Release
ReportedFixed*
SAS SystemSAS/STATOpenVMS VAX
z/OS
Microsoft® Windows® for 64-Bit Itanium-based Systems
Microsoft Windows Server 2003 Datacenter 64-bit Edition
Microsoft Windows Server 2003 Enterprise 64-bit Edition
Microsoft Windows XP 64-bit Edition
Microsoft® Windows® for x64
OS/2
Microsoft Windows 95/98
Microsoft Windows 2000 Advanced Server
Microsoft Windows 2000 Datacenter Server
Microsoft Windows 2000 Server
Microsoft Windows 2000 Professional
Microsoft Windows NT Workstation
Microsoft Windows Server 2003 Datacenter Edition
Microsoft Windows Server 2003 Enterprise Edition
Microsoft Windows Server 2003 Standard Edition
Microsoft Windows XP Professional
Windows Millennium Edition (Me)
Windows Vista
64-bit Enabled AIX
64-bit Enabled HP-UX
64-bit Enabled Solaris
ABI+ for Intel Architecture
AIX
HP-UX
HP-UX IPF
IRIX
Linux
Linux for x64
Linux on Itanium
OpenVMS Alpha
OpenVMS on HP Integrity
Solaris
Solaris for x64
Tru64 UNIX
* For software releases that are not yet generally available, the Fixed Release is the software release in which the problem is planned to be fixed.