Contents

Preface............................................................. vii

Acknowledgments............................................. ix

Chapter 1 Introduction........................................... 1

1.1 About Receiver Operating Characteristic Curves.............. 1

1.2 Summary of Chapters........................................................... 3

Chapter 2 Single Binary Predictor......................... 5

2.1 Introduction. 5

2.2 Frost Forecast Example. 5

2.3 Misclassification Rate. 6

2.4 Sensitivity and Specificity. 7

2.5 Computations Using PROC FREQ.. 8

Chapter 3 Single Continuous Predictor............... 15

3.1 Dichotomizing a Continuous Predictor 15

3.2 The ROC Curve. 18

3.3 Empirical ROC Curve and the Conditional Distributions
of the Marker 20

3.4 Area under the ROC Curve. 21

3.5 Selecting an Optimal Threshold. 25

3.6 The Binormal ROC Curve. 26

3.7 Transformations to Binormality. 30

3.8 Direct Estimation of the Binormal ROC Curve. 32

3.9 Bootstrap Confidence Intervals for the Area under
the Curve. 33

Chapter 4 Comparison and Covariate
Adjustment of ROC Curves................... 37

4.1 Introduction......................................................................... 37

4.2 An Example from Prostate Cancer Prognosis. 38

4.3 Paired versus Unpaired Comparisons. 38

4.4 Comparing the Areas under the Empirical ROC Curves. 39

4.5 Comparing the Binormal ROC Curves. 42

4.6 Discrepancy between Binormal and Empirical
ROC Curves. 46

4.7 Bootstrap Confidence Intervals for the Difference
in the Area under the Empirical ROC Curve. 48

4.8 Covariate Adjustment for ROC Curves. 48

4.9 Regression Model for the Binormal ROC Curve. 49

Chapter 5 Ordinal Predictors............................... 53

5.1 Introduction......................................................................... 53

5.2 Credit Rating Example. 54

5.3 Empirical ROC Curve for Ordinal Predictors. 54

5.4 Area under the Empirical ROC Curve. 56

5.5 Latent Variable Model 57

5.6 Comparing ROC Curves for Ordinal Markers. 63

Chapter 6 Lehmann Family of ROC Curves.......... 67

6.1 Introduction. 67

6.2 Lehmann Family of Distributions. 67

6.3 Magnetic Resonance Example. 70

6.4 Adjusting for Covariates. 73

6.5 Using Estimating Equations to Handle Clustered Data. 75

6.6 Comparing Markers Using the Lehmann Family of
ROC Curves. 79

6.7 Advantages and Disadvantages of the Lehmann
Family of ROC Curves. 80

Chapter 7 ROC Curves with Censored Data........ 81

7.1 Introduction. 81

7.2 Lung Cancer Example. 81

7.3 ROC Curves with Censored Data. 82

7.4 Concordance Probability with Censored Data. 88

7.5 Concordance Probability and the Cox Model 91

Chapter 8 Using the ROC Curve to Evaluate
Multivariable Prediction Models......... 95

8.1 Introduction......................................................................... 95

8.2 Liver Surgery Example. 96

8.3 Resubstitution Estimate of the ROC Curve. 97

8.4 Split-Sample Estimates of the ROC Curve. 99

8.5 Cross-Validation Estimates of the ROC Curve. 102

8.6 Bootstrap-Validated Estimates of the ROC Curve. 106

Chapter 9 ROC Curves in SAS Enterprise
Miner.................................................. 109

9.1 Introduction....................................................................... 109

9.2 Home Equity Loan Example. 110

9.3 ROC Curves from SAS Enterprise Miner
for a Single Model 111

9.4 ROC Curves from SAS Enterprise Miner
for Competing Models. 113

9.5 ROC Curves Using PROC GPLOT with Exported
Data from SAS Enterprise Miner 116

Appendix An Introduction to PROC NLMIXED.. 119

A.1 Fitting a Simple Linear Model: PROC GLM vs
PROC NLMIXED.. 119

A.2 PROC NLMIXED and the Binormal Model 124

References............................................................ 127

Index...................................................................... 129