Multiple Comparisons and Multiple Tests Using the SAS System

Chapter 1 Introduction 1

• 1.1 The Multiplicity Problem
• 1.2 Examples of Multiplicity in Practice
• 1.2.1 Multiple Comparisons in a Marketing Experiment
• 1.2.2 Subgroup Analysis in a Clinical Trial
• 1.2.3 Analysis of a Sociological Survey
• 1.2.4 An Epidemiology Example: Data Snooping
• 1.2.5 Industrial Experimentation and Engineering
• 1.2.6 Identifying Clinical Practice Improvement Opportunities for Hospital Surgeries
• 1.3 When Are Multiple Comparisons/Multiple Testing Methods (MCPs) Needed?
• 1.4 Selecting an MCP Using This Book
• 1.4.1 Statistical Modeling Assumptions
• 1.4.2 Multiple Comparisons/Multiple Testing Objectives
• 1.4.3 The Set (Family) of Elements to Be Tested
• 1.5 Controversial Aspects of MCPs
• 1.5.1 Size of the Family
• 1.5.2 Composite Inferences vs. Individual Inferences
• 1.5.3 Bayesian Methods

Chapter 2 Concepts and Basic Methods for Multiple Comparisons and Tests

• 2.1 Introduction
• 2.2 Families of Hypotheses or Inferences
• 2.3 Error Rates
• 2.3.1 Comparisonwise Error Rate (CER)
• 2.3.2 Familywise Error Rate (FWE)
• 2.3.3 Control of the FWE: Weak and Strong
• 2.3.4 Directional Decisions and (Type III) Error rates
• 2.3.5 False Discovery Rate
• 2.4 Bonferroni and SidŽ ak Methods
• 2.4.1 Adjusted p-Values
• 2.4.2 An Example with Multiple p-Values
• 2.4.3 Example: Multiple Comparisons from a Questionnaire
• 2.5 Sequentially Rejective Methods
• 2.5.1 Bonferroni-Holm Method
• 2.5.2 SidŽak-Holm Method
• 2.5.3 Simes' Modified Bonferroni Procedure
• 2.5.4 Hommel's Procedure
• 2.5.5 Hochberg's Method - A Step-Up Test
• 2.5.6 Rom's Method
• 2.5.7 Benjamini and Hochberg's FDR-Controlling Method
• 2.5.8 Sequential Testing with Fixed Sequences
• 2.6 Graphical Presentation of Multiple Testing Results
• 2.6.1 The Schweder-Spjotvoll p-Value Plot
• 2.6.2 A Testing Procedure for Nominal FWE Protection
• 2.7 Concluding Remarks

Chapter 3 Multiple Comparisons among Treatment Means in the One-Way Balanced ANOVA

• 3.1 Introduction
• 3.2 Overview of Methods
• 3.2.1 The Model and Estimates
• 3.2.2 Simultaneous Confidence Intervals
• 3.3 All Pairwise Comparisons
• 3.3.1 Example of Pairwise Comparisons with Simultaneous Confidence Intervals
• 3.3.2 Displaying Pairwise Comparisons Graphically
• 3.3.3 Simultaneous Tests of Hypotheses
• 3.4 Pairwise Comparisons with a Control
• 3.4.1 Two-Sided Comparisons with a Control
• 3.4.2 One-Sided Comparisons with a Control
• 3.5 Multiple Inferences for Independent Estimates
• 3.5.1 Simultaneous Intervals for the Treatment Means
• 3.5.2 Orthogonal Comparisons
• 3.6 Concluding Remarks

Chapter 4 Multiple Comparisons among Treatment Means in the One-Way Unbalanced ANOVA

• 4.1 Introduction
• 4.2 All Pairwise Comparisons
• 4.2.1 The Tukey-Kramer Method
• 4.2.2 Graphical Comparisons - LINES Option
• 4.2.3 Simulation-based Methods
• 4.2.4 Tests of Hypotheses Adjusted p-Values
• 4.3 Pairwise Comparisons with Control
• 4.3.1 Distributions
• 4.3.2 Two-Sided Comparisons
• 4.3.3 One-Sided Comparisons
• 4.4 Concluding Remarks

Chapter 5 Multiple Comparisons among Treatment Means in the General Linear Model

• 5.1 Introduction
• 5.2 The Model and Estimates
• 5.2.1 The Model
• 5.2.2 The Estimates
• 5.2.3 The Multivariate t Distribution
• 5.2.4 Simultaneous Inferences for Estimable Functions
• 5.3 All Pairwise Comparisons in ANCOVA Models
• 5.3.1 Confidence Intervals
• 5.3.2 The %SimIntervals Macro
• 5.3.3 Tests of Hypotheses and Adjusted p-values
• 5.4 Comparisons with a Control
• 5.5 Multiple Comparisons when There are CLASSvar x Covariate Interactions
• 5.6 Concluding Remarks

Chapter 6 Inferences for General Linear Functions of Means

• 6.1 Introduction
• 6.2 Inferences for Any Finite Set of Linear Functions
• 6.2.1 Comparisons with Control, Including Dose-response Contrasts
• 6.2.2 Example: Evaluating Dose Response of Litter Weights in Rats
• 6.3 Inferences for Infinite Sets of Linear Functions
• 6.3.1 ANOVA tests
• 6.3.2 Confidence Bands for Regression Functions
• 6.3.3 Confidence Bands for Partial Functions
• 6.3.4 Confidence Band for Difference of Regression Functions; Interaction with Classification Variable and Covariate
• 6.3.5 Comparing the Discrete Method with the Continuous Method
• 6.4 Concluding Remarks

Chapter 7 Power and Sample Size in Simultaneous Inference

• 7.1 Introduction
• 7.2 Definitions of Power
• 7.2.1 Complete Power
• 7.2.2 Minimal Power
• 7.2.3 Individual Power
• 7.2.4 Proportional Power
• 7.3 Examples Using Individual Power
• 7.3.1 All Pairwise Comparisons
• 7.3.2 Comparisons with a control
• 7.3.3 Simultaneous Confidence Intervals for Means
• 7.4 Examples Using Combined Power Definitions
• 7.4.1 All Pairwise Comparisons
• 7.4.2 True FWE and Directional FWE
• 7.4.3 Comparisons with a Control
• 7.4.4 A Macro to Simulate and Graph Combined Power
• 7.5 Concluding Remarks

Chapter 8 Stepwise and Closed Testing Procedures

• 8.1 Introduction
• 8.2 The Closure Principle
• 8.3 Step-down Procedures for Testing All Subset Homogeneity Hypotheses
• 8.3.1 Tukey-Welsch
• 8.3.2 Example: Comparing Cholesterol Reduction Using Five Treatments
• 8.3.3 More about REGWQ
• 8.3.4 Power of the REGWQ Method
• 8.3.5 Begun and Gabriel
• 8.3.6 On Shortcut Closure Procedures
• 8.3.7 A Closed Procedure for Combination Tests
• 8.3.8 Caveats Concerning Closed Testing Methods
• 8.4 Step-down Procedures for Testing against a Common Control
• 8.5 Step-down and Closed Testing Procedures for Dose-Response Analysis
• 8.5.1 Tukey's Trend Test
• 8.5.2 A Closed Testing Procedure for Dose Response
• 8.5.3 Williams' Test
• 8.6 Closed Tests for General Contrasts
• 8.6.1 All Pairwise Comparisons Cholesterol Reduction Data Revisited
• 8.6.2 Using the %SimTests Macro with General Contrasts
• 8.6.3 Using the %SimTests Macro with General Contrasts and Covariates
• 8.6.4 Example: Evaluating Dose Response of Litter Weights in Rats Revisited
• 8.7 Concluding Remarks

Chapter 9 Simultaneous Inference in Two-Way and Higher-Way ANOVA and ANCOVA

• 9.1 Introduction
• 9.2 Two-Way ANOVA
• 9.2.1 Balanced with Replication
• 9.2.2 The Cell Means Model
• 9.2.3 Simultaneous Inference for Both Sets of Main Effects
• 9.2.4 Interaction Contrasts
• 9.2.5 Balanced ANOVA without Replication
• 9.2.6 Unbalanced Designs
• 9.2.7 Incomplete Two-way Designs
• 9.2.8 More Complex ANOVAS
• 9.3 Examples
• 9.3.1 Example: Effect of Protein in Diet on Weight Gain in Pigs: Three Way ANOVA with a Covariate
• 9.3.2 Example: Sub-group and Whole-group Analysis of a Respiratory Therapy Drug: Three-Way ANOVA with Weighted Contrasts
• 9.4 Multiple F-Tests
• 9.5 Concluding Remarks

Chapter 10 Multiple Comparisons in Heteroscedastic, Mixed, and Multivariate Models Using PROC MIXED

• 10.1 Introduction
• 10.2 Multiple Comparisons in the Heteroscedastic ANOVA
• 10.2.1 Simultaneous Intervals
• 10.2.2 Simultaneous Tests
• 10.3 Multiple Comparisons Among Treatments When Blocks Are Random
• 10.3.1 RCBD with One Observation per Cell
• 10.3.2 Incomplete Blocks
• 10.3.3 Comparing Means When Random Factors Interact
• 10.4 Repeated Measures Analysis
• 10.4.1 Repeated Measures Experimental Design
• 10.5 Multivariate Analysis
• 10.5.1 Testing Mean Differences in the MANOVA Model Multiple Outcomes
• 10.5.2 Multiple Comparisons in MANCOVA using PROC MIXED
• 10.5.3 Multiple Comparisons of Simple Effects in Repeated Measures
• 10.6 Concluding Remarks

Chapter 11 Multiple Comparisons of Means in Univariate and Multivariate Models, Allowing Nonnormality, Using PROC MULTTEST

• 11.1 Introduction
• 11.2 Univariate Means Tests Using PROC MULTTEST
• 11.2.1 Bootstrap Resampling
• 11.2.2 Permutation Resampling
• 11.2.3 A Caveat: The Subset Pivotality Condition, Heteroscedasticity and Excess Type I Errors
• 11.2.4 Incorporating Covariates with PROC MULTTEST
• 11.3 Testing Means from Multivariate Data Using PROC MULTTEST
• 11.3.1 Bootstrap and Permutation Resampling
• 11.3.2 Multiple Endpoints and the Subset Pivotality Condition
• 11.3.3 Missing Value Handling
• 11.4 Inferences for Multiple Contrasts and Multiple Variables Simultaneously
• 11.5 Concluding Remarks

Chapter 12 Multiple Comparisons with Binary Data Using PROC MULTTEST

• 12.1 Introduction
• 12.2 Multivariate Two-Sample Binary Outcomes
• 12.2.1 Resampling-based Multiplicity Adjustment
• 12.2.2 Multiplicity Adjustments Ignoring Correlations
• 12.3 Multiple Pairwise Comparisons with Binary Data
• 12.3.1 The Good and the Bad of Multiple Pairwise Comparisons: Two Examples
• 12.3.2 Using the Discrete Bonferroni Method to Avoid the Subset Pivotality Pitfall
• 12.3.3 Closed Binary Comparisons against a Control Using PROC MULTTEST
• 12.4 Improving the Power of Multiple Binary Tests
• 12.4.1 Comparison with PROC MULTTEST
• 12.5 Multiple Linear Contrast Tests
• 12.6 Multiple Animal Carcinogenicity Tests
• 12.7 Miscellaneous PROC MULTTEST Applications
• 12.7.1 Freeman-Tukey (FT) Test
• 12.7.2 Mixing Binary and Continuous Variables
• 12.7.3 Multiple Comparisons of Survival Functions
• 12.8 Concluding Remarks

Chapter 13 Bayesian Multiple Comparisons and Multiple Tests

• 13.1 Introduction
• 13.2 The Variance Component Model
• 13.3 Analysis of an Incomplete Block Design
• 13.3.1 The Model
• 13.3.2 Generating the Sample
• 13.3.3 Simultaneous Intervals
• 13.3.4 Multiple Hypothesis Testing: A Loss Function Approach
• 13.4 Multiple Bayesian Tests of Point Nulls
• 13.5 Concluding Remarks

Chapter 14 Additional Topics

• 14.1 Introduction
• 14.2 Large-Sample Multiple Comparisons Using PROC LOGISTIC, PROC LIFEREG, PROC PHREG, PROC CATMOD, PROC GENMOD
• 14.3 Multiple Comparisons with the Best
• 14.4 Infinitely Many Comparisons with Multivariate Data
• 14.5 Interim Analysis and Repeated Significance Tests in Clinical Trials
• 14.6 Concluding Remarks

Appendix: Macro Code

• A.1 The %Rom Macro
• A.2 The %HochBen Macro
• A.3 The %SimIntervals Macro
• A.4 The %MakeGLMStats Macro
• A.5 The %IndividualPower Macro
• A.6 The %SimPower Macro
• A.7 The %PlotSimPower Macro
• A.8 The %BegGab Macro
• A.9 The %RCC Macro
• A.10 The%Williams Macro
• A.11 The%SimTests Macro
• A.12 The%RomEx Macro
• A.13 The%RomMC Macro
• A.14 The%BayesIntervals Macro
• A.15 The%BayesTests Macro
• A.16 The%MCB Macro
• A.17 The%MCW Macro
• A.18 The%UMCB Macro
• A.19 The%UMCW Macro

References

Index