Logistic model selection using area under curve (AUC) or R-square selection criteria

Contents:

Purpose / History / Requirements / Usage / Details / Limitations

PURPOSE:

The SELECT macro performs model selection methods for categorical-response models that can be fit in PROC LOGISTIC. These include models using the logit, probit, cloglog, cumulative logit, or generalized logit links. The macro supports binary as well as ordinal and nominal multinomial models.

Standard model selection is done by choosing candidate effects for entry to or removal from the model according to their significance levels. After completion, the set of models selected at each step of this process is sorted on the selected criterion - AUC, R-square, max-rescaled R-square, AIC, or BIC. The requested number of best models on the selected criterion is displayed.

HISTORY:

Version	Update Notes
1.0	Initial coding

REQUIREMENTS:

Base SAS^® and SAS/STAT^® software are required.

USAGE:

Follow the instructions in the Downloads tab of this sample to save the SELECT macro definition. Replace the text within quotation marks in the following statement with the location of the SELECT macro definition file on your system. In your SAS program or in the SAS editor window, specify this statement to define the SELECT macro and make it available for use:

   %inc "<location of your file containing the SELECT macro>";

Following this statement, you can call the SELECT macro. See the Results tab for examples.

The following parameters are required when using the SELECT macro:

response=variable

Specifies the name of the response variable to be modeled. The variable can be binary or multinomial with ordered or unordered levels. The response variable should not be continuous. Response variable options, such as EVENT= or ORDER= specified in parentheses, can follow the response variable name. For example, RESPONSE=Y(EVENT="1").

model=effect-list

Specifies the set of candidate effects from which a final model is selected based on their significance levels. Any type of list allowed in the MODEL statement in PROC LOGISTIC can be used, such as: VAR:, X1-X10, A--K, A|B|C@2.

class=variable-list and options

Required only if some variables in the candidate effects in the MODEL= option are categorical variables. Specify categorical variables and variable options using the same syntax as the CLASS statement in PROC LOGISTIC. For example, CLASS=A B(REF="1") / PARAM=REF.

The following parameters are optional:

data=data-set-name

Specifies the data set containing the response variable and variables in the candidate effects. The default is the last-created data set.

method=selection-method

Specifies the model selection to use: FORWARD, BACKWARD, or STEPWISE. The SCORE method in PROC LOGISTIC is not available. METHOD=STEPWISE is the default.

choose=selection-criterion

Specifies the criterion statistic to order the selected models: AUC, RSQUARE, RSQUARE_RESCALED, AIC, or BIC. CHOOSE=AUC is the default. The AUC criterion is not available when LINK=GLOGIT.

best=number|MAX

Specifies the number of selected models to display that are best on the CHOOSE= criterion. Number must be zero or a positive integer. BEST=MAX is the default which displays all selected models.

include=number

Specifies the number of candidate effects in the MODEL statement to include in every model during model selection. INCLUDE=number requests that the first number of effects in the MODEL= option be included in every model. The default is INCLUDE=0.

link=LOGIT|PROBIT|CLOGLOG|GLOGIT

Specifies the link function. The default is LINK=LOGIT. For an ordinal response, the default LINK=LOGIT uses cumulative logits. For a nominal (unordered) response, specify LINK=GLOGIT.

sle=number

Specifies the significance level for entering an effect into the model in the FORWARD or STEPWISE method. Number should be between 0 and 1, inclusive. The default is SLE=0.05. The SLE= option has no effect when METHOD=BACKWARD.

sls=number

Specifies the significance level for an effect to stay in the model in a backward elimination step. Number should be between 0 and 1, inclusive. The default is SLS=0.05. The SLS= option has no effect when METHOD=FORWARD.

modelopts=option-list

Specifies any additional options which can appear following a slash in the MODEL statement in PROC LOGISTIC. An option available in the SELECT macro, even its default is not specified, takes precedence over specifying the option in the MODELOPTS= option. For example, specifying MODELOPTS=INCLUDE=3 does not change the default INCLUDE=0. To do this, specify the INCLUDE=3 option.

details=SUMMARY|ALL

Requests display of only the model selection summary table from PROC LOGISTIC (DETAILS=SUMMARY), or the display of all results produced by the DETAILS option of PROC LOGISTIC (DETAILS=ALL). If omitted, only the final list of models sorted on the CHOOSE= criterion is displayed and all model selection results from PROC LOGISTIC are suppressed.

out=data-set-name

Specifies the name of the final data set containing the list of selected models sorted by the CHOOSE= criterion. By default, OUT=BestMods.

binwidth=value

Specifies the size of the bins on the predicted probabilities to be used when computing the area under the ROC curve (AUC) for binary and ordinal response models. Value can be 0 to 1, inclusive (except for ordinal response models where value must be greater than zero). Smaller values of BINWIDTH= provide more precise estimates of the AUC but require more time and memory. The default is BINWIDTH=1E-6.

freq=variable

Specifies a variable that contains the frequency of occurrence of each observation. Each observation is treated as if it appears n times, where n is the value of the FREQ= variable for the observation. If it is not an integer, the frequency value is truncated to an integer. If the frequency value is less than 1 or missing, the observation is not used in the model fitting. When FREQ= is not specified, each observation is assigned a frequency of 1.

weight=variable

Specifies a variable that weights each observation in the input data set by its value on the WEIGHT= variable. Unlike the FREQ= variable, the values of the WEIGHT= variable can be nonintegral and are not truncated. Observations with negative, zero, or missing values for the WEIGHT variable are not used in the model fitting. When WEIGHT= is not specified, each observation is assigned a weight of 1.

The version of the SELECT macro that you are using is displayed when you specify version (or any string) as the first argument. For example:

    %SELECT(version, response=y, model=x1-x10)

DETAILS:

Model selection is done in the same manner as PROC HPLOGISTIC when the CHOOSE= and SELECT=SL method options are specified with the forward, backward, or stepwise methods. That is, effects are added to or removed from the model according to their significance levels resulting in a set of models. The CHOOSE= criterion is then used to sort the models. In addition to the AIC and BIC criteria available in PROC HPLOGISTIC, the SELECT macro can also choose models using the area under the ROC curve (CHOOSE=AUC), the R-square statistic (CHOOSE=RSQUARE), or the max-rescaled R-square statistic (CHOOSE=RSQUARE_RESCALED).

Note that since the SELECT macro uses PROC LOGISTIC to fit models, differences from PROC HPLOGISTIC can occur due to differences in defaults. For example, PROC HPLOGISTIC does not require model hierarchy by default and uses a Newton-Raphson algorithm rather than the Fisher scoring method. Using the HIERARCHY=NONE and TECHNIQUE=NEWTON options in the MODEL statement of PROC LOGISTIC removes these differences.

Macro updates:

The SELECT macro attempts to check for a later version of itself. If it is unable to do this (such as if there is no active Internet connection available), the macro issues the following message:

   SELECT: Unable to check for newer version

The computations performed by the macro are not affected by the appearance of this message.

LIMITATIONS:

The best subsets selection method (SELECTION=SCORE) available in PROC LOGISTIC is not available in the SELECT macro. The area under the ROC curve criterion (CHOOSE=AUC) is not available with nominal, multinomial (LINK=GLOGIT) models.

---

These sample files and code examples are provided by SAS Institute Inc. "as is" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Recipients acknowledge and agree that SAS Institute shall not be liable for any damages whatsoever arising out of their use of this material. In addition, SAS Institute will provide no support for the materials contained herein.

---

---

These sample files and code examples are provided by SAS Institute Inc. "as is" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Recipients acknowledge and agree that SAS Institute shall not be liable for any damages whatsoever arising out of their use of this material. In addition, SAS Institute will provide no support for the materials contained herein.

EXAMPLE 1: Stepwise selection using AUC with categorical predictors

This example uses the data in the example titled "Logistic Modeling with Categorical Predictors" in the LOGISTIC documentation. The data are from a study comparing the analgesic effects of two test treatments and placebo on neuralgia patients. Researchers recorded patient age and gender, the duration of complaint before treatment, and a binary response of whether they experience pain or not under treatment.

The following call of the SELECT macro uses the stepwise method to select a model from a set of candidate effects that includes the main effects and two-way interactions of the four predictor variables. Effects are allowed to enter and stay in the model if they are significant at the 0.25 level. The set of selected models are then ordered by their AUC values and displayed. Before calling the SELECT macro, specify a %INC statement once per SAS session to define the macro and make it available for use during that SAS session.

      %inc "<location of your file containing the SELECT macro>";
      %select(data=neuralgia,
              response=Pain(event="No"),
              class=Treatment(ref="P") Sex(ref="M") / param=ref,
              model=Treatment|Sex|Age|Duration@2,
              method=stepwise, sle=.25, sls=.25, 
              choose=auc)

The table of selected models displayed by the macro shows that the model with the best AUC (0.926) contains Treatment, Sex, and Age main effects as well as the Treatment*Age interaction. This model was selected in the fourth and final step of the stepwise selection method.

The SELECT Macro AUC eff1 eff2 eff3 eff4 Step 0.926286 Age Age*Treatment Sex Treatment 4 0.905714 Age Sex Treatment   3 0.867429 Age Treatment     2 0.728571 Treatment       1

Note that the selection of model effects occurs in the same way as by PROC LOGISTIC or PROC HPLOGISTIC, but these procedures do not allow final selection of a model based on AUC or R-square statistics. That is, the same set of four models is found as follows, but not sorted using those two statistics. Note that the TECHNIQUE=NEWTON and HIERARCHY=SINGLE options are used to make the procedures match.

      proc logistic data=neuralgia;
         class Treatment(ref="P") Sex(ref="M") / param=ref;
         model Pain(event="No") = Treatment|Sex|Age|Duration@2 / 
               selection=stepwise sle=.25 sls=.25 technique=newton;
         run;
      proc hplogistic data=neuralgia;
         class Treatment(ref="P") Sex(ref="M") / param=ref;
         model Pain(event="No") = Treatment|Sex|Age|Duration@2;
         selection method=stepwise(select=SL sle=.25 sls=.25) hierarchy=single;
         run;

EXAMPLE 2: Selecting an ordinal, partial proportional odds model with highest R-square

This example uses the data in the example titled "Partial Proportional Odds Model" in the LOGISTIC documentation. The following call of the SELECT macro uses the default stepwise method to fit a series of ordinal logistic models selecting predictors from the list in the MODEL= option. The SEX--CHCOND2 variable list represents a set of twelve variables in the DOCVISIT data set beginning with SEX and ending with CHCOND2. Note that the UNEQUALSLOPES and EQUALSLOPES options in the PROC LOGISTIC MODEL statement are added. This allows PROC LOGISTIC to determine not only if a candidate predictor is important, but also whether it can appear in the model with equal slopes (that is, maintaining proportional odds) or whether unequal slopes are needed.

After the set of models is selected, the models are ordered by their max-rescaled R-square values for display. The BEST=3 option limits the display to the three models with the largest max-rescaled R-square values. The DETAILS=SUMMARY option also requests display of the stepwise method summary table from PROC LOGISTIC.

      %select(data=docvisit,
              response=dvisits, 
              model=sex--chcond2,
              choose=rsquare_rescaled, 
              best=3,
              modelopts=unequalslopes equalslopes, 
              details=summary)

The selected model has equal slopes for four of the predictors and unequal slopes for two others. It was the last model selected and has the largest max-rescaled R-square (0.185).

The LOGISTIC Procedure Summary of Stepwise Selection Step Effect DF Number In Score Chi-Square Wald Chi-Square Pr > ChiSq Entered Removed 1 U_actdays   2 1 636.7246   <.0001 2 illness   1 2 225.2868   <.0001 3 U_agesq   2 3 50.2475   <.0001 4 sex   1 4 16.3442   <.0001 5 hscore   1 5 13.5518   0.0002 6 freepoor   1 6 11.2484   0.0008 The SELECT Macro RSquare_rescaled eff1 eff2 eff3 eff4 eff5 eff6 Step 0.185463 U_actdays U_agesq freepoor hscore illness sex 6 0.182407 U_actdays U_agesq hscore illness sex   5 0.179314 U_actdays U_agesq illness sex     4

Right-click on the link below and select Save to save the SELECT macro definition to a file. It is recommended that you name the file select.sas.

Download and save select.sas

The SELECT macro provides forward, backward, and stepwise model selection methods for categorical-response models and sorts models on the specified criterion - area under the ROC curve (AUC), R-square, max-rescaled R-square, AIC, or BIC.

Type:	Sample
Topic:	Analytics ==> Categorical Data Analysis Analytics ==> Regression SAS Reference ==> Procedures ==> LOGISTIC

Date Modified:	2014-12-16 14:01:50
Date Created:	2014-12-16 13:38:07

Operating System and Release Information

Product Family	Product	Host	SAS Release
			Starting	Ending
SAS System	SAS/STAT	z/OS
		Z64
		OpenVMS VAX
		Microsoft® Windows® for 64-Bit Itanium-based Systems
		Microsoft Windows Server 2003 Datacenter 64-bit Edition
		Microsoft Windows Server 2003 Enterprise 64-bit Edition
		Microsoft Windows XP 64-bit Edition
		Microsoft® Windows® for x64
		OS/2
		Microsoft Windows 8 Enterprise 32-bit
		Microsoft Windows 8 Enterprise x64
		Microsoft Windows 8 Pro 32-bit
		Microsoft Windows 8 Pro x64
		Microsoft Windows 8.1 Enterprise 32-bit
		Microsoft Windows 8.1 Enterprise x64
		Microsoft Windows 8.1 Pro
		Microsoft Windows 8.1 Pro 32-bit
		Microsoft Windows 95/98
		Microsoft Windows 2000 Advanced Server
		Microsoft Windows 2000 Datacenter Server
		Microsoft Windows 2000 Server
		Microsoft Windows 2000 Professional
		Microsoft Windows NT Workstation
		Microsoft Windows Server 2003 Datacenter Edition
		Microsoft Windows Server 2003 Enterprise Edition
		Microsoft Windows Server 2003 Standard Edition
		Microsoft Windows Server 2003 for x64
		Microsoft Windows Server 2008
		Microsoft Windows Server 2008 R2
		Microsoft Windows Server 2008 for x64
		Microsoft Windows Server 2012 Datacenter
		Microsoft Windows Server 2012 R2 Datacenter
		Microsoft Windows Server 2012 R2 Std
		Microsoft Windows Server 2012 Std
		Microsoft Windows XP Professional
		Windows 7 Enterprise 32 bit
		Windows 7 Enterprise x64
		Windows 7 Home Premium 32 bit
		Windows 7 Home Premium x64
		Windows 7 Professional 32 bit
		Windows 7 Professional x64
		Windows 7 Ultimate 32 bit
		Windows 7 Ultimate x64
		Windows Millennium Edition (Me)
		Windows Vista
		Windows Vista for x64
		64-bit Enabled AIX
		64-bit Enabled HP-UX
		64-bit Enabled Solaris
		ABI+ for Intel Architecture
		AIX
		HP-UX
		HP-UX IPF
		IRIX
		Linux
		Linux for x64
		Linux on Itanium
		OpenVMS Alpha
		OpenVMS on HP Integrity
		Solaris
		Solaris for x64
		Tru64 UNIX