IMSTAT Procedure (Analytics)

GENMODEL Statement

The GENMODEL statement is used to fit statistical models from the class of generalized linear models and some related models.

Syntax

GENMODEL dependent-variable <(class-variables)> = model-effects </ options>;

Required Arguments

dependent-variable

specifies the variable to model. This variable is also referred to as the response variable.

model-effects

specifies a list of variables to use for modeling the dependent variable.

Optional Argument

class-variables

specifies a list of variables to use as classification variables. The variables in this list take the place of the CLASS statement in traditional SAS procedures.

GENMODEL Statement Options

ALLIDVARS

requests that all variables in the input table are treated as ID variables when a scoring table is produced. In other words, if this option is specified, all variables from the input table, including computed columns, are transferred to the scoring table.

ALPHA=number

specifies a number between 0 and 1 from which to determine the confidence level for approximate confidence intervals of the parameter estimates. The default is α = 0.05, which leads to 100 x (1- α)% = 95% confidence limits for the parameter estimates.

Default 0.05

CI

specifies to add confidence intervals to the table of parameter estimates. The confidence level is 100*(1-α)% where α is determined by the ALPHA= option. The default value is α = 0.05. This value is equivalent to a 95% confidence limit.

Default 0.05

CLASSFORMATS=("format-name1"<, "format-name2" ...>)

specifies the formats for the classification variables in the model. If you do not specify the CLASSFORMATS= option, the default format is applied for the classification variable. That default format was determined when the table was originally loaded into the server. In the following example, the CLASSFORMAT= values apply to variables x1 and x2.

Alias CLASSFMT=
Example
genmodel y (x1 x2) = x3-x7 / classformats=("YN.", "F8.");

CODE <(code-generation-options)>

requests that the server produce SAS scoring code based on the actions that it performed during the analysis. The server generates DATA step code. By default, the code is replayed as an ODS table by the procedure as part of the output of the statement. More frequently, you might want to write the scoring code to an external file by specifying options.

The scoring code computes the predicted value of the response variable on the data scale (the inverse link scale) and prefixes the name with "P_". For example, if the response variable is Y, the generated code stores the predicted value as P_Y. The name of the variable is truncated to fit within the SAS name length requirements.

COMMENT

specifies to add comments to the code in addition to the header block. The header block is added by default.

FILENAME='path'

specifies the name of the external file to which the scoring code is written. This suboption applies only to the scoring code itself. If you request that the server generate IMSTAT programming statements with the IMSTAT suboption, then these statements are saved as an ODS table.

Alias FILE=

FORMATWIDTH=k

specifies the width to use in formatting derived numbers such as parameter estimates in the scoring code. The server applies the BEST format, and the default format for code generation is BEST20.

Alias FMTWIDTH=
Range 4 to 32

IMSTAT

specifies to generate IMSTAT programming statements that reproduce the analysis in addition to the scoring code. For example, this option is helpful when you perform variable selection and you want to capture the modeling code that reflects only the selected variables.

IMSTATONLY

specifies to generate the IMSTAT programming statements only. No scoring code is produced.

LINESIZE=n

specifies the line size for the generated code.

Alias LS=
Default 72
Range 64 to 256

NOTRIM

requests that the comparison of the formatted values for class variables and group-by variables is based on the full format width with padding. By default, the leading and trailing blanks are removed from the formatted values.

REPLACE

specifies to overwrite the external file with the new contents if the file already exists. This option has no effect unless you specify the FILENAME= option.

DIST=distribution

specifies the distribution of the response variable. See the following list for the available values:

  • BETA
  • EXPONENTIAL | EXPO
  • GAMMA | GAM
  • GENPOISSON
  • GEOMETRIC | GEOM
  • INVGAUSS | IGAUSSIAN | IG
  • NEGBINOMIAL | NEGBIN | NB
  • NORMAL | GAUSSIAN | GAUSS
  • POISSON | PO
  • T | STUDENT
  • WEIBULL

EXCLUDE=(list-of-ODS-tables)

specifies the result tables that you want to exclude from being generated on the server and from being sent to the client. The GENMODEL statement can generate the following tables:

Table Name
Table Alias
Description
Condition
ModelInfo
Information about the model—constant across groups or partitions.
This table is shown by default.
ClassLevels
Class
Information about the classification variables, such as the number of levels and their values.
This table is shown when classification variables are present in the model.
ConvStatus
Convergence
Convergence status of optimization
This table is shown by default.
Dimensions
Dim
Model dimensions
This table is shown by default.
FitStatistics
Fit
Fit statistics customary for generalized linear models
This table is shown when it is requested with the SELECT= option.
IterHistory
IterHist
Iteration history
This table is shown when the ITDETAILS option is used or when the table is requested with the SELECT= option.
ParmEstimates
ParameterEstimates
Pest
The solutions for the linear model coefficients
This table is shown by default.
Tests3
Type III tests of model effects
This table is shown when the distribution is in the exponential family, the effects contain classification variables, and the NOSTDERR option is not specified.
Whether a table is shown by default or not, you can request any table with the SELECT= option in the GENMODEL statement. The Condition column in the table identifies when a table is produced by default.

FORMATS=("format-specification"<,...>)

specifies the formats for the GROUPBY variables. If you do not specify the FORMATS= option, or if you omit the entry for a GROUPBY variable, the default format is applied for that variable.

Enclose each format specification in quotation marks and separate each format specification with a comma.
Example
proc imstat data=lasr1.table1;
   statement / groupby=(a b) formats=("8.3", "$10");
quit;

FCONV=r

specifies a relative function convergence criterion. For all techniques except NMSIMP, termination requires a small relative change of the function value in successive iterations. Suppose that Ψ is the p × 1 vector of parameter estimates in the optimization and the objective function at the kth iteration is denoted, f(Ψ)k. Then, the FCONV criterion is met if

fconv
Default r=10-FDIGITS where FDIGITS is -log10(e) and e is the machine precision.

FREQ=variable-name

specifies the numeric variable that provides frequencies for the analysis. For example, if the FREQ= variable has the value 5, then it implies that the record represents five such observations with identical values for the modeling variables. If you specify a FREQ= variable, then only the observations with a value that is not missing and greater than zero for the variable are used in the analysis.

GCONV=r

specifies a relative gradient convergence criterion. For all optimization techniques except CONGRA and NMSIMP, termination requires that the normalized predicted function reduction is small. The default value is r = 1e-8. Suppose that Ψ is the p × 1 vector of parameter estimates in the optimization with ith element Ψi. . The objective function, its p × 1 gradient vector, and its p × p Hessian matrix are denoted, f(Ψ), g(Ψ), and H(Ψ ), respectively. Then, if superscripts denote the iteration count, the normalized predicted function reduction at iteration k is

gconv
The GCONV convergence criterion is assumed to be met if that value is less than or equal to r.
Note that it is possible that the relative gradient reduction is small, even if one or more gradients is still substantial in absolute value. If this situation occurs, you can disable the GCONV criterion by setting r=0. If the optimization would have stopped early due to meeting the GCONV criterion, the iterative process usually takes one more step until the gradients are small in absolute value.

GROUPBY=(variable-list)

specifies a list of variable names, or a single variable name, to use as GROUPBY variables in the order of the grouping hierarchy. If you do not specify any GROUPBY variable names, then the calculation is performed across the entire table—possibly subject to a WHERE clause.

GROUPFILTER=(filter-options)

specifies a section of the group-by hierarchy to be included in the computation. With this option, you can request that the server performs the analysis for only a subset of all possible groupings. The subset is determined by applying the group filter to a temporary table that you generate with the GROUPBY statement.

You can specify the following suboptions in the GROUPFILTER option:

DESCENDING

specifies the top or the bottom section of the groupings to be collected. If the DESCENDING option is specified, the top LIMIT=n (where n > 0) groupings are collected. Otherwise, the bottom LIMIT=n groupings are collected.

Alias DESC

LIMIT=n

specifies the maximum number of distinct groupings to be collected, where integer n >= 0. If n is zero, then all distinct groupings (up to 2^31-1) that satisfy the boundary constraints, such as LOWERSCORE=f, are collected.

CAUTION:
High Cardinality Data Sets
Setting n to zero with high-cardinality data sets can significantly delay the response of the server.

SCOREGT=f

specifies the exclusive lower bound for the numeric scores of the distinct groupings to collect.

Alias SGT=

SCORELT=f

specifies the exclusive upper bound for the numeric scores of the distinct groupings to collect.

Alias SLT=

VALUEGT=("format-name1" <, "format-name2" ...>)

specifies the exclusive lower bound of the group-by variable’s formatted values for the distinct groupings to collect.

Alias VGT=

VALUELT=("format-name1" <, "format-name2" ...>)

specifies the exclusive upper bound of the group-by variable’s formatted values for the distinct groupings to collect.

Alias VLT=

TABLE=table-with-groupby-results

specifies the in-memory table from which to load the group-by hierarchy. If the TABLE= option is not specified, then all other GROUPFILTER= options are ignored.

The following program request all the groupings of State, City, and then Trade_In_Model in the Cars_Program_All table. The groupings are ordered by the maximum value of New_Vehicle_Msrp for each grouping:
proc imstat;
    table example.CARS_PROGRAM_ALL;
    groupby state city trade_in_model / TEMPTABLE 
                 WEIGHT=new_vehicle_MSRP 
                 AGG   =(MAX) 
                 ORDER =WEIGHT;  
run;
The TEMPTABLE option in the GROUPBY statement directs the server to save all the groupings in a temporary in-memory table. The following DISTINCT statement requests the count of the distinct unformatted values of Sales_Type for each of the selected groupings of State, City, and Trade_In_Model.
    table example.CARS_PROGRAM_ALL;
    distinct sales_type / GROUPFILTER=(
                 table  =mylasr.&_TEMPLAST_
                 scoregt=40000
                 valuelt=("FL","Ft Myers","")
                 limit  =20
                 descending);
run;
This example only considers groupings that have maximum values of the New_Vehicle_Msrp above 40,000 and with formatted values that are less than State="FL" and City="Ft Myers." The empty quotation marks result in no restriction on Trade_In_Model values. These groupings are ordered according to the maximum values of New_Vehicle_Msrp. Because of the DESCENDING option, this example collects the 20 top groupings within the specified group-by range for the DISTINCT analysis.
Interaction If you specify the GROUPFILTER= option, then the GROUPBY= and FORMATS= options have no effect.

IDVARS=(variable-list)

IDVARS=variable-name

specifies the variables from the active table to transfer to the temporary table that is created by scoring the input table. This option has no effect unless the SCORE option is also specified. (See the SCORE option for details about which variables are added to the temporary table by default.) The IDVARS= option should be used to transfer additional columns from the input table to the scoring table.

Alias ID=
Tip Instead of this option, you can specify the ALLIDVARS option to transfer all variables from the input table to the scoring table.

ITDETAILS

requests to add details about the iterative model fitting process (an iteration history) to the ODS output tables.

Alias ITDETAIL

KEYORDER

requests that the results for a partitioned analysis are displayed in the order of the partition keys. If this option is not specified, then results are displayed by using the partitions on the first worker node followed by the partitions on the second node, and so on. Without this option, the results are likely to have random ordering of the partitions. The KEYORDER option makes result collection less efficient but produces a natural, predictable order.

LINK=function

specifies the link function to use for the model fitting process. If you do not specify a link function, the server selects the most appropriate function for the distribution of the data. See the following list for the available functions:

  • IDENTITY
  • LOGIT
  • PROBIT
  • LOG
  • LOGLOG
  • CLOGLOG
  • RECIP
  • POWMINUS2
  • POWER(v) | POW(v) | POM(v)

MAXFUNC=n

specifies the maximum number n of function calls in the iterative model fitting process. The default value depends on the optimization technique as follows:

Optimization Technique
Default Number of Function Calls
TRUREG, NRRIDG, and NEWRAP
125
QUANEW and DBLDOG
500
CONGRA
1000
NMSIMP
3000
Alias MAXFU=

MAXITER=i

specifies the maximum number of iterations in the iterative model fitting process. The default value depends on the optimization technique as follows:

Optimization Technique
Default Number of Iterations
TRUREG, NRRIDG, and NEWRAP
50
QUANEW and DBLDOG
200
CONGRA
400
NMSIMP
1000
Alias MAXIT=

MAXTESTLEV=n

specifies the maximum number of levels in an effect for which the server generates Type III tests. The idea behind the MAXTESTLEV= option is that testing effects for significance that have a large number of levels is typically not meaningful. The effects tend to be highly significant anyway, but determining the exact significance level is computationally intensive. The default value is 300 and implies that no test statistics are produced for any effect that has more than 300 levels.

Default 300

NOCLPRINT <=n>

specifies the number of levels for each classification variables to show in the Class Level Information ODS table. If you do not specify the NOCLPRINT option, all unique values are shown in the order of the class variable levelization. If you specify NOCLPRINT=n, then the values are shown for those classification variables that have less than n levels only. The value for n must be at least 1.

If you specify the NOCLPRINT option without specifying a value for n, then n = 0 is assumed. This enables you to get a listing of the classification variables in the model. This might be useful if you did not identify classification variables explicitly—without listing their (possibly many) levels.
For example, the following Class Level Information table is displayed with NOCLPRINT=4. Because the number of levels for variable Smoking_Status exceeds 4, the values are not displayed.
Class Level Information ODS table

NOINT

suppresses the inclusion of an intercept in the model. By default, all models contain an intercept term.

NOPREPARSE

prevents the procedure from pre-parsing and pre-generating code for temporary expressions, scoring programs, and other user-written SAS statements.

When this option is specified, the user-written statements are sent to the server "as-is" and then the server attempts to generate code from it. If the server detects problems with the code, the error messages might not to be as detailed as the messages that are generated by SAS client. If you are debugging your user-written program, then you might want to pre-parse and pre-generate code in the procedure. However, if your SAS statements compile and run as you want them to, then you can specify this option to avoid the work of parsing and generating code on the SAS client.
When you specify this option in the PROC IMSTAT statement, the option applies to all statements that can generate code. You can also exclude specific statements from pre-parsing by using the NOPREPARSE option in statements that allow temporary columns or the SCORE statement.
Alias NOPREP

NOSTDERR

prevents the computation of the covariance matrix and the standard errors of the parameter estimates. When you specify this option, the Type III tests for the model effects are also not available.

Alias NOSTD

OFFSET=variable-name

specifies the offset variable for the analysis. An offset variable can be thought of as a regressor variable whose regression coefficient is known to be 1. Offsets are used to shift the linear predictors by a certain amount. For example, an offset can be used to accommodate constants in the underlying model. In generalized linear models, offsets arise frequently when the data represents a value relative to some measure of size. For example, if you model the number of stops (Y) for each trip and the trips are of different length (t), then you are really interested in the random variable Y/t. The generalized linear model becomes as follows:

Generalized linear model
If you choose the log link function, this can be written as follows:
Log link function
The value log(t) is the offset of the model.

PARTITION<=partition-key>

specifies to fit the model separately for each value of the partition key. In other words, the partition variables function as automatic group-by variables for the request.

If you do not specify a value for partition-key, then the analysis is performed for all partitions. If you do specify a value, then the analysis is performed for the specified key value only. You can use the PARTITIONINFO statement to retrieve the valid partition-key values for a table.
Alias PART=

ROLEVAR=variable-name

specifies a variable in the in-memory table that defines whether an observation belongs to the training set, the validation set, or is to be excluded from the analysis. The role variable can have a numeric or character type, and it can be a temporary computed variable.

In case of a numeric role variable, the values of variable-name are interpreted as follows:
  • value = 1: this observation is in the training set
  • value = 2: this observation is in the validation set
  • any other value: this observation is to be excluded from the analysis
In case of a character role variable, the values of variable-name are interpreted as follows:
  • If the first non-blank character is 't' or 'T', then the observation is in the training set.
  • If the first non-blank character is 'v' or 'V', then he observation is in the validation set.
  • Any other value for the first non-blank character, including an all blank entry, leads to the exclusion of the observation from the analysis.
Alias ROLE=
Interactions You can divide the data at random into training and validation sets by providing the VALIDATE= and SEED= options.
If you specify both the ROLEVAR= and the VALIDATE= options, then the ROLEVAR= setting supersedes the VALIDATE= option.

SELECT=(list-of-ODS-tables)

specifies the list of ODS tables that you want to display for the analysis. The specified list replaces the default tables that are generated by the server and displayed. See the EXCLUDE= option for the list of default tables and the table names that you can display.

TECHNIQUE=value

specifies the optimization technique for the iterative model-fitting process. The valid values are as follows:

  • CONGRA (CG): performs a conjugate-gradient optimization
  • DBLDOG (DD): performs a version of the double-dogleg optimization
  • NMSIMP (NS): performs a Nelder-Mead simplex optimization
  • NONE: does not perform any optimization. This value can be useful to perform a grid search without optimization.
  • NEWRAP (NRA): performs a (modified) Newton-Raphson optimization that combines a line-search algorithm with ridging
  • NRRIDG (NRR): performs a (modified) Newton-Raphson optimization with ridging
  • QUANEW (QN): performs a quasi-Newton optimization
  • TRUREG (TR): performs a trust-region optimization
The factors that go into choosing a particular optimization technique for a particular problem are complex. Trial and error can be involved.
Default NRRIDG

TEMPEXPRESS="SAS-expressions"

TEMPEXPRESS=file-reference

specifies either a quoted string that contains the SAS expression that defines the temporary variables or a file reference to an external file with the SAS statements.

Alias TE=

TEMPNAMES=variable-name

TEMPNAMES=(variable-list)

specifies the list of temporary variables for the request. Each temporary variable must be defined through SAS statements that you supply with the TEMPEXPRESS= option.

Alias TN=

TEMPTABLE

generates an in-memory temporary table from the result set. The IMSTAT procedure displays the name of the table and stores it in the &_TEMPLAST_ macro variable, provided that the statement executed successfully.

When the IMSTAT procedure exits, all temporary tables created during the IMSTAT session are removed. Temporary tables are not displayed on a TABLEINFO request, unless the temporary table is the active table for the request.

VALIDATE=f

specifies the proportion f in the validation data set.

Alias VALPROP=
Range 0 to 1
Interaction If you specify both the ROLEVAR= and the VALIDATE= options, then the ROLEVAR= setting supersedes the VALIDATE= option.

WEIGHT=variable-name

specifies the numeric variable to use as a weighing variable in solving the linear model.

Details

The GENMODEL statement offers the options that are necessary to formulate a model in the generalized linear model family, and to specify some other, closely related, models. For example, you can fit data to the generalized Poisson distribution for overdispersed Poisson counts. Or, you can use the shifted T distribution for symmetric data that is heavier in the tails than the normal distribution.