The following section discusses the statistical tests performed in the MULTTEST procedure. For continuous data, a t test for the mean (MEAN) is available. For discrete variables, available tests are the CochranArmitage linear trend test (CA), the FreemanTukey double arcsine test (FT), the Peto mortalityprevalence test (PETO), and the Fisher exact test (FISHER).
Throughout this section, the discrete and continuous variables are denoted by and , respectively, where v is the variable, g is the treatment group, s is the stratum, and r is the replication. Let denote the sample size for a binary variable v within group g and stratum s. A plus sign (+) subscript denotes summation over an index. Note that the tests are invariant to the location and scale of the contrast coefficients .
The CochranArmitage linear trend test (Cochran, 1954; Armitage, 1955; Agresti, 2002) is implemented by using a Zscore approximation, an exact permutation distribution, or a combination of both.
The pooled probability estimate for variable v and stratum s is
The expected value (under constant withinstratum treatment probabilities) for variable v, group g, and stratum s is
Letting denote the contrast trend coefficients specified by the CONTRAST statement, the test statistic for variable v has numerator
The binomial variance estimate for this statistic is
where
The hypergeometric variance estimate (the default) is
For any strata s with , the contribution to the variance is taken to be zero.
PROC MULTTEST computes the Zscore statistic
The pvalue for this statistic comes from the standard normal distribution. Whenever a 0 is computed for the denominator, the pvalue is set to 1. This pvalue approximates the probability obtained from the exact permutation distribution, discussed in the following text.
The Zscore statistic can be continuitycorrected to better approximate the permutation distribution. With continuity correction c, the uppertailed pvalue is computed from
For twotailed, noncontinuitycorrected tests, PROC MULTTEST reports the pvalue as , where p is the uppertailed pvalue. The same formula holds for the continuitycorrected test, with the exception that when the noncontinuitycorrected Z and the continuitycorrected Z have opposite signs, the twotailed pvalue is 1.
When the PERMUTATION= option is specified and no STRATA variable is specified, PROC MULTTEST uses a continuity correction selected to optimally approximate the uppertail probability of permutation distributions with smaller marginal totals (Westfall and Lin, 1988). Otherwise, the continuity correction is specified by the CONTINUITY= option in the TEST statement.
The CA Zscore statistic is the HoelWalburg (MantelHaenszel) statistic reported by Dinse (1985).
When you use the PERMUTATION= option for CA in the TEST statement, PROC MULTTEST computes the exact permutation distribution of the trend score
where the contrast trend coefficients must be integer valued. The observed value of this trend is compared to the permutation distribution to obtain the pvalue
where X is a random variable from the permutation distribution and where uppertailed tests are requested. This probability can be viewed as a binomial probability, where the withinstratum probabilities are constant and where the probability is conditional with respect to the marginal totals . It also can be considered a rerandomization probability.
Because the computations can be quite timeconsuming with large data sets, specifying the PERMUTATION=number option in the TEST statement limits the situations where PROC MULTTEST computes the exact permutation distribution. When marginal total success or total failure frequencies exceed number for a particular stratum, the permutation distribution is approximated by a continuitycorrected normal distribution. You should be cautious when using the PERMUTATION= option in conjunction with bootstrap resampling because the permutation distribution is recomputed for each bootstrap sample. This recomputation is not necessary with permutation resampling.
The permutation distribution is computed in two steps:
The permutation distributions of the trend scores are computed within each stratum.
The distributions are convolved to obtain the distribution of the total trend.
As long as the total success or failure frequency does not exceed number for any stratum, the computed distributions are exact. In other words, if number or number for all s, then the permutation trend distribution for variable v is computed exactly.
In step 1, the distribution of the withinstratum trend
is computed by using the multivariate hypergeometric distribution of the , provided number is not exceeded. This distribution can be written as
The distribution of the withinstratum trend is then computed by summing these probabilities over appropriate configurations. For further information about this technique, see Bickis and Krewski (1986) and Westfall and Lin (1988). In step 2, the exact convolution distribution is obtained for the trend statistic summed over all strata having totals that meet the threshold criterion. This distribution is obtained by applying the fast Fourier transform to the exact withinstratum distributions. A description of this general method can be found in Pagano and Tritchler (1983) and Good (1987).
The convolution distribution of the overall trend is then computed by convolving the exact distribution with the distribution of the continuitycorrected standard normal approximation. To be more specific, let denote the subset of stratum indices that satisfy the threshold criterion, and let denote the subset of indices that do not satisfy the criterion. Let denote the combined trend statistic from the set , which has an exact distribution obtained from Fourier analysis as previously outlined, and let denote the combined trend statistic from the set . Then the distribution of the overall trend is obtained by convolving the analytic distribution of with the continuitycorrected normal approximation for . Using the notation from the section ZScore Approximation, this convolution can be written as






where Z is a standard normal random variable, and
In this expression, the summation of s in is over , and c is the continuity correction discussed under the Zscore approximation.
When a twotailed test is requested, the expected trend is computed
The twotailed pvalue is reported as the permutation tail probability for the observed trend plus the permutation tail probability for , the reflected trend.
For this test, the contrast trend coefficients are centered to the values , where , , and G is the number of groups. The numerator of this test statistic is
where the weights take on three different types of values depending upon your specification of the WEIGHT= option in the STRATA statement. The default value is the withinstrata sample size , ensuring comparability with the ordinary CA trend statistic. WEIGHT=HARMONIC sets equal to the harmonic mean
where is the number of nonmissing groups and the summation is over only the nonmissing elements. The harmonic means analysis places more weight on the smaller sample sizes than does the default sample size method, and is similar to a Type 2 analysis in PROC GLM. WEIGHT=EQUAL sets for all v and s, and is similar to a Type 3 analysis in PROC GLM.
The function is the double arcsine transformation:
The variance estimate is
The test statistic is
The FreemanTukey transformation and its variance are described by Freeman and Tukey (1950) and Miller (1978). Since its variance is not weighted by the pooled probabilities, as is the CA test, the FT test can be more useful than the CA test for tests involving only a subset of the groups.
The Peto test is a modified CochranArmitage procedure incorporating mortality and prevalence information. The Peto test is computed like two CochranArmitage Zscore approximations, one for prevalence and one for mortality (Peto et al., 1980). It represents a special case in PROC MULTTEST because the data structure requirements are different, and the resampling methods used for adjusting pvalues are not valid. The TIME= option variable is required to specify “death” times or, more generally, times of occurrence. In addition, the test variables must assume one of the following three values:
0 = no occurrence
1 = incidental occurrence
2 = fatal occurrence
Use the TIME= option variable to define the mortality strata, and use the STRATA statement variable to define the prevalence strata.
In the following notation, the subscript v represents the variable, g represents the treatment group, s represents the stratum, and t represents the time. Recall that a plus sign in a subscript location denotes summation over that subscript.
Let be the number of incidental occurrences, and let be the total sample size for variable v in group g, stratum s, excluding fatal tumors.
Let be the number of fatal occurrences in time period t, and let be the number of patients alive at the end of time t – 1.
The pooled probability estimates are given by






The expected values are






Let denote a contrast trend coefficient, and define the numerator terms as follows:






Define the denominator variance terms by using the binomial variance:






The hypergeometric variances (the default) are calculated by weighting the withinstrata variances as discussed in the section ZScore Approximation.
The Peto statistic is computed as



where c is a continuity correction. The pvalue is determined from the standard normal distribution unless the PERMUTATION=number option is used. When you use the PERMUTATION= option for PETO in the TEST statement, PROC MULTTEST computes the “discrete approximation” permutation distribution described by Mantel (1980) and Soper and Tonkonoh (1993). Specifically, the permutation distribution of is computed, assuming that and are independent over all s and t. Note that the contrast trend coefficients must be integer valued. The pvalues are exact under this independence assumption. However, the independence assumption is valid only asymptotically, which is why these pvalues are called “approximate.”
An exact permutation distribution is available only under the assumption of equal risk of censoring in all treatment groups; even then, computing this distribution can be cumbersome. Soper and Tonkonoh (1993) describe situations where the discrete approximation distribution closely fits the exact permutation distribution.
The CONTRAST statement in PROC MULTTEST enables you to compute Fisher exact tests for twogroup comparisons. No stratification variable is allowed for this test. Note, however, that the FISHER exact test is a special case of the exact permutation tests performed by PROC MULTTEST and that these permutation tests allow a stratification variable. Recall that contrast coefficients can be –1, 0, or 1 for the Fisher test. The frequencies and sample sizes of the groups scored as –1 are combined, as are the frequencies and sample sizes of the groups scored as 1. Groups scored as 0 are excluded. The –1 group is then compared with the 1 group by using the Fisher exact test.
Letting x and m denote the frequency and sample size of the 1 group, and letting y and n denote those of the –1 group, the pvalue is calculated as
where X and Y are independent binomially distributed random variables with sample sizes m and n and common probability parameters. The hypergeometric distribution is used to determine the stated probability; Yates (1984) discusses this technique. PROC MULTTEST computes the twotailed pvalues by adding probabilities from both tails of the hypergeometric distribution. The first tail is from the observed x and y, and the other tail is chosen so that the resulting probability is as large as possible without exceeding the probability from the first tail. If the variable being tested has only one level, then the pvalue is set to 1.
For continuous variables, PROC MULTTEST automatically centers the contrast trend coefficients, as in the FreemanTukey test. These centered coefficients are then used to form a t statistic contrasting the withingroup means. Let denote the sample size within group g and stratum s; it depends on variable v only when there are missing values. Determine the weights as in the FreemanTukey test with replacing . Define
as the sample mean within a groupandstratum combination, and let denote the treatment means. Write the null hypothesis as
Also define
as the pooled sample variance.
Assuming constant variance for all groupandstratum combinations, the t statistic for the mean is
Then under the null hypothesis and assuming normality, independence, and homoscedasticity, follows a t distribution with degrees of freedom.
Whenever a denominator of 0 is computed, the pvalue is set to 1. When missing data force , the contribution to the denominator of the pooled variance is 0 and not –1. This is also true for the degrees of freedom.
If you do not assume constant variance for all groupandstratum combinations, then the approximate t test is
Under the null hypothesis and assuming normality and independence, the Satterthwaite (1946) approximation for the degrees of freedom of the t test is given by
under the restriction .
Whenever a denominator of 0 for is computed, the pvalue is set to 1. If the denominator for is computed as 0, then set . When missing data force , that groupandstratum combination does not contribute to the computation.