The FREQ Procedure

Tests and Measures of Agreement

When you specify the AGREE option in the TABLES statement, PROC FREQ computes tests and measures of agreement for square tables (that is, for tables where the number of rows equals the number of columns). For two-way tables, these tests and measures include McNemar’s test for $2 \times 2$ tables, Bowker’s test of symmetry, the simple kappa coefficient, and the weighted kappa coefficient. For multiple strata (n-way tables, where n > 2), PROC FREQ also computes the overall simple kappa coefficient and the overall weighted kappa coefficient, and tests for equal kappas (simple and weighted) among strata. Cochran’s Q is computed for multiway tables when each variable has two levels, that is, for $h \times 2 \times 2$ tables.

PROC FREQ computes the kappa coefficients (simple and weighted), their asymptotic standard errors, and their confidence limits when you specify the AGREE option in the TABLES statement. If you also specify the KAPPA option in the TEST statement, PROC FREQ computes the asymptotic test of the hypothesis that simple kappa equals 0. Similarly, if you specify the WTKAP option in the TEST statement, PROC FREQ computes the asymptotic test for weighted kappa.

In addition to the asymptotic tests described in this section, PROC FREQ provides exact p-values for McNemar’s test, the simple kappa coefficient test, and the weighted kappa coefficient test. You can request these exact tests by specifying the corresponding options in the EXACT statement. See the section Exact Statistics for more information.

The following sections provide the formulas that PROC FREQ uses to compute the AGREE statistics. For information about the use and interpretation of these statistics, see Agresti (2002, 2007); Fleiss, Levin, and Paik (2003), and the other references cited for each statistic.

McNemar’s Test

PROC FREQ computes McNemar’s test (McNemar 1947) for $2 \times 2$ tables when you specify the AGREE option. This test is appropriate when you are analyzing data from matched pairs of subjects with a dichotomous (yes-no) response. By default, the null hypothesis for McNemar’s test is marginal homogeneity, which can be expressed as $p_{1 \cdot } = p_{\cdot 1}$; this is equivalent to a discordant proportion ratio ($p_{12}/p_{21}$) of 1. The corresponding test statistic is computed as

\[ Q_{\mi{M}} = (n_{12}-n_{21})^{2} ~ / ~ (n_{12} + n_{21}) \]

Under the null hypothesis, $Q_{\mi{M}}$ has an asymptotic chi-square distribution with 1 degree of freedom.

Optionally, you can specify the null ratio of discordant proportions ($p_{12} / p_{21}$) in the AGREE(MNULLRATIO=) option. When the null ratio is r, McNemar’s test is computed as

\[ Q_{\mi{M}}(r) = (n_{12}-e_{12})^2) / e_{12} ~ + ~ (n_{21}-e_{21})^2 / e_{21} \]

where $e_{12} = D / ( 1 + 1/r )$, $e_{21} = D / ( 1 + r )$, and D is the number of discordant pairs, ($n_{12} + n_{21}$). Under the null hypothesis, $Q_{\mi{M}}(r)$ has an asymptotic chi-square distribution with 1 degree of freedom.

PROC FREQ also computes an exact p-value for McNemar’s test when you specify the MCNEM option in the EXACT statement.

Bowker’s Test of Symmetry

For Bowker’s test of symmetry, the null hypothesis is that the cell proportions are symmetric, or that $p_{ij} = p_{ji}$ for all pairs of table cells. For $2 \times 2$ tables, Bowker’s test is identical to McNemar’s test, and so PROC FREQ provides Bowker’s test for square tables larger than $2 \times 2$.

Bowker’s test of symmetry is computed as

\[ Q_{\mi{B}} = \mathop {\sum \! \sum }_{i < j}~ (n_{ij}-n_{ji})^2 ~ / ~ (n_{ij}+n_{ji}) \]

For large samples, $Q_{\mi{B}}$ has an asymptotic chi-square distribution with $R(R-1)/2$ degrees of freedom under the null hypothesis of symmetry. For more information, see Bowker (1948).

Simple Kappa Coefficient

The simple kappa coefficient, introduced by Cohen (1960), is a measure of interrater agreement. PROC FREQ computes the simple kappa coefficient as

\[ \hat{\kappa } = \left( P_ o - P_ e \right) ~ / ~ \left( 1-P_ e \right) \]

where $P_ o = \sum _ i p_{ii}$ and $P_ e = \sum _ i p_{i.} p_{.i}$. If the two response variables are viewed as two independent ratings of the n subjects, the kappa coefficient equals +1 when there is complete agreement of the raters. When the observed agreement exceeds chance agreement, kappa is positive, with its magnitude reflecting the strength of agreement. Although this is unusual in practice, kappa is negative when the observed agreement is less than chance agreement. The minimum value of kappa is between –1 and 0, depending on the marginal proportions.

The asymptotic variance of the simple kappa coefficient is computed as

\[ \mr{Var}(\hat{\kappa }) = \left( A + B - C \right) ~ / ~ (1-P_ e)^2~ n \]

where

\begin{eqnarray*} A & = & \sum _ i p_{ii} \left( 1-(p_{i \cdot } + p_{\cdot i})(1-\hat{\kappa }) \right)^2 \\[.08in] B & = & (1-\hat{\kappa })^2 ~ \mathop {\sum \! \sum }_{i \neq j}~ p_{ij} (p_{\cdot i} + p_{j \cdot })^2 \\[.10in] C & = & \left( ~ \hat{\kappa } - P_ e (1-\hat{\kappa }) ~ \right)^2 \end{eqnarray*}

For more information, see Fleiss, Cohen, and Everitt (1969).

ROC FREQ computes confidence limits for the simple kappa coefficient as

\[ \hat{\kappa } ~ \pm ~ \bigl ( ~ z_{\alpha /2} \times \sqrt {\mr{Var}(\hat{\kappa })} ~ \bigr ) \]

where $z_{\alpha /2}$ is the $100(1-\alpha /2)$th percentile of the standard normal distribution. The value of $\alpha $ is determined by the value of the ALPHA= option, which, by default, is 0.05 and produces 95% confidence limits.

To compute an asymptotic test for the kappa coefficient, PROC FREQ uses the standardized test statistic $\hat{\kappa }^\ast $, which has an asymptotic standard normal distribution under the null hypothesis that kappa equals 0. The standardized test statistic is computed as

\[ \hat{\kappa }^\ast = \hat{\kappa } ~ / ~ \sqrt {\mr{Var}_0(\hat{\kappa })} \]

where $\mr{Var}_0(\hat{\kappa })$ is the variance of the kappa coefficient under the null hypothesis,

\[ \mr{Var}_0(\hat{\kappa }) = \left( P_ e + P_ e^2 - \sum _ i p_{i \cdot } p_{\cdot i} (p_{i \cdot } + p_{\cdot i}) \right) ~ / ~ (1 - P_ e)^2 ~ n \]

For more information, see Fleiss, Levin, and Paik (2003).

PROC FREQ also provides an exact test for the simple kappa coefficient. You can request the exact test by specifying the KAPPA or AGREE option in the EXACT statement. See the section Exact Statistics for more information.

Weighted Kappa Coefficient

The weighted kappa coefficient is a generalization of the simple kappa coefficient that uses weights to quantify the relative difference between categories. For $2 \times 2$ tables, the weighted kappa coefficient equals the simple kappa coefficient. PROC FREQ displays the weighted kappa coefficient only for tables larger than $2 \times 2$. PROC FREQ computes the kappa weights from the column scores, by using either Cicchetti-Allison weights or Fleiss-Cohen weights, both of which are described in the following section. The weights $w_{ij}$ are constructed so that $0 \leq w_{ij}<1$ for all $i \not= j$, $w_{ii} = 1$ for all i, and $w_{ij} = w_{ji}$. The weighted kappa coefficient is computed as

\[ \hat{\kappa }_ w = \left( P_{o(w)} - P_{e(w)} \right) ~ / ~ \left( 1-P_{e(w)} \right) \]

where

\[ P_{o(w)} = \sum _ i \sum _ j w_{ij} p_{ij} \]
\[ P_{e(w)} = \sum _ i \sum _ j w_{ij} p_{i \cdot } p_{\cdot j} \]

The asymptotic variance of the weighted kappa coefficient is

\[ \mr{Var}(\hat{\kappa }_ w) = \left( \sum _ i\sum _ j p_{ij} \left( w_{ij}-(\overline{w}_{i \cdot }+\overline{w}_{\cdot j}) (1-\hat{\kappa }_ w) \right)^2 - \left( \hat{\kappa }_ w - P_{e(w)}(1-\hat{\kappa }_ w) \right)^2 \right) ~ / ~ (1-P_{e(w)})^2~ n \]

where

\[ \overline{w}_{i \cdot } = \sum _ j p_{\cdot j}w_{ij} \]
\[ \overline{w}_{\cdot j} = \sum _ i p_{i \cdot }w_{ij} \]

For more information, see Fleiss, Cohen, and Everitt (1969).

PROC FREQ computes confidence limits for the weighted kappa coefficient as

\[ \hat{\kappa }_ w ~ \pm ~ \bigl ( ~ z_{\alpha /2} \times \sqrt {\mr{Var}(\hat{\kappa }_ w)} ~ \bigr ) \]

where $z_{\alpha /2}$ is the $100(1-\alpha /2)$th percentile of the standard normal distribution. The value of $\alpha $ is determined by the value of the ALPHA= option, which, by default, is 0.05 and produces 95% confidence limits.

To compute an asymptotic test for the weighted kappa coefficient, PROC FREQ uses the standardized test statistic $\hat{\kappa }_ w^\ast $, which has an asymptotic standard normal distribution under the null hypothesis that weighted kappa equals 0. The standardized test statistic is computed as

\[ \hat{\kappa }_ w^\ast = \hat{\kappa }_ w ~ / ~ \sqrt {\mr{Var}_0(\hat{\kappa }_ w)} \]

where $\mr{Var}_0(\hat{\kappa }_ w)$ is the variance of the weighted kappa coefficient under the null hypothesis,

\[ \mr{Var}_0(\hat{\kappa }_ w) = \left( \sum _ i \sum _ j p_{i \cdot } p_{\cdot j} \left( w_{ij} - (\overline{w}_{i \cdot } + \overline{w}_{\cdot j}) \right) ^2 - P_{e(w)}^2 \right) ~ / ~ (1 - P_{e(w)})^2~ n \]

For more information, see Fleiss, Levin, and Paik (2003).

PROC FREQ also provides an exact test for the weighted kappa coefficient. You can request the exact test by specifying the WTKAPPA or AGREE option in the EXACT statement. See the section Exact Statistics for more information.

Weights

PROC FREQ computes kappa coefficient weights by using the column scores and one of the two available weight types. The column scores are determined by the SCORES= option in the TABLES statement. The two available types of kappa weights are Cicchetti-Allison and Fleiss-Cohen weights. By default, PROC FREQ uses Cicchetti-Allison weights. If you specify (WT=FC) with the AGREE option, PROC FREQ uses Fleiss-Cohen weights to compute the weighted kappa coefficient.

PROC FREQ computes Cicchetti-Allison kappa coefficient weights as

\[ w_{ij} = 1 - \frac{|C_ i - C_ j|}{C_ C - C_1} \]

where $C_ i$ is the score for column i and C is the number of categories or columns. For more information, see Cicchetti and Allison (1971).

The SCORES= option in the TABLES statement determines the type of column scores used to compute the kappa weights (and other score-based statistics). The default is SCORES=TABLE. For more information, see the section Scores. For numeric variables, table scores are the values of the variable levels. You can assign numeric values to the levels in a way that reflects their level of similarity. For example, suppose you have four levels and order them according to similarity. If you assign them values of 0, 2, 4, and 10, the Cicchetti-Allison kappa weights take the following values: $ w_{12}$ = 0.8, $ w_{13}$ = 0.6, $ w_{14}$ = 0, $ w_{23}$ = 0.8, $ w_{24}$ = 0.2, and $ w_{34}$ = 0.4. Note that when there are only two categories (that is, C = 2), the weighted kappa coefficient is identical to the simple kappa coefficient.

If you specify (WT=FC) with the AGREE option in the TABLES statement, PROC FREQ computes Fleiss-Cohen kappa coefficient weights as

\[ w_{ij} = 1 - \frac{(C_ i - C_ j)^2}{(C_ C - C_1)^2} \]

For more information, see Fleiss and Cohen (1973).

For the preceding example, the Fleiss-Cohen kappa weights are: $ w_{12}$ = 0.96, $ w_{13}$ = 0.84, $ w_{14}$ = 0, $ w_{23}$ = 0.96, $ w_{24}$ = 0.36, and $ w_{34}$ = 0.64.

Overall Kappa Coefficient

When there are multiple strata, PROC FREQ combines the stratum-level estimates of kappa into an overall estimate of the supposed common value of kappa. Assume there are q strata, indexed by $h=1,2,\ldots , q$, and let $\mr{Var}(\hat{\kappa }_ h)$ denote the variance of $\hat{\kappa }_ h$. The estimate of the overall kappa coefficient is computed as

\[ \hat{\kappa }_{\mi{T}} = \sum _{h=1}^ q \frac{\hat{\kappa }_ h}{\mr{Var}(\hat{\kappa }_ h)} ~ / ~ \sum _{h=1}^ q \frac{1}{\mr{Var}(\hat{\kappa }_ h)} \]

For more information, see Fleiss, Levin, and Paik (2003).

PROC FREQ computes an estimate of the overall weighted kappa in the same way.

Tests for Equal Kappa Coefficients

When there are multiple strata, the following chi-square statistic tests whether the stratum-level values of kappa are equal:

\[ Q_{\mi{K}} = \sum _{h=1}^ q \left( \hat{\kappa }_ h - \hat{\kappa }_{\mi{T}} \right)^2 ~ / ~ {\mr{Var}(\hat{\kappa }_ h)} \]

Under the null hypothesis of equal kappas for the q strata, $Q_{\mi{K}}$ has an asymptotic chi-square distribution with q–1 degrees of freedom. See Fleiss, Levin, and Paik (2003) for more information. PROC FREQ computes a test for equal weighted kappa coefficients in the same way.

Cochran’s Q Test

Cochran’s Q is computed for multiway tables when each variable has two levels, that is, for $2 \times 2 \cdots \times 2$ tables. Cochran’s Q statistic is used to test the homogeneity of the one-dimensional margins. Let m denote the number of variables and N denote the total number of subjects. Cochran’s Q statistic is computed as

\[ Q_{\mi{C}} = m (m-1) \left( \sum _{j=1}^ m T_ j^2 - T^2 \right) ~ / ~ \left( mT - \sum _{k=1}^ N S_ k^2 \right) \]

where $T_ j$ is the number of positive responses for variable j, T is the total number of positive responses over all variables, and $S_ k$ is the number of positive responses for subject k. Under the null hypothesis, Cochran’s Q has an asymptotic chi-square distribution with m–1 degrees of freedom. For more information, see Cochran (1950). When there are only two binary response variables (m=2), Cochran’s Q simplifies to McNemar’s test. When there are more than two response categories, you can test for marginal homogeneity by using the repeated measures capabilities of the CATMOD procedure.

Tables with Zero-Weight Rows or Columns

The AGREE statistics are defined only for square tables, where the number of rows equals the number of columns; if a table is not square, PROC FREQ does not compute AGREE statistics for the table. In the kappa statistic framework, where two independent raters assign ratings to each of n subjects, suppose one of the raters does not use all possible r rating levels. If the corresponding table contains r rows but only r–1 columns, the table is not square and PROC FREQ does not compute AGREE statistics. To create a square table in this situation, you can use the ZEROS option in the WEIGHT statement, which includes zero-weight observations in the analysis. You can include zero-weight observations in the input data set to represent any rating levels that are not used by a rater, so that the input data set has at least one observation for each possible rater and rating combination. When you use this input data set and specify the ZEROS option, the analysis includes all rating levels (even when all levels are not actually assigned by both raters). The resulting table (of rater 1 by rater 2) is a square table, and AGREE statistics can be computed.

For more information, see the description of the ZEROS option in the WEIGHT statement. By default, PROC FREQ does not process observations that have weights of 0 because these observations do not contribute to the total frequency count, and because many of the tests and measures of association are undefined for tables that contain zero-weight rows or columns. However, kappa statistics are defined for tables that contain zero-weight rows or columns, and the ZEROS option enables you to input zero-weight observations and construct the tables needed to compute kappa statistics.