In order to correctly compute agreement statistics, the table must be square and row labels must match corresponding column labels. The table will not be square if, for example, one rater never uses one of the categories of a rating scale but the other rater uses all of the categories. This implies that a row or column of the table contains all zero counts. Even if the data contains observations for such rows or columns, the default behavior of PROC FREQ is to delete all-zero rows or columns before computations are done. The result is a nonsquare table and this message in the SAS log:

NOTE: AGREE statistics are computed only for tables where the number of rows equals the number of columns. To include zero-weight observations in the analysis, use the ZERO option in the WEIGHT statement.

Note that if both raters do not use one category but the unused categories are not the same one, then the table after deleting zero rows and columns will be square and agreement statistics will be computed. However, these statistics are not correct because the rows and columns do not match. You should always check the table to verify that the rows and columns match before using the statistics.

To compute agreement statistics in these cases, the zero rows or columns must be retained in the table. With raw data (one observation per subject being rated), an unused category simply means that there are no observations that would fall in a particular row or column. So, PROC FREQ has no way to know that such rows or columns should be included. You can make the table square by adding an observation to at least one cell in each missing category. Additionally, you need to use the WEIGHT statement with the weights of the original observations set to 1. Beginning with SAS 9, set the weights of the added observations to zero and use the ZEROS option in the WEIGHT statement. Prior to SAS 9, set the weights of the added observations to something very small. Both are illustrated in the example below. Note that the small weight method will not allow computation of an exact test for Kappa via the EXACT statement because exact computations are not available with non-integer weights.

In some cases, summarized data (one observation per cell of the table) is provided. Again, it might be that there are no observations in such data for cells in an unused category's row or column and an observation with zero or small weight, as above, must be added. However, if all cells of the square table are in the data with the missing category represented by all of its cells having zero weight, then you need only add the ZEROS option in the WEIGHT statement (SAS 9 or later), or change one of them to a small value (prior to SAS 9).

This example illustrates the process of making the table square when one rater does not use one category of the rating scale. The following DATA step creates some simulated raw data in which two raters rate 50 subjects on a scale from 1 to 4. In this simple example, the raters' ratings are completely independent of each other. The important feature is that Rater2 has zero probability of using category 3 while Rater1 has equal probabilities of using all of the categories. As a result, there are no 3 ratings from Rater2, and this results in a nonsquare table in PROC FREQ which prevents it from computing agreement statistics.

data a; do subject=1 to 50; rater1=rantbl(4198374,.25,.25,.25,.25); rater2=rantbl(4198374,.50,.25,0,.25); output; end; run; proc freq data=a; tables rater1*rater2 / agree nocol norow nopercent; run;

Table of rater1 by rater2 | ||||
---|---|---|---|---|

rater1 | rater2 | Total | ||

1 | 2 | 4 | ||

1 | 10 | 4 | 3 | 17 |

2 | 5 | 3 | 3 | 11 |

3 | 2 | 6 | 3 | 11 |

4 | 6 | 3 | 2 | 11 |

Total | 23 | 16 | 11 | 50 |

Beginning with SAS 9, you can make the table square by adding an observation with zero weight in each missing category. It is necessary to create an observation in only one cell of the category, not all cells. The following DATA steps create a single, zero-weight observation and adds it to the original data. All observations in the original data set are given a weight of 1.

data b; rater1=1; rater2=3; wt=0; output; run; data c; set a b; if wt=. then wt=1; run;

The analysis is done by using the WEIGHT statement and the ZEROS option, which tells PROC FREQ to allow zero-weighted cells to remain in the table. Notice that the table is now square and that the agreement statistics are computed.

proc freq data=c; weight wt / zeros; tables rater1*rater2 / agree nocol norow nopercent; run;

Table of rater1 by rater2 | |||||
---|---|---|---|---|---|

rater1 | rater2 | Total | |||

1 | 2 | 3 | 4 | ||

1 | 10 | 4 | 0 | 3 | 17 |

2 | 5 | 3 | 0 | 3 | 11 |

3 | 2 | 6 | 0 | 3 | 11 |

4 | 6 | 3 | 0 | 2 | 11 |

Total | 23 | 16 | 0 | 11 | 50 |

Kappa Statistics | ||||
---|---|---|---|---|

Statistic | Value | ASE | 95% Confidence Limits | |

Simple Kappa | 0.0342 | 0.0779 | -0.1185 | 0.1869 |

Weighted Kappa | 0.0592 | 0.0990 | -0.1349 | 0.2533 |

Prior to SAS 9, you can make the table square by adding an observation with
small, rather than zero, weight in each missing category. Again, it is only
necessary to create an observation in one cell of the category, not all cells.
The following DATA steps create a single observation with weight
1×10^{-20} and adds it to the original data. All observations in
the original data set are given a weight of 1.

data b; rater1=1; rater2=3; wt=1e-20; output; run; data c; set a b; if wt=. then wt=1; run;

The analysis is done by using the WEIGHT statement. Notice that the table is now square and that the agreement statistics are computed.

proc freq data=c; weight wt; tables rater1*rater2 / agree nocol norow nopercent; run;

Table of rater1 by rater2 | |||||
---|---|---|---|---|---|

rater1 | rater2 | Total | |||

1 | 2 | 3 | 4 | ||

1 | 10 | 4 | 1E-20 | 3 | 17 |

2 | 5 | 3 | 0 | 3 | 11 |

3 | 2 | 6 | 0 | 3 | 11 |

4 | 6 | 3 | 0 | 2 | 11 |

Total | 23 | 16 | 1E-20 | 11 | 50 |

Kappa Statistics | ||||
---|---|---|---|---|

Statistic | Value | ASE | 95% Confidence Limits | |

Simple Kappa | 0.0342 | 0.0779 | -0.1185 | 0.1869 |

Weighted Kappa | 0.0592 | 0.0990 | -0.1349 | 0.2533 |

Product Family | Product | System | SAS Release | |

Reported | Fixed* | |||

SAS System | SAS/STAT | All | n/a |

Type: | Usage Note |

Priority: | low |

Topic: | SAS Reference ==> Procedures ==> FREQ Analytics ==> Longitudinal Analysis Analytics ==> Nonparametric Analysis Analytics ==> Categorical Data Analysis Analytics ==> Descriptive Statistics Analytics ==> Exact Methods |

Date Modified: | 2016-06-01 17:03:32 |

Date Created: | 2002-12-16 10:56:40 |