Contents: | Purpose / History / Requirements / Usage / Details / References |
%inc "<location of your file containing the INTRACC macro>";
Following this statement, you may call the %INTRACC macro. See the Results tab for an example.
The following options are available. The TARGET=, RATER=, and DEPVAR= options are required.
If there are n targets and k ratings for each target, each target-rating occupies one observation, or in other words, there are n*k observations in the dataset.
The macro uses PROC GLM to break the total variability into that due to between targets, between judges, and residual. For the formulae which assume a one-way design, the SS and DF for between judges and residual are added to give simply a within-targets SS. This macro assumes that all targets are rated by judges numbered with the same judge numbers, even if they are not the same judges. In other words, each subject is rated by k judges, labeled, say, 1,2,...,k, even if they are not the same judges for each subject. That is so PROC GLM can break out a between judges SS.
In Shrout and Fleiss notation, the six correlations and their uses are as follows:
Winer, B.J. (1971), Statistical Principles in Experimental Design, New York: McGraw Hill.
These sample files and code examples are provided by SAS Institute Inc. "as is" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Recipients acknowledge and agree that SAS Institute shall not be liable for any damages whatsoever arising out of their use of this material. In addition, SAS Institute will provide no support for the materials contained herein.
These sample files and code examples are provided by SAS Institute Inc. "as is" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Recipients acknowledge and agree that SAS Institute shall not be liable for any damages whatsoever arising out of their use of this material. In addition, SAS Institute will provide no support for the materials contained herein.
data ratings; do product=1 to 5; do judge=1 to 3; input rating @@; output; end; end; datalines; 1 1 5 3 2 6 5 3 7 7 4 8 9 5 9 ; /* Define the INTRACC macro */ %inc "<location of your file containing the INTRACC macro>"; %intracc(depvar=rating,target=product,rater=judge,nrater=10); %intracc(data=ratings,depvar=rating,target=product,rater=judge, print=3,out=intclcor);
Intraclass Correlations for Inter-Rater Reliability Calculate all reliabilities in one fell swoop _NAME_ msw msb wms ems edf bms bdf jms jdf k theta rating 4.66667 13.3333 4.66667 0.83333 8 13.3333 4 20 2 3 0.61905 Winer Winer Winer reliability: reliability: Shrout-Fleiss Shrout-Fleiss reliability: mean of mean of 10 reliability: reliability: _NAME_ single score k scores scores single score random set rating 0.38235 0.65 0.86093 0.38235 0.47170 Shrout-Fleiss Shrout-Fleiss Shrout-Fleiss Shrout-Fleiss reliability: reliability: rel: rand set rel: fxd set _NAME_ fixed set mean k scores mean k scrs mean k scrs rating 0.83333 0.65 0.72816 0.9375These results come from the second call to %INTRACC:
Intraclass Correlations for Inter-Rater Reliability The GLM Procedure Class Level Information Class Levels Values product 5 1 2 3 4 5 judge 3 1 2 3 Number of Observations Read 15 Number of Observations Used 15 Intraclass Correlations for Inter-Rater Reliability The GLM Procedure Dependent Variable: rating Sum of Source DF Squares Mean Square F Value Pr > F Model 6 93.3333333 15.5555556 18.67 0.0003 Error 8 6.6666667 0.8333333 Corrected Total 14 100.0000000 R-Square Coeff Var Root MSE rating Mean 0.933333 18.25742 0.912871 5.000000 Source DF Type I SS Mean Square F Value Pr > F product 4 53.33333333 13.33333333 16.00 0.0007 judge 2 40.00000000 20.00000000 24.00 0.0004 Source DF Type III SS Mean Square F Value Pr > F product 4 53.33333333 13.33333333 16.00 0.0007 judge 2 40.00000000 20.00000000 24.00 0.0004 Intraclass Correlations for Inter-Rater Reliability Statistics from 2-way ANOVA w/o Interaction Obs _NAME_ _SOURCE_ _TYPE_ DF SS F PROB 1 rating ERROR ERROR 8 6.6667 . . 2 rating judge SS1 2 40.0000 24 .000416493 3 rating judge SS3 2 40.0000 24 .000416493 4 rating product SS1 4 53.3333 16 .000694339 5 rating product SS3 4 53.3333 16 .000694339 Intraclass Correlations for Inter-Rater Reliability Calculate all reliabilities in one fell swoop _NAME_ msw msb wms ems edf bms bdf jms jdf k rating 4.66667 13.3333 4.66667 0.83333 8 13.3333 4 20 2 3 Winer Winer reliability: Shrout-Fleiss Shrout-Fleiss reliability: mean of reliability: reliability: _NAME_ theta single score k scores single score random set rating 0.61905 0.38235 0.65 0.38235 0.47170 Shrout-Fleiss Shrout-Fleiss Shrout-Fleiss Shrout-Fleiss reliability: reliability: rel: rand set rel: fxd set _NAME_ fixed set mean k scores mean k scrs mean k scrs rating 0.83333 0.65 0.72816 0.9375
In data set TABLE1, each score on page 293 of the article is referenced by a JUDGE / PAIR combination. This is the format that PROC GLM will need. The data set will be transposed later for PROC CORR.
data table1; length pair $ 6; input judges $ 1-7 @@; do pair='first','second','third','fourth','fifth','sixth'; input score @; output; end; datalines; France 5.9 5.8 5.5 5.6 5.4 5.5 Czech 5.9 5.7 5.7 5.8 5.5 5.2 Austl 5.9 5.8 5.7 5.6 5.4 5.2 USA 5.9 5.7 5.8 5.7 5.4 5.3 Germany 5.8 5.7 5.3 5.4 5.4 5.1 Canada 5.8 5.7 5.6 5.6 5.4 5.4 Italy 5.9 5.8 5.5 5.3 5.4 5.2 Unified 5.9 5.8 5.7 5.6 5.4 5.2 UK 5.9 5.8 5.3 5.7 5.5 5.3 ; proc print noobs; title 'Interrater Reliability with the %INTRACC macro'; run;
The following statements invoke the %INTRACC macro and compute the following:
/* Define the INTRACC macro */ %inc "<location of your file containing the INTRACC macro>"; %intracc(data=table1,target=pair,depvar=score,rater=judges,print=3);
Abbreviated output from the %INTRACC macro call follows:
Interrater Reliability with the %INTRACC macro Intraclass Correlations for Inter-Rater Reliability The GLM Procedure Dependent Variable: score Sum of Source DF Squares Mean Square F Value Pr > F Model 13 2.35796296 0.18138177 16.14 <.0001 Error 40 0.44962963 0.01124074 Corrected Total 53 2.80759259 R-Square Coeff Var Root MSE score Mean 0.839852 1.900168 0.106022 5.579630 Source DF Type I SS Mean Square F Value Pr > F pair 5 2.18537037 0.43707407 38.88 <.0001 judges 8 0.17259259 0.02157407 1.92 0.0838 Source DF Type III SS Mean Square F Value Pr > F pair 5 2.18537037 0.43707407 38.88 <.0001 judges 8 0.17259259 0.02157407 1.92 0.0838 Interrater Reliability with the %INTRACC macro Intraclass Correlations for Inter-Rater Reliability Calculate all reliabilities in one fell swoop _NAME_ msw msb wms ems edf bms bdf jms jdf score 0.012963 0.43707 0.012963 0.011241 40 0.43707 5 0.021574 8 Winer Winer reliability: Shrout-Fleiss Shrout-Fleiss reliability: mean of reliability: reliability: _NAME_ k theta single score k scores single score random set score 9 3.63524 0.78426 0.97034 0.78426 0.78495 Shrout-Fleiss Shrout-Fleiss Shrout-Fleiss Shrout-Fleiss reliability: reliability: rel: rand set rel: fxd set _NAME_ fixed set mean k scores mean k scrs mean k scrs score 0.80803 0.97034 0.97046 0.97428
Compute (2b) from the output data set. (3b) as above can be computed here too.
data alpha; set _stats_; if upcase(_source_)='JUDGES' and _type_='SS1' then do; alpha=1-(1/f); stat='reliability/consistency across skaters'; output; end; if upcase(_source_)='PAIR' and _type_='SS1' then do; alpha=1-(1/f); stat='reliability/consistency across judges'; output; end; run; proc print data=alpha noobs; var alpha stat; title; run;The above statements produce the following output:
alpha stat 0.47897 reliability/consistency across skaters 0.97428 reliability/consistency across judges
title 'Another approach pp. 294-5: Transpose and use PROC CORR ALPHA'; proc transpose data=table1 out=out(where=(upcase(_name_) eq 'SCORE')); id pair; by judges notsorted; run; proc print data=out noobs; title2 'examine the skaters'; run; proc corr alpha data=out; var first -- sixth; run;The above statements produce the following abbreviated output:
Another approach pp. 294-5: Transpose and use PROC CORR ALPHA examine the skaters judges _NAME_ first second third fourth fifth sixth France score 5.9 5.8 5.5 5.6 5.4 5.5 Czech score 5.9 5.7 5.7 5.8 5.5 5.2 Austl score 5.9 5.8 5.7 5.6 5.4 5.2 USA score 5.9 5.7 5.8 5.7 5.4 5.3 Germany score 5.8 5.7 5.3 5.4 5.4 5.1 Canada score 5.8 5.7 5.6 5.6 5.4 5.4 Italy score 5.9 5.8 5.5 5.3 5.4 5.2 Unified score 5.9 5.8 5.7 5.6 5.4 5.2 UK score 5.9 5.8 5.3 5.7 5.5 5.3 Cronbach Coefficient Alpha Variables Alpha ---------------------------- Raw 0.478970 Standardized 0.537838
proc sort data=table1 out=table1; by pair; run; proc transpose data=table1 out=out(where=(upcase(_name_) eq 'SCORE')); id judges; by pair; run; proc print data=out noobs; title2 'examine the judges'; run; proc corr alpha data=out; var france -- uk; run;The above statements produce the following abbreviated output:
Another approach pp. 294-5: Transpose and use PROC CORR ALPHA examine the judges pair _NAME_ France Czech Austl USA Germany Canada Italy Unified UK fifth score 5.4 5.5 5.4 5.4 5.4 5.4 5.4 5.4 5.5 first score 5.9 5.9 5.9 5.9 5.8 5.8 5.9 5.9 5.9 fourth score 5.6 5.8 5.6 5.7 5.4 5.6 5.3 5.6 5.7 second score 5.8 5.7 5.8 5.7 5.7 5.7 5.8 5.8 5.8 sixth score 5.5 5.2 5.2 5.3 5.1 5.4 5.2 5.2 5.3 third score 5.5 5.7 5.7 5.8 5.3 5.6 5.5 5.7 5.3 Cronbach Coefficient Alpha Variables Alpha ---------------------------- Raw 0.974282 Standardized 0.977863
Right-click on the link below and select Save to save
the %INTRACC macro definition
to a file. It is recommended that you name the file
intracc.sas
.
Type: | Sample |
Topic: | Analytics ==> Nonparametric Analysis SAS Reference ==> Procedures ==> GLM Analytics ==> Descriptive Statistics Analytics ==> Regression Analytics ==> Longitudinal Analysis SAS Reference ==> Procedures ==> CORR Analytics ==> Analysis of Variance |
Date Modified: | 2007-08-14 03:03:11 |
Date Created: | 2005-01-17 08:41:30 |
Product Family | Product | Host | SAS Release | |
Starting | Ending | |||
SAS System | SAS/STAT | All | n/a | n/a |