Stat Computing > SAS > FAQ

SAS FAQ
How can I calculate a kappa statistic for variables with unequal score ranges?

Suppose we would like to compare two raters using a kappa statistic but the raters have different range of scores. This situation most often presents itself where one of the raters did not use the same range of scores as the other rater. 

Let us consider an example where two graduate students where asked to rate 12 movies based on a scale from 1-3. One rater used all of the three scores possible while rating the movies whereas the other student did not like any of the movies and therefore rated all of them as either a 1 or a 2. Thus, the range of scores is the not the same for the two raters.

To obtain the kappa statistic in SAS we are going to use proc freq with the test kappa statement. By default, SAS will only compute the kappa statistics if the two variables have exactly the same categories, which is not the case in this particular instance. We can get around this problem by adding a fake observation and a weight variable shown below. The weight variable takes value of 1 for all the real observations and value of 0 for the fake observation that we have just added. The trick is then to use the option "zeros" for the weight statement.

data kappa;
input  rater1    rater2;
datalines;
  1         1
  1         1
  1         1
  1         1
  2         2
  2         2
  2         2
  2         2
  3         2
  3         2
  3         2
  3         2
;
run;
data kappa;
  if _n_=1 then do;
  rater1 =3;
  rater2 = 3;
  weight = 0;
  output;
  end; 
  set kappa;
  weight = 1;
  output;
run;
proc freq data= kappa;
  tables rater1*rater2;
  test kappa;
  weight weight / zeros;
run;

Statistics for Table of rater1 by rater2

    Simple Kappa Coefficient
--------------------------------
Kappa                     0.5000
ASE                       0.1559
95% Lower Conf Limit      0.1944
95% Upper Conf Limit      0.8056

     Test of H0: Kappa = 0

ASE under H0              0.1667
Z                         3.0000
One-sided Pr >  Z         0.0013
Two-sided Pr > |Z|        0.0027


   Weighted Kappa Coefficient
--------------------------------
Weighted Kappa            0.5714
ASE                       0.1323
95% Lower Conf Limit      0.3122
95% Upper Conf Limit      0.8307

Sample Size = 12

UCLA Researchers are invited to our Statistical Consulting Services
We recommend others to our list of Other Resources for Statistical Computing Help
These pages are Copyrighted (c) by UCLA Academic Technology Services