SAS FAQ
Kappa statistic for variables with unequal ranges of scores
Suppose we would like to compare two raters using a kappa statistic but the raters have
different range of scores. This situation most often presents itself where one of the raters did not
use the same range of scores as the other rater.
Let us consider an example where
two graduate students where asked to rate 12 movies based on a scale from 1-3. One rater
used all of the three scores possible while rating the movies whereas the other student did
not like any of the movies and therefore rated all of them as either a 1 or a 2. Thus,
the range of scores is the not the same for the two raters.In Stata
Using the kap command in Stata it is no problem that there is an unequal range of
scores for the two raters. The code to produce the kappa statistic would be:
kap rater1 rater2
Expected
Agreement Agreement Kappa Std. Err. Z Prob>Z
-----------------------------------------------------------------
66.67% 33.33% 0.5000 0.1667 3.00 0.0013
In SAS
To obtain the kappa statistic in SAS we have to use the proc freq with the
agree option in the tables statement. Unfortunately, SAS will only compute the
statistics from the agree option if the two variables have exactly the same
categories, which is not the case in this particular instance.
The solution to this problem is to create an extra observation, which will have the
missing category for rater2. Thus, in this example we will create observation
13 which will be equal to 3 for both raters and then rater2 will have scores ranging
from 1 to 3 just like rater1. Since we do not want this extra observation to influence
the kappa statistic we now create a weight variable which will give a very small weight
(i.e., weight = 0.001) to the extra observation and observations from the original data set are given
comparatively large weights (i.e., weight = 1). The following code will give us the result
that we want:
data kappa;
set 'd:\sas\data\kappa';
proc print data=kappa ;
run;
Obs rater1 rater2
1 1 1
2 1 1
3 1 1
4 1 1
5 2 2
6 2 2
7 2 2
8 2 2
9 3 2
10 3 2
11 3 2
12 3 2
data kappa;
if _n_=1 then do;
rater1 =3;
rater2 = 3;
end;
output;
set kappa;
run;
data kappa;
set kappa;
if rater2 ne 3 then weight = 1;
if rater2 = 3 then weight = 0.0001;
run;
proc print data=kappa;
run;
Obs rater1 rater2 weight
1 3 3 0.0001
2 1 1 1.0000
3 1 1 1.0000
4 1 1 1.0000
5 1 1 1.0000
6 2 2 1.0000
7 2 2 1.0000
8 2 2 1.0000
9 2 2 1.0000
10 3 2 1.0000
11 3 2 1.0000
12 3 2 1.0000
13 3 2 1.0000
proc freq data= kappa;
tables rater1*rater2;
test kappa;
weight weight;
run;
Statistics for Table of rater1 by rater2
Simple Kappa Coefficient
--------------------------------
Kappa 0.5000
ASE 0.1559
95% Lower Conf Limit 0.1944
95% Upper Conf Limit 0.8056
In SPSS
In SPSS we encounter the same difficulty as we saw in SAS, where the crosstab
command will not compute the kappa statistic when the raters do not have the same
range of scores. For SPSS version 11.0 and lower there is no way to obtain
the kappa statistic in this situation. For SPSS version 11.5 we can
actually obtain the kappa statistic and the solution is also very similar to the solution for SAS in that
we generate an extra observation, which will have the
missing category for rater2. Thus, in this example we will create observation
13 which will be equal to 3 for both raters and then rater2 will have scores ranging
from 1 to 3 just like rater1. Since we do not want this extra observation to influence
the kappa statistic we now create a weight variable which will give a very small weight
(i.e., weight = 0.001) to the extra observation and observations from the original data set are given
comparatively large weights (i.e., weight = 1).
First go into the data editor and manually add the extra observations to rater1 and rater2.
Then use the following code to generate the weight variable and compute the kappa statistic.
COMPUTE weight = 1.
IF rater2 = 3 weight = 0.001.
EXE.
WEIGHT
BY weight.
CROSSTABS
/TABLES=rater1 BY rater2
/STATISTIC=KAPPA.
UCLA Researchers are invited to our Statistical Consulting Services
We recommend others to our list of Other Resources for Statistical Computing Help
These pages are Copyrighted (c) by UCLA Academic Technology Services
The content of this web site should not be
construed as an endorsement of any particular web site, book, or software
product by the University of California
