Interrater reliability or Kappa Statistic in excel

[Dohko]

Anyone know how to do this?

 

[shg]

How is your data arranged?

 

[Dohko]

Something like this.

Progress TOTAL ABC CDE EFG HIG
Cases Entered 1166 150 161 478 149
Cases Re-Adju 3 0 0 3 0
Agreement 3 0 0 3 0
Disagreement 0 0 0 0 0
KAPPA

 

[shg]

That's not much to work with for agreements and disagreements.

This would be pair-wise between each pair of raters?

       ------A------ --B-- ----C----- ---D--- ---E--- ---F---
   1                 TOTAL     A         B       C       D   
   2   Cases Entered  938        150     161     478     149 
   3   Cases Re-Adj     3        -       -         3     -   
   4   Agreement        3        -       -         3     -   
   5   Disagreement   -          -       -       -       -   
   6                                                         
   7                                                         
   8                           A         B       C       D   
   9                   A              k(A, B) k(A, C) k(A, D)
  10                   B                      k(B, C) k(B, D)
  11                   C                              k(C, D)

 

[Dohko]

Yes there's is not a lot of agreement at the moment.
But I just need a formula that would work when the whole table is eventually populated.

In total there will be 2 raters, and 16 categories.

 

[shg]

Function Kappa(YY As Long, YN As Long, _
               NY As Long, NN As Long) As Double
    ' Understanding Interobserver Agreement: The Kappa Statistic, Viera & Garrett
    ' [URL]http://widhiarso.staff.ugm.ac.id/files/memahami_kesepakatan_antar_rater_dengan_koefisien_kappa_(interobserver_agreement).pdf[/URL]
 
    Dim Pa          As Double
    Dim Pe          As Double
    Dim iTot        As Long
 
    iTot = YY + YN + NY + NN
    Pa = (YY + NN) / iTot
    Pe = ((YY + YN) * (YY + NY) + (NY + NN) * (YN + NN)) / iTot ^ 2
    Kappa = (Pa - Pe) / (1 - Pe)
End Function

      -A- B -C- -D- -E-- -----------F-----------
  1         Bob Bob                             
  2          Y   N                              
  3   Ali Y  15   5                             
  4   Ali N  10  70 0.57 E4: =Kappa(C3,D3,C4,D4)

 

[Dohko]

So can I do it straight from the data I first posted?
and then just call the Kappa function?

 

[shg]

You would do it as shown in the example.

 

[Dohko]

Actually what I need is a lot more complicated than that.

I have the following data..

UID Study Name Original Subtype Readj Study Subtype
701195000 GEOS Other Cause Evident Other Cause Evident
701514000 GEOS Other Cause Evident Other Cause Evident
702409000 GEOS Other Cause Probable Other Cause Probable
232323233 GEOS Other2 Other3

In this case 3 studies agree and one does not.
How can I calculate the Kappa for this?

Do I have to set up a big matrix with all the possible studies?

 

[shg]

As I (recently) understand it, Kappa is a measure of agreement between two raters based on a four-number matrix as in the example I posted. I don't see any numbers at all in that example, or any indication of who the raters are.