Which correlation test can be used on low count contigency table?

Question

I need to find the correlation between the ratings given by two parties.

Let's say we have 2 coaches, namely A and B.

Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "Weak".
Coach B will rate the same 12 players.

If Coach A rated all the players as "Strong", and Coach B rated 10 players as "Strong" and 2 players as "Weak".

I have tried putting the results into a contingency table form and ran Somers D test. However the Somers D shows 0% which does not truly representing the correlation of the ratings. What test could we carry out to find the correlation between the ratings of the coaches?

Erwan · Answer

I don't think it's worth doing any kind of significance test for such a small sample.

option 1: pure categorical values

You could just count the number of identical ratings between the two ratings and divide by the total number of players

option 2: take the order into account: weak < moderate < strong

Define a similarity measure such as:

sim(X, X) = 1
sim(weak,moderate) = 0.5
sim(moderate,strong) = 0.5
sim(weak,strong) = 0

Then use the same process as option 1 except that for every player you add sim(ratingA, ratingB), and divide by the total. This will also give you a normalized score between 0 and 1 which is a bit more precise (the value will be higher or equal than in option 1).

Brian Spiering · Answer

You are describing the field of inter-rater reliability.
There are a variety of statistical tests for inter-rater reliability. One of the most common for that scenario is Cohen's kappa.

Which correlation test can be used on low count contigency table?

2 Answers

option 1: pure categorical values

option 2: take the order into account: weak < moderate < strong

Add your own answers!

Ask a Question