Calculating Significantly Above Chance for 3-AFC Test

Question

I am trying to figure out which participants who took a 18-item multiple choice test scored significantly above chance. Each item has 3 choices, so the probability of getting each question correct is 0.33. To know how many items correct a person would have to get in order to be significantly above chance, does it make sense to use a binomial probability calculator like this: https://stattrek.com/online-calculator/binomial.aspx and try different numbers of successes until the probability is < p=0.05 (assuming this is my significance criteria)?

For 18 items where the probability correct is 0.33, the probability of getting 9 correct is 0.061 and the probability of getting 10 correct is p=0.027, so does it make sense to use “10 items correct” as my criteria for “performed significantly above chance?”

cognitive psychology learning measurement statistics

For 18 items where the probability correct is 0.33, the probability of getting 9 correct is 0.061 and the probability of getting 10 correct is p=0.027, so does it make sense to use "10 items correct" as my criteria for "performed significantly above chance?"

Tony Mobbs · Answer

The hypothesis would normally be expressed as a one-sided test as p(X≥10). The online calculator you have selected is fine and the significant result could be expressed in an article as follows: (p=0.33, q=0.66, K=10, n=18, p-value = 0.043).

Another way to think of this is that 9 or fewer correct answers is more likely to be random chance.

At the threshold of 10, it is unlikely that the result could be explained by random chance.

Calculating Significantly Above Chance for 3-AFC Test

One Answer

Add your own answers!

Ask a Question