Contextualising post-hoc tests following repeated one-way ANOVA

Question

I have a large dataset of 631 people that have been repeatedly measured across five different variables on some continuous parameter. The means (and SDs) for these variables are as follows:
Var1: 7.77 (3.8)
Var2: 6.87 (3.48)
Var3: 7.62 (3.83)
Var4: 3.96 (1.92)
Var5: 7.25 (3.62)

Having run a one-way repeated measures ANOVA, I have found a significant difference between them. Expecting this to be mainly a consequence of Var4, I ran post-hoc tests with Bonferroni correction. However, to my surprise, I have found a significant difference between each pairwise comparison. I'm obviously missing something but I find this really strange, especially considering the similarity between Vars 1 and 3.
I'm assuming (perhaps incorrectly) that the large size of the sample is the cause (?!?). If this seems plausible, is there a way I can adequately communicate that whilst there is a statistical significance between them, this doesn't relate to a practical difference? It seems incomplete to remark that there is a statistical difference when the means between them are so similar.

rep_ho · Accepted Answer

As you wrote, standard errors are very small, which is the reason why you are getting statistically significant results. The test is testing differences between means, not between whole distributions, so even a small numerical difference in means can be highly statistically significant if it's high with respect to standard errors of the mean.
95% confidence interval is roughly +/- 2SE, so the mean of Var1 is estimated to be 7.77 +/- 0.012, which doesn't even come close to Var3 7.62 +/- 0.006
Yes, as you correctly pointed out, statistical significance does not mean practical differences. In your case, even a small difference in means is statistically significant, simply because you have a lot of data.
The correct interpretation of your results is that you are quite sure that all of the means are different; however, you are also quite sure that their difference is small, except for Var 4.  In situations with wider standard errors, you cannot make this conclusion.

Contextualising post-hoc tests following repeated one-way ANOVA

One Answer

Add your own answers!

Ask a Question