# Contextualising post-hoc tests following repeated one-way ANOVA

Cross Validated Asked by DC_Liv on November 17, 2020

I have a large dataset of 631 people that have been repeatedly measured across five different variables on some continuous parameter. The means (and SDs) for these variables are as follows:

Var1: 7.77 (3.8)
Var2: 6.87 (3.48)
Var3: 7.62 (3.83)
Var4: 3.96 (1.92)
Var5: 7.25 (3.62)


Having run a one-way repeated measures ANOVA, I have found a significant difference between them. Expecting this to be mainly a consequence of Var4, I ran post-hoc tests with Bonferroni correction. However, to my surprise, I have found a significant difference between each pairwise comparison. I’m obviously missing something but I find this really strange, especially considering the similarity between Vars 1 and 3.

I’m assuming (perhaps incorrectly) that the large size of the sample is the cause (?!?). If this seems plausible, is there a way I can adequately communicate that whilst there is a statistical significance between them, this doesn’t relate to a practical difference? It seems incomplete to remark that there is a statistical difference when the means between them are so similar.

As you wrote, standard errors are very small, which is the reason why you are getting statistically significant results. The test is testing differences between means, not between whole distributions, so even a small numerical difference in means can be highly statistically significant if it's high with respect to standard errors of the mean.

95% confidence interval is roughly +/- 2SE, so the mean of Var1 is estimated to be 7.77 +/- 0.012, which doesn't even come close to Var3 7.62 +/- 0.006

Yes, as you correctly pointed out, statistical significance does not mean practical differences. In your case, even a small difference in means is statistically significant, simply because you have a lot of data.

The correct interpretation of your results is that you are quite sure that all of the means are different; however, you are also quite sure that their difference is small, except for Var 4. In situations with wider standard errors, you cannot make this conclusion.

Correct answer by rep_ho on November 17, 2020