Between-subject design and within-subject anlyses

This is a crude exploration of my concern about false discovery:

Below I 'deconstruct' a 2-factor ANOVA into t tests, looking A vs B, x vs y, and interaction and finding nothing significant at the 5% level. Then looking at x vs y for B alone, I find a significant result.

set.seed(2020)
ax = rnorm(12,100, 15)
ay = rnorm(12,101, 15)
bx = rnorm(12,102, 15)
by = rnorm(12, 98, 15)

t.test(c(ax,by), c(ay,bx))$p.val # "int"
[1] 0.05200111
t.test(c(ax,bx), c(ay,by))$p.val # x vs y
[1] 0.3837236
t.test(c(ax,ay), c(bx,by))$p.val # a vs b
[1] 0.8644972
t.test(bx,by)$p.val              # x vs y for b
[1] 0.006613897                  # **

So maybe the reviewer has a valid point.

But wait, there's more. Change the seed, but keep exactly the same parameters--as if someone wants to replicate your work. Same tests, but finding nothing anywhere.

set.seed(720)
ax = rnorm(12,100, 15)
ay = rnorm(12,101, 15)
bx = rnorm(12,102, 15)
by = rnorm(12, 98, 15)
t.test(c(ax,by), c(ay,bx))$p.val # "int"
[1] 0.1583043
t.test(c(ax,bx), c(ay,by))$p.val # x vs y
[1] 0.5041701
t.test(c(ax,ay), c(bx,by))$p.val # a vs b
[1] 0.668125
t.test(bx,by)$p.val              # x vs y for b
[1] 0.1737395

Maybe your reviewer is inviting you to a "P-hacking" party.

Addendum: With 12 observations from each distribution, the power of a Welch t test to distinguish between $mathsf{Norm}(102,15)$ and $mathsf{Norm}(98,15)$ is less than 10%.

set.seed(1234)
pv = replicate(10^5, t.test(rnorm(12,102,15),rnorm(12,98,15))$p.val)
mean(pv <= .05)
[1] 0.09417