Does the recent "3-sigma" result at LHCb account for the number of different tests of beyond standard model physics that have been done?

Question

Recently there has been quite a lot of media interest generated by a reported observation of beyond-standard-model physics at the LHC with a "three sigma" degree of statistical significance.
My understanding (correct me if this is wrong), is that this roughly means that, in a world where there is no physics beyond the standard model, there would be ~1/1000 chance that this experiment would see the degree of difference to the standard model prediction that it sees.
This sounds compelling, but on the other hand I assume that a huge number of different tests have been run on LHC data looking for beyond standard model physics. I would be surprised if that number isn't easily greater than 1000, over the course of the LHC's life (again, please correct me if this wrong). So, in that light, it doesn't seem surprising that one may occasionally see such deviations from the standard model in the data, even without any actual new physics.
I understand it is possible to correct for multiple tests when calculating the statistical significance of a result, but it is not clear to me from the accounts in the media whether this has been done. So my question is: does the claimed significance of the LHCb result account for the multiple tests for beyond standard model physics that have been done on LHC data?

benrg · Accepted Answer

Look-elsewhere effect is the name for this, or one name for it. You'll sometimes see two different significance measures quoted for an effect, one with a look-elsewhere correction and one without.
The LHCb report is arXiv:2103.11769. At a glance I don't see evidence that they've considered the look-elsewhere effect. That's fine: results like this are useful when correctly interpreted. The correct interpretation is not that the effect exists, but that it's worth devoting extra resources to investigating it. Three-sigma effects usually disappear with additional data, but you never know.
Tommaso Dorigo, who is part of the CMS collaboration, says in a blog post "the odds that this is instead only a fluke are really, really high".
On the other hand, BaBar and Belle have seen evidence for a similar anomaly, at least according to Wikipedia (last paragraph of the section). That may be cause for optimism.

Andrew Steane · Answer

Let's take "3 sigma" to mean three standard deviations, and look at a couple of probabilities.
If a datum is taken randomly from a sample distributed according to a Gaussian distribution, then the probability the datum will be found to lie within 3 standard deviations of the mean is
$$
frac{1}{sqrt{2 pi}} int_{-3}^3 e^{-x^2/2} dx simeq 0.9973
$$
so the probability a datum falls outside this range is about $0.0027$ or 1 in 370.
But in practice it is rare to get a pure Gaussian statistic. Usually, owing to a variety of causes, the wings of the distribution are higher than those of a Gaussian function. Let's model this in a rough-and-ready way by supposing that the underlying distribution is really a combination of some Gaussian of given width, with a small (say 2 percent) addition of a wider 'pedestal' which we will also take to be Gaussian but with 5 times the width. Then the distribution function is
$$
f(x) = 0.98 N(x,1) + 0.02 N(x,5)
$$
where $N(x,sigma) = exp(-x^2/2 sigma^2) / sigma sqrt{2 pi}$. The integral of this distribution between $pm 3$ is $0.986$ so now the probability of falling at $3 sigma$ or more is about $0.014$ or one in 73. This is just a rough picture to get a feel for the numbers.
Note I have not bothered to use the standard deviation of this second distribution, which is $1.16$, since the two distributions look alike when plotted on a linear scale, and with the kind of limited data sample available, I doubt whether it is possible to tell, from the data alone, whether they are distributed as the second or the first distribution, or some other distribution.
In the first calculation, if the team did roughly the same experiment 370 times then they would be quite likely to see the result as a random fluctuation, and in the second case they would only need to do the same experiment about 73 times. I don't know what situation the team is in, but they will be aware of all this so their next move will be to look again, both at what they already did, and, more importantly, to get more data if that is feasible.

Does the recent "3-sigma" result at LHCb account for the number of different tests of beyond standard model physics that have been done?

2 Answers

Add your own answers!

Ask a Question