Hello,
I have a question about interpretation of result of ProfileLikelihoodCalculator…
If I understand correctly, the Wilk’s theorem tells that if background hypothesis
is correct, then -2log(ratio of likelihoods for background only and signal+background)
should be distributed as chi^2. So far so good.
If I compare b and s+b hypothesis
and resulting likelihoods ratio is well consistent with that distribution, then
I should be essentially happy with my background hypothesis.
Now more interesting case: suppose likelihood ratio is big enough, say, corresponds to
probability of 5%. Then I could make statement, that my background hypothesis is not
correct on 95% C.L. However, I can tell nothing about my s+b hypothesis, because by
assuming any hypothesis other than the background one, I would break the base assumption
of Wilk’s theorem. Should I do this, I could not clime chi^2 distribution for likelihood ratio anymore,
and thus could not do any quantitative conclusions about my signal hypothesis.
The consistency of signal hypothesis must be tested by other means.
So the statement that “ProfileLikelihoodCalculator excludes background hypothesis
on the corresponding C.L.” is the only one I can make, no statement about signal hypothesis.
This brings me to interesting conclusion: as soon as this statement characterizes particular
data sample and the background hypothesis, and is independent from the used signal hypothesis,
I should expect to get consistent results using any arbitrary signal hypothesis which adds one extra
degree of freedom. Although this statement looks suspicions to me, it is a logical conclusion
from the reasoning above, isn’t it?
What is the right interpretation of ProfileLikelihoodCalculator results then?
Cheers,
-Fedor
Hi,
I posted your question to the roostats developers. Here is the
answer from Eilam Gross (Atlas stat comittee)
Wouter
Somehow I do not manage to login to answer this question
So here us my answer to
root.cern.ch/phpBB2/viewtopic.php?t=9168
To my understanding the PLR and the Wilk theorem is abouit the hypothesis you want to test
�By testing the H_0 hypothesis you can only say things about the CL of the background hypothesis
�If your p-value is less than 5% it means that you reject the bg hypothesis in the 1.64\sigma level
�The issue of 1-sided 2-sided is a delicate issue here
�Likewise if your p-value when testing H_0 is ~2.4*10^-7 you reject the BG hypothesis at the 5\sigma level and you will announce a dicovery
�IF you want to make statements about the s+B hypothesis, test the H_1 hypothesis and try to reject it.
�But the PLR is about testing the H_0 or H_1 hypotehsis
�The wilk theorem will say that when you test hypoithesis H_\mu
�the pdf of experiments under the assumption of H_\mu is a Chi**2.
�PLease read the ATLAS CSC note about the combined Higgs performance, it contains 10 pages of explanation about the PL and the Wilk theorem
and it is published.
�Best
�Eilam
Dear Fedor,
It’s probably worth a refresher on classical hypothesis testing. Your question seems to be about power (1-beta) in classical hypothesis testing. A test with a given significance level (alpha) with respect to the null will have different powers against different alternatives. Different tests, all with the same significance level with respect to the null, will typically have different powers with respect to any specified alternative. To brush up on this, you can start with slides 40-42 of my talk at HCPSS 2009, indico.cern.ch/conferenceOtherVi … nfId=44587 , and read the pages of Fred James’s 2nd edition of Eadie et al from which I took the figures (including discussion of the Neyman-Pearson Lemma).
Best regards,
Bob