I have three questions about the implementation of HybridCalculatorOriginal.
My first question is
Is the idea same as a method known as Cousins&Highland ?
Phys. Rev. D 67, 012002 (2003)
In the class, nuisance parameters are fluctuated during the generation of toy MC.
And likelihood ratio(or delta-NLL) is calculated for each toy experiment.
My second question is,
Are following two equivalent to each other ?
(A) As it is done in HybridCalculatorOriginal, delta-NLL is used as a test statistic, nuisance parameters are fluctuated once per experiment.
(B) As it can be read from Conrad paper, likelihood is integrated for numerator(H0) and denominator(H1) of likelihood ratio separately, then the ratio of integrated likelihood is used as the test statistics.
And my third question is,
Do we need to do the integration of likelihood also for data ?
As far as I read the code, for data, NLL is calculated only once. I don’t find integration over nuisance parameters.
First of all the HybridCalculator performs an hypothesis test, while the method in the paper you refer to, is a method for confidence level estimation.
In Roostats there is the class HypoTestInverter, which scans the results from the HybridCalculatorOriginal to find the limit.
In this case a simple ordering based on the observed test statistics distribution is used for finding just the upper limit.
In the Conrad paper, I think the profiled likelihood ratio is used and the ordering is based on the shortest interval, and it should be the same described in the Feldman-Cousins paper By default, the HybridCalculatorOriginal uses a simple likelihood ratio as test statistics.
In the HybridCalcuator the likelihood is integrated in the nuisance parameters using
toy Monte Carlo. The nuisance parameters are fluctuated once per experiment and toys are generated using those parameters.
I am not sure what is used exactly in the referred paper. So I cannot comment on your (B) point.
Concerning your third question, there is no need to integrate the likelihood for the data. But, if your test statistics is the profile likelihood ratio, you need to fit for the best value of the nuisance parameters at the given parameter of interest point you want to test.
Yes, HybridCalculatorOriginal is basically for CLs; while in Conrad paper, interval of the parameter of interest is (as far as I understand) calculated in Feldman-Cousins method with including systematics by integration.
I would like to ask a nature of HybridCalculatorOriginal.
Suppose we want to calculate significance of Gaussian signal on a flat background.
And we have uncertainty in the mean of the signal Gaussian.
When calculating CLb, nuisance parameters (i.e. the mean) are varied in the generation of toy MC, but it does not affect things, because 0 signals are generated.
As a result, significance of a given data does not change whether or not you include systematics.
It is the same when we have uncertainty in the sigma of signal Gaussian.
Is my understanding correct ?
(In case of other methods, Bayesian integration or profiling, significance can change by signal PDF uncertainty.)