About FeldmanCousins

Roofit and RooStats
1

Hi all,

does an easy way exist to retrieve the value of R ordering paramter ( P(n/mu) / P(n/mu_best) ) in roostats::FeldmanCousins class (as defined in F&C paper) - for each value of input parameter?
Moreover does a method exist to plot it as function of the parameter mu?

Thank you.
Michele.

2

Hi,

I think again it is easier to use the HypoTestInverter (see post /viewtopic.php?f=15&t=14839 )
In this case, when using the StandardHypoTestInvDemo macro you also get the test statistic distributions ® for the different values of the parameter of interest that is tested

Lorenzo

3

Hi Lorenzo,

thank you for your replies, but I need a further important clarification.
Suppose I would like to show the value of a test statistics, scanning just one parameter of my likelihood (with nuisance), and plot the value of the test statistics.

I see two possible approaches:
I could use FeldmanCousins class, making a PointSetInterval and using EvaluateTestStatistics() method for each scanned point. Given a CL, FeldmanCousing class retrieves also the acceptance interval, i.e. my FC confidence interval.
On the other hand I could loop on the array elements of an HypoTestInverterResult (StandardHypoTestInverter-like macro) , and use GetTestStatisticData() method. Given a CL my acceptance interval in this case should be given by Upper- and LowerLimit methods.

My question is: are both these two approaches correct? Are they really equivalent?

Thank you!
Michele.

4

Hi,
Yes, the two approaches are equivalent, you just need to run the HypoTestInverter using the FrequentistCalculator and a two-sided profile likelihood test statistics.
The Feldman-Cousins class will tell you if each scanned point belongs to the interval or not.
The HypoTestInverterResult will contain for each scanned point, the p-values (CLs+b) and also the test statistic distribution obtained from the toys and the test statistic value from the data.
The HypoTestInverterResult works in one dimension, but you can probably extend the scan yourself without too much work to work in multi-dimens running many different runs.

Best Regards

Lorenzo

5

Hi,

sorry for the late response and thank you for you reply. So, if I well understand HypoTestInverter (given a PL double sided ts) should return a Feldman-Cousins confidence interval on the estimated parameter, right?
May you point me out any reference in literature that could explain in detail the theoretical basis for this interval construction method?

Thank you!

Michele.

6

Hi,

You can find the description of the Feldman-Cousins construction in the Feldman-Cousins original paper.
For the HypoTestInverter is equivalent, you can find some information on the slides I have presented at the Desy stat school in April (they are linked also from the RooStats twiki page)

indico.desy.de/getFile.py/acces … onfId=5065

Lorenzo

7

Hi Lorenzo,

thank you.
I have another question indeed.
Looking for further documentation about the inclusion of systematics uncertainties in Feldman Cousins method, I found also arxiv.org/pdf/physics/0302057.pdf.
It looks like a (known) counter intuitive behaviour of uncertainty inclusion method (by Conrad et al.) is the reduction of confidence interval whenever signal efficiency uncertainties increase and nobs <~ nexp.
This effect is due to the choice of integration strategy of test statistics.
Does RooStat take into account this effect in any way or does it use different strategies?

Thank you.

Michele.

8

Hi Michele,

The paper you cite is quite old and I am not sure is fully correct, it looks a bit bizarre to me the construction of the test statistics.
The effect of better results when including systematic uncertainty is known and it is due to the discreteness of the Poisson distribution.
For FeldmanCousins it is recommended not to use the hybrid integration of systematic uncertainty, as described in the paper, but to treat the uncertainty as auxiliary measurements and introduce for each one of them global observables, which will be varied for every pseudo-experiment.
More information on the procedure for this frequents sampling, implemented in RooStats in the FrequentistCalculator and the ToyMCSampler classes, is described in this note (see page 35)
cdsweb.cern.ch/record/1375842/fi … 11-011.pdf

Best Regards

Lorenzo

9

Hi Lorenzo,

could you please expand on why For FeldmanCousins it is recommended not to use the hybrid integration of systematic uncertainty ? What are the drawbacks? What differences should be expected with the results obtained with the frequentist approach ? Any other reference on this point ?

Thanks a lot in advance
Giacomo