RooStats::HypoTestInvert results Nonsensical

Roofit and RooStats
1

Hello RooStats experts. I’ve implemented a class to calculate asymptotic limits using the RooStats::HypoTestInverter class, almost precisely configured as suggested here .

This seems to work fairly well, however I occasionally get strange behavior where the ±sigma values don’t make sense in the context when returning htresult->GetExpectedUpperLimit(+1) (and other indices.) Below is such an example showing the different expected value results of a limit calculation:

<exp_upper_med>                                         <Tag:double>                  0.545454545454545                                                                                                
<exp_upper_p1s>                                         <Tag:double>                  1.71817994542153                                                                                                 
<exp_upper_m1s>                                         <Tag:double>                  0.383838383838384                                                                                                
<exp_upper_p2s>                                         <Tag:double>                  1.71813563735354                                                                                                 
<exp_upper_m2s>                                         <Tag:double>                  1.17272654090067

As you can see, the +1 and +2 sigma values are virtually identical, but the +2 value is lower than the +1. Also, the -2 sigma value is higher than the median value. How can this make sense? I don’t see anything in the output that points to why this is happening. Is this explainable?

Thanks for the consideration,
Nate

2

May be @moneta can help.

3

Hi,
I agree the results you obtain do not make much sense. In that case you should look at the scan plot obtained to compute the limit to understand exactly what happened. It could be that some of the fit failed at some scanned points and their results do not make sense.
If you cannot fix this issue for example by using more scanned points or more stable fits, could you please upload your RooFit workspace and your macro computing the limit, so we can investigate this in detail.

Cheers

Lorenzo

4

Hi Lorenzo,

I already use quite a coarse grid, 200 scan points across a range of 0 to 2.0 for the results above, is this not actually that coarse? I’ve used the same input workspaces in a different “CalculateLimits” script (which is a giant 2k line black box that I wanted to avoid using for this reason), that I’ve validated for some other mass points of my signal production that don’t exhibit this behavior.

As for our macro, we have it included into a class in our Statistical Framework but it follows verbatim the originally linked RooStats directions.

Thanks,
Nate

5

Can you somehow save the fit status or maybe the complete fit results for these points? It’s maybe worth looking if there are failures.
Also, checking for warnings might be a way to go forward.

6

Hi,
Can you then upload maybe the scan plot giving this non-sense results ? Do the obtained curves make sense ?

Lorenzo

7

Hi again,

Yes I will add this, which function output exactly in this case? None of the result class functions stick out as the scan plot. Do I need to manually save this somehow?

8

Hi,

From the obtained result, the HypoTestInverterResult object , which is returned from HypoTestInverter::GetInterval() , you need to use the HypoTestInverterPlot class

see the code in the tutorial,

Lorenzo

9

Hi, Sorry again for not understanding exactly what this plot object is. I see it is just an inherited TObject. If I want to save this somehow, the best way would be to save the canvas with c1->SaveAs()?

10

Hi,

If you want just to save the plots, yes it is maybe easier to save the canvas. Otherwise if you want to change slightly the plots later on, the best is to save the HypoTestInverterResult object, that you can read afterwards and re-do the plotting.

Lorenzo

11

OK, so for completeness I add the new output values

    <exp_upper_med>                                         <Tag:double>                  1.01109235990623                                            
    <exp_upper_p1s>                                         <Tag:double>                  1.01085833721167                                            
    <exp_upper_m1s>                                         <Tag:double>                  0.363636363636364                                           
    <exp_upper_p2s>                                         <Tag:double>                  1.01010049710288                                            
    <exp_upper_m2s>                                         <Tag:double>                  0.262626262626263

And here is the scan plot. In this example, you can see the p1 and p2 points are all lower than the medium (even if just slightly) and the m1 and m2 points are much much lower (and match the values that I get from the alternative fitting script).
TestScan.pdf (20.2 KB)

12

Hi,

It is clear from the scanned plot that for values of your parameter of interest larger than 0.5 there is a problem in getting the correct CL(s+b) and CL(b). This is due to the fits which are performed for that parameter values. I would try to understand why these fits fail. Maybe it is a problem on how your model is defined or you might need to adjust some parameters , like starting points of the fit, to make the fit working.
If I have access to the workspace, I could try helping you with the fits

Best regards

Lorenzo

13

Hi Lorenzo,

Yes I see this now. So in this case should I make the upper end of the scan range lower? Here is the workspace attached.workspace-preFit-GGF-2200.root (1.6 MB)

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.