Comparison among different calculators results


I produced a model in RooStats, and I would like to set a limit on a single observable.
Some indipendent uncertainty sources have been included, and a set of nuisance parameters has been defined.
I would like to compare the results given by different calculators, study the impact of uncorrelated uncertainty sources, possibly turning on/off some of them.
I tested my model using a ProfileLikelihoodCalcuator, and I got some numbers.
For other calculators I just used StandardHypoTestInverterDemo.C out of the box. I tested

  • Hybrid Calculator (CLs, 1-Sided PL)
  • Asymptotic Calculator (CLs, 2-Sided PL)
  • Frequentist Calculator (CLs+b, 2-Sided PL -> FeldmanCousins)

AsymptoticCalculator gave an upper limit very close to those produced by ProfileLikelihood Calculator; Hybrid Calculator limit looks a bit tighter.
However FC strategy fails when the largest systematic uncertainties sources are on (plots attached refer to the same Workspace).

Why (just) a FC approach should fail in a such drmatical way? Have you got an opinion about that?


Asym_CLs_grid_ts3_BNVWorkspaceMuon-3.pdf (14.2 KB)
Freq_Cls+b_grid_ts2_BNVWorkspaceMuon-3.pdf (14.9 KB)
Hybr_CLs_grid_ts3_BNVWorkspaceMuon-3.pdf (14.9 KB)


It is strange that the FrequentistCalculator does not work for you. I suspect a problem in the model definition, for example setting correctly global observables. I would need to see your full model to understand the problem.
Can you please post your workspace including the data set and ModelConfig (or the macro creating it) that you pass to the StandardHypoTestInvDemo macro ?

Best Regards