How to get nuisance parameters values in hypothesis testing

Roofit and RooStats
1

Dear all,

i’m using a code based on StandardHypoTestDemo to do a CLs limit calculation on a s+b hypothesis.

I’m using the MC toys and i would like to plot the distribution of a given nuisance parameter for each
likelihood fit for each toy (for both b and s+b cases).

I’ve access to HypoTestInverterResult but i’m not sure i can get this info out of it.

Any help would be much appreciated.

best,
Max

2

Hi,

In the latest version of StandardHypoTestInbDemo , it has been added the option to use the detailed output of the test statistics. This output contains for example the initial and the fitted parameter results for each pseudo-experiment.
See “enableDetailedOutput” in root.cern.ch/root/html534/tutori … emo.C.html

I am not sure if the detailed output contains also the generated nuisance parameter value in case of hybrid sampling. In case it is not there, please let me know and I could add also that information.

To retrieve the Detailed output from the HypoTestInverterResult, just look for example at this attached macro

Best Regards

Lorenzo
readHypoTestInvDetailedOutput.C (7.17 KB)

3

Hi Lorenzo,

thanks for your reply. I already write out the HypoTestInverterResult to file but this hasn’t the methods
like GetNullDetailedOutput and GetAltDetailedOutput. Btw are they for B(mu=0) and B+S cases?

The code i’m using is basically what follows

   FrequentistCalculator* hc = new FrequentistCalculator(*data, *bModel, *sbModel);
   ToyMCSampler *toymcs = (ToyMCSampler*)hc->GetTestStatSampler();
   toymcs->SetTestStatistic(testStat);
   hc->SetToys(ntoys,ntoys);
   hc->UseSameAltToys();
   RooMsgService::instance().getStream(1).removeTopic(RooFit::NumIntegration);
   HypoTestInverter calc(*hc);
   calc.SetConfidenceLevel(0.95);
   calc.UseCLs(true);
   calc.SetVerbose(true);
   ROOT::Math::MinimizerOptions::SetDefaultMinimizer("Minuit2");
   ROOT::Math::MinimizerOptions::SetDefaultStrategy(2);
   calc.RunOnePoint(mu);
   HypoTestInverterResult * r = calc.GetInterval();

Then i write the result to file. Is this enough to get the detailed output written out?
Sorry i’ve checked the demo but it seems indeed doing something similar.

A related question is how to debug particular values in the distribution for the test statistics.
In my case i see that the distribution for B only has a peak at zero that doesn’t go away even for large
mu values making the exclusion at +2 sigma quite problematic. I would like to debug these toys
with q(mu=0) ~ 0 to understand what’s going on.

best,
Max

4

Hi Max,

In order to get the detailed outputs from the fits to every toy that is generate you have to set a flag in the test statistic class. Before running the inverter (calc.GetOnePoint() or calc.GetInterval() , you need to do :

testStat.EnableDetailedOutput();

You need to call the method on the concrete test statistic class (e.g. RooStats::ProfileLikelihoodTestStat) and not on the base class, since it is not implemented there.

The detailed output will be contained then in the HypoTestResult, which is the result of the the hypothesis test at each mu values. The HypoTestInverterResult is just a collections of HypoTestResult.
The full results of each scan points, including the detailed output information, will be written in the file if you write the HypoTestInverterResult
So you can retrieve the detailed output from the HypoTestInverterResult, by using
HypoTestInverterResult::GetResult(index), for example for the first scanned point (index = 0), you can do :

HypoTestInverterResult * r = (HypoTestInverterResult * ) file.Get("resultName"); 
RooDataSet * nullData = r->GetResult(0)->GetNullDetailedOutput();   // output information from every toy generated using the null hypothesis (e.g. S+B boys) 
RooDataSet * altData = r->GetResult(0)->GetAltDetailedOutput();  // output information from alt toys (e.g  B only ) 
RooDataSet * fitInfo  = r->GetResult(0)->GetFitInfo();  // data fit results used to profile nuisances before generating the null and alt toys

Hope you can get all the information you need. If you have further problems please let me know

Best Regards

Lorenzo