Hi Lorenzo,
Thanks a lot for helping me.
Yes, I have the global observables and nuisance parameters in ModelConfig. This is my ModelConfig with few nuisance parameters:
=== Using the following for ModelConfig ===
Observables: RooArgSet:: = (obs_x_channel0)
Parameters of Interest: RooArgSet:: = (norm)
Nuisance Parameters: RooArgSet:: = (Lumi,alpha_BJesUnc,alpha_btag,alpha_jeff,alpha_jer,alpha_jvf,alpha_mistag)
Global Observables: RooArgSet:: = (nominalLumi,nom_alpha_BJesUnc,nom_alpha_btag,nom_alpha_jeff,nom_alpha_jer,nom_alpha_jvf,nom_alpha_mistag)
PDF: RooProdPdf::model_channel0[ lumiConstraint * alpha_BJesUncConstraint * alpha_btagConstraint * alpha_jeffConstraint * alpha_jerConstraint * alpha_jvfConstraint * alpha_mistagConstraint * alpha_A_NormConstraint * alpha_B_NormConstraint * channel0_model(obs_x_channel0) ] = 74.0024
If I include few systematic uncertainties sources in the ModelConfig, then expected and observed limits using Frequentist and Asymptotic calculator are similar.
But if I include too many systematic uncertainties sources, then observed limit using Frequentist calculator is much higer than the one using Asymptotic calculator, but expected limits are similar using both clalculator.
Here are the files which I use in case of few systematics:
- file for creating example.root file:
[code]void makeChannel0()
{
TFile* example = new TFile(“data/example.root”,“RECREATE”);
TH1F* data = new TH1F(“data”,“data”, 1,0,1);
TH1F* signal = new TH1F(“signal”,“signal histogram (pb)”, 1,0,1);
TH1F* background_A = new TH1F(“background_A”,“background histogram (pb)”, 1,0,1);
TH1F* background_B = new TH1F(“background_B”,“background histogram (pb)”, 1,0,1);
TH1F* background_C = new TH1F(“background_C”,“background histogram (pb)”, 1,0,1);
TH1F* background_D = new TH1F(“background_D”,“background histogram (pb)”, 1,0,1);
TH1F* background_E = new TH1F(“background_E”,“background histogram (pb)”, 1,0,1);
TH1F* background_F = new TH1F(“background_F”,“background histogram (pb)”, 1,0,1);
TH1F* background_G = new TH1F(“background_G”,“background histogram (pb)”, 1,0,1);
TH1F* background_H = new TH1F(“background_H”,“background histogram (pb)”, 1,0,1);
TH1F* background_I = new TH1F(“background_I”,“background histogram (pb)”, 1,0,1);
data->Fill(0.5,69.0000);
background_A->Fill(0.5,16.8921);
background_B->Fill(0.5,40.4409);
background_C->Fill(0.5,6.5852);
background_D->Fill(0.5,1.9575);
background_E->Fill(0.5,3.8127);
background_F->Fill(0.5,4.8727);
background_G->Fill(0.5,0.1594);
background_H->Fill(0.5,0.0422);
background_I->Fill(0.5,1.0451);
signal->Fill(0.5,166.91);
example->Write();
example->Close();
}
[/code]
- config/example.xml - file:
[code]
./config/example_channel0.xml
norm
Lumi
alpha_A_Norm
alpha_B_Norm
[/code]
3. config/example_channel0.xml file:
[code]
<Sample Name="background_A" HistoPath="" NormalizeByTheory="True" HistoName="background_A">
<OverallSys Name="BJesUnc" Low="1.0061" High="1.0642"/>
<OverallSys Name="btag" Low="0.9694" High="1.0657"/>
<OverallSys Name="jeff" Low="0.9753" High="1.0247"/>
<OverallSys Name="jer" Low="0.9413" High="1.0587"/>
<OverallSys Name="jvf" Low="1.0226" High="1.0228"/>
<OverallSys Name="mistag" Low="0.9823" High="1.0924"/>
<OverallSys Name="A_Norm" Low="0.8590" High="1.1410"/>
</Sample>
<Sample Name="background_B" HistoPath="" NormalizeByTheory="True" HistoName="background_B">
<OverallSys Name="BJesUnc" Low="1.0450" High="1.0255"/>
<OverallSys Name="btag" Low="1.0690" High="1.0516"/>
<OverallSys Name="jeff" Low="0.9314" High="1.0686"/>
<OverallSys Name="jer" Low="0.8250" High="1.1750"/>
<OverallSys Name="jvf" Low="1.0393" High="0.9988"/>
<OverallSys Name="mistag" Low="1.0726" High="1.0897"/>
<OverallSys Name="B_Norm" Low="0.8650" High="1.1350"/>
</Sample>
<Sample Name="background_C" HistoPath="" NormalizeByTheory="True" HistoName="background_C">
<OverallSys Name="BJesUnc" Low="1.0731" High="1.0983"/>
<OverallSys Name="btag" Low="1.1072" High="0.8890"/>
<OverallSys Name="jeff" Low="0.9568" High="1.0432"/>
<OverallSys Name="jer" Low="0.8769" High="1.1231"/>
<OverallSys Name="jvf" Low="1.0463" High="1.0277"/>
<OverallSys Name="mistag" Low="1.0002" High="1.0524"/>
</Sample>
<Sample Name="background_D" HistoPath="" NormalizeByTheory="True" HistoName="background_D">
<OverallSys Name="BJesUnc" Low="1.0031" High="1.0001"/>
<OverallSys Name="btag" Low="1.1350" High="0.8650"/>
<OverallSys Name="jeff" Low="1.0000" High="1.0000"/>
<OverallSys Name="jer" Low="0.3510" High="1.6490"/>
<OverallSys Name="jvf" Low="1.0000" High="1.0000"/>
<OverallSys Name="mistag" Low="0.9739" High="1.0258"/>
</Sample>
<Sample Name="background_E" HistoPath="" NormalizeByTheory="True" HistoName="background_E">
<OverallSys Name="BJesUnc" Low="0.9755" High="0.9928"/>
<OverallSys Name="btag" Low="0.9671" High="0.9279"/>
<OverallSys Name="jeff" Low="0.9482" High="1.0518"/>
<OverallSys Name="jer" Low="0.8532" High="1.1468"/>
<OverallSys Name="jvf" Low="0.9488" High="0.9982"/>
<OverallSys Name="mistag" Low="0.9213" High="1.0132"/>
</Sample>
<Sample Name="background_F" HistoPath="" NormalizeByTheory="True" HistoName="background_F">
<OverallSys Name="BJesUnc" Low="0.9792" High="0.9869"/>
<OverallSys Name="btag" Low="1.0322" High="0.9288"/>
<OverallSys Name="jeff" Low="0.9799" High="1.0201"/>
<OverallSys Name="jer" Low="0.9873" High="1.0127"/>
<OverallSys Name="jvf" Low="0.9938" High="0.9794"/>
<OverallSys Name="mistag" Low="0.9895" High="0.9707"/>
</Sample>
<Sample Name="background_G" HistoPath="" NormalizeByTheory="True" HistoName="background_G">
<OverallSys Name="BJesUnc" Low="1.0019" High="1.0006"/>
<OverallSys Name="btag" Low="1.0620" High="0.9243"/>
<OverallSys Name="jeff" Low="1.0000" High="1.0000"/>
<OverallSys Name="jer" Low="0.6301" High="1.3699"/>
<OverallSys Name="jvf" Low="1.0000" High="1.0000"/>
<OverallSys Name="mistag" Low="1.0069" High="0.9931"/>
</Sample>
<Sample Name="background_H" HistoPath="" NormalizeByTheory="True" HistoName="background_H">
<OverallSys Name="BJesUnc" Low="1.0025" High="1.0161"/>
<OverallSys Name="btag" Low="1.1241" High="0.8483"/>
<OverallSys Name="jeff" Low="0.9566" High="1.0434"/>
<OverallSys Name="jer" Low="0.9130" High="1.0870"/>
<OverallSys Name="jvf" Low="1.0678" High="1.0073"/>
<OverallSys Name="mistag" Low="0.9905" High="0.9859"/>
</Sample>
<Sample Name="background_I" HistoPath="" NormalizeByTheory="True" HistoName="background_I">
<OverallSys Name="BJesUnc" Low="1.0103" High="1.0538"/>
<OverallSys Name="btag" Low="0.9790" High="1.0465"/>
<OverallSys Name="jeff" Low="0.9488" High="1.0512"/>
<OverallSys Name="jer" Low="0.9746" High="1.0254"/>
<OverallSys Name="jvf" Low="1.0399" High="1.0399"/>
<OverallSys Name="mistag" Low="1.0298" High="1.0188"/>
</Sample>
[/code]
If I use the files shown above, then I get the follownig results:
Asymptotic calculator:
The computed upper limit is: 0.154325 +/- 0
Expected upper limits, using the B (alternate) model :
expected limit (median) 0.184132
expected limit (-1 sig) 0.135239
expected limit (+1 sig) 0.250728
expected limit (-2 sig) 0.0978015
expected limit (+2 sig) 0.300834
Frequentist calculator (10000 toy):
The computed upper limit is: 0.177149 +/- 0.00358786
Expected upper limits, using the B (alternate) model :
expected limit (median) 0.183857
expected limit (-1 sig) 0.136362
expected limit (+1 sig) 0.250136
expected limit (-2 sig) 0.112155
expected limit (+2 sig) 0.295055
In this case the results are similar and everythig is fine, but if I use the xml files which are in attachment (all systematic uncertainties sources are included) then I get the following results:
Asymptotic calculator:
The computed upper limit is: 0.18238 +/- 0
Expected upper limits, using the B (alternate) model :
expected limit (median) 0.198307
expected limit (-1 sig) 0.145693
expected limit (+1 sig) 0.282874
expected limit (-2 sig) 0.112837
expected limit (+2 sig) 0.34469
Frequentist calculator (10000 toy):
The computed upper limit is: 0.314777 +/- 0.00241345
Expected upper limits, using the B (alternate) model :
expected limit (median) 0.207605
expected limit (-1 sig) 0.161386
expected limit (+1 sig) 0.299007
expected limit (-2 sig) 0.0788636
expected limit (+2 sig) 0.39817
Here we can see that observed limits are significantly different.
Sorry that my post is too big
but it’s very important for me to understand what’s happening and if the results are correct.
Best regards,
Archil
example.tar.gz (4.29 KB)