Failed CLs limits when adding systematics using HistFactory

Dear experts,

I have been using HistFactory to construct my model and then using the model to set limits using the asymptotic calculator.

However, I find when I start to add my systematics to my background using the HistFactory AddHistoSys() function, the limits fail when I include the full list of systematics. Giving me the following result:

AsymptoticCalculator::EvaluateNLL -  value = nan	fit time : Real time 0:00:00, CP time 0.760
[#0] ERROR:Minization -- AsymptoticCalculator:  failure in fitting for qmu or qmuA 1 return a dummy result 
CLb:                   nan
CLs:                   nan
CLs+b:                 nan
CLs+b/CLb:             nan
Signifignace:          nan

It seems to work fine when I don’t include any systematics, and when I include 1 - 3 when constructing my models.

I’ve included an example where this issue is reproduced along with the root files used by it. Output.txt contains the full output of the script incase that will be helpful. I am unsure what I am doing wrong. Any help will be appreciated.

Thanks,
Deshan

testLimits_withsys.py (4.5 KB)
inputSig_pureCI_LL.root (144.3 KB)
sysHist_ee.root (47.8 KB)
output.txt (126.1 KB)

Do any of your systematic uncertainties predict negative yields? You cannot construct a PDF that predicts negative probabilities. This is something that you could check for all of the systematic uncertainties you added.

Hi Stephan,

Thanks for looking into this.

None of the systematics predict negative yields. I have also checked adding them individually and running one at a time and there doesn’t seem to be a problem with that. But, including more that 3 or 4 results in a failed CLs calculation.

Thanks,
Deshan

Hi Deshan,

the problem is not in the number of systematic uncertainties, the fact that it fails with NaN is just a symptom. Even with two systematic uncertainties, I get this:

 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 1.00985 added to diagonal of error matrix
 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 2 in Error Matrix
 MINUIT WARNING IN MIGRAD  
 ============== 1.11769 added to diagonal of error matrix

The problem is in the model or how systematic uncertainties are defined. I could imagine that the histograms to parametrise the systematic uncertainties are not well defined.

  • Could you plot the nominal and ± 1 sigma histograms on top of each other?
  • Could it be that they are not (at least more or less) symmetric? For example, both sysUp and sysDown point in the same direction?
  • Could it be that their normalisation is completely different from the nominal histogram?

Hi Stephan,

Ahh missed these warnings.
Thanks for your suggestions, I checked the systematic histograms and indeed there was a problem with them. The up and down variation histograms were not properly defined (They were pointing in the same direction)

I have now fixed this issue and tried to run over them again. The systematic should now all be symmetric and the normalisation is correct. I have included plots showing the ratio of sys / nom where nom is the bkg template. the new sys histograms have been included in a root file.

However, I am still seeing the following warning and a nan result:

 MINUIT WARNING IN MIGRAD  
 ============== Negative diagonal element 3 in Error Matrix

I looked at adding the systematic histograms individually to see if there were some problematic ones I could narrow down. It looks like Variation1, Variation2, Variation4, Variation5 and Variation11 (defined in the python script) produces this warning by it self.

But, once I start to include a couple of the others to the model (that did not complain initially) it starts to complain again giving the same warning and then eventually resulting in a nan result once more systematics have been added.

Thanks,
Deshan

new_sysHist.root (47.8 KB)
sysPlot.pdf (157.6 KB)

Now, I’m out of ideas. Try to check if there are nasty things in the histograms:

  • A bin in all histograms is zero or negative
  • Normalisation issues (did you check if the systematics change normalisations? Do these look reasonable?)
  • Is the bin width fixed or does it vary?

Hi Stephan,

Thanks for your suggestions. Indeed the problem is quite confusing.

  • I checked the binning and it looks like there’s no zero or negative entries.
  • I normalised the histograms to 140/fb and that is consistent with all the histograms.
  • The bin width is fixed with 57 bins going from 300 GeV to 6 TeV (100 GeV bins)

I was made a suggestion where I was asked to check reducing the NormRange of mu from 0 - 100 to 0 - 10. Interestingly, this produced a CLs result in the end. However, when I checked another signal mass (e.g sigName = "diff_CI_LL_const_40_TeV" -> sigName = "diff_CI_LL_const_12_TeV") it fails. Also the warnings regarding the Negative matrix elements still persists, so probably the underlying problem wasn’t fixed…

Thanks,
Deshan

It will be some work, but it might be very insightful:
I suggest to

  1. Create a histogram using
    createHistogram or
  2. Plot the PDFs using plotOn

For each channel, for nominal and both systematic up/down (by setting the corresponding nuisance parameter to 0 or ±1).
In this way, you will see what RooFit sees.

Thanks! I’ll try this out and see how everything looks.

Hi Stephan,

I’ve been looking through the models etc. I looked at the PDF setting the individual nuisance params to -1 / +1 and it doesn’t look like anything crazy going on.

One this that stands out that I can’t seem to understand is that when I plot the data set that is created (obsData)

x = w.obj("obs_x_channel1")
dataHist = w.obj("obsData")
c = ROOT.TCanvas()
pl = x.frame()
dataHist.plotOn(pl)
pl.Draw()
c.SetLogx()
c.SetLogy()
c.Draw()

I get the following error:

Error in <THistPainter::PaintInit>: log scale requested with a negative argument (-4481.439143)
Error in <THistPainter::PaintInit>: log scale requested with a negative argument (-4481.439143)
Error in <THistPainter::PaintInit>: log scale requested with a negative argument (-4481.439143)
Error in <THistPainter::PaintInit>: log scale requested with a negative argument (-4481.439143)

However, looking at the plot without setting the log y axis it doesn’t look like there are any negative elements
dataSet.pdf (17.6 KB)

I also use the createHistogram function to see if there any negative bins in the dataset

roo_data_hist = ROOT.RooDataHist("h", "h", ROOT.RooArgSet(w.var("obs_x_channel1")), w.data("obsData"))
h_chan_data = roo_data_hist.createHistogram("chan_data", w.var("obs_x_channel1"))
c2 = ROOT.TCanvas()
h_chan_data.Draw()
c2.SetLogx()
c2.SetLogy()
c2.Draw()

it doesn’t complain when I plot it has a histogram and set the log y,
data_TH1.pdf (15.8 KB)

Do you have any ideas why this might be?

Thanks,
Deshan

Hi Deshan,

that’s very interesting. I have two more ideas:

gStyle->SetOptStat(11111111111)
  • Does your dataset contain events with negative weights? Or maybe events in the underflow bin?

“a lot of” = 9

Hi,

It looks like the underflow and overflow bins are 0.
I checked the weights using:

dataHist.weight(), " ", dataHist.weightError()

and it doesn’t look like anything is negative there.

But looking at the error bars on the DataSet -> TH1 and the original TH1 that was given. They seem to be very different.
for example:
The dataHist has:

Bin Value GetBinErrorLow()
0 0.0   0.0 
1 44814.3867188   44814.3860846 
2 14990.0078125   14990.0082917 
3 6038.33398438   6038.33394093 

and the original TH1 that was given as the data set has:

Bin Value GetBinErrorLow()
0 0.0   0.0
1 44814.3860846   78.4271916153
2 14990.0082917   49.359170279
3 6038.33394093   32.7222163641

The lower error bars seem to be going negative and this might cause problems in plotOn?
But, could this be causing the overall issue as well?

Thanks,
Deshan

The bin content for the DataHist and Original TH1
DataHist2TH1.txt (2.1 KB)
OrigTH1.txt (2.0 KB)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.