How exactly does RooPlot::chisquare() compute the result?

Roofit and RooStats
1

Hello friends!

I got a question on how exactly does RooPlot::chisquare() compute the result? To be specific, I have a distribution with several 0-entry bins. And the chi2 calculation based on SumW2 would not work in such situation. However, the method RooPlot::chisquare() still somehow gave me an answer and I am just wondering why. Is it calculation based on Poisson errors?

I accessed Poisson based chi2 using:
RooChi2Var chi2 (“chi2”, “chi2”, *model,datah,DataError(RooAbsData::Poisson));
but the result was not compatible with RooPlot::chisquare() (a very huge difference, assuming RooChi2Var computed chi2, and I divided ndof from it). I believe RooChi2Var with Poisson errors should be able to treat 0 entries properly.

I looked into codes and it seems RooPlot::chisquare() is actually referencing RooHist::GetEYhigh() / low() to get the errors but unfortunately I did not manage to find the implementation on the calculation. So maybe the question can be reduced to how exactly does RooHist::GetEYhigh() / low() compute the result?

Any help?

Thanks a lot!
Sebastian

2

I think @StephanH can help you on this

3

Hi Sebastian,

we discussed Chi2 a bit in this thread here:

A very important difference is that the chi2 from the plot is calculated in terms of plot bins, and it is divided by ndf, but the number of fit parameters is not subtracted from ndf. The Chi2Var just computes Chi2 (no ndf), and it does that in bins of the observable(s), which don’t necessarily coincide with the plot bins.

For the bin errors, the histogram uses asymmetric Poisson errors. Bins that have a model prediction of zero are skipped.

4

Hi @StephanH,

Thanks a lot for your reply.

So here is what I get from those two methods:

RooChi2Var chi2 ("chi2", "chi2", *model,datah,DataError(RooAbsData::Poisson)) ： 24.94
frame->chiSquare(fun_name,"data",nfloparam)： 0.4834

My original data is RooDataSet “data”, I created RooDataHist “datah” in order to call RooChi2Var.

I have 120 bins in my RooDataHist (set in RooDataSet and confirmed by datah.numEntries()) and nfloparam = 2. I also specified the binning when doing data.plotOn with the same range used in datah, so I am not sure if “The Chi2Var … and it does that in bins of the observable(s), which don’t necessarily coincide with the plot bins.” is the case…

If so, chi2/ndof given by RooChi2Var would be 24.94/(120-2)=0.21, which deviates a lot from 0.48.

As for the 0-bins treatment, does RooChi2Var also skip bins with 0 model predicted value? What would be the threshold for model prediction to be considered as 0? (I looked and the minimum predicted is like 0.001 entries)

I tried to do the fitting as well as chi2 calculation again in a range where all bins have non-zero entries and model prediction is larger than 1. This time: nbins = 30, # of param = 2, and I got:

RooChi2Var chi2 ("chi2", "chi2", *model,datah,DataError(RooAbsData::Poisson)) ： 24.876
frame->chiSquare(fun_name,"data",nfloparam)： 0.8805

This time those results are close as 24.876/(30-2)=0.888 but they still differ.

I think there may be some much deeper things going on, and really look forward your suggestions!

Thanks a lot!

Best,
Sebastian

5

Hi Stephan,

I dug around a little bit on the results given by frame->chisquare() when I have 120 bins:

chi2/ndof = 0.4835, chi2/(adjusted ndof) = 0.5045 (sorry that 0.4834 in my last post is actually obtained without nfloparam fed in, a small typo), suppose they come from the same plain chi2, we have:

0.5045*(ndof-2) = 0.4835*(ndof) -> ndof = 48.0019 -> 48, which does not make any sense.

But if we plug 48 into, 48*0.4835 = 23.208, which is somehow close to 24.94 obtained from RooChi2Var but again they still differ.

Hope this may help! Thanks a lot!

Best,
Sebastian

6

Hi Sebastian,

it’s a bit hard to follow. So I see that you are running different ways of computing chi2, and you are trying different ways to compute ndf, or compute the reduced chi2. Here’s one thing that I noticed in your description, but I cannot check if that makes a difference:

You are using the RooDataHist datah to compute chi2, but when you ask the frame to compute a reduced chi2, you are using data. Given that datasets are not binned, the binning will only be decided when you plot. I e.g. got 100 bins when creating plot for a variable that doesn’t have a binning assigned. I cannot check if you actually have 120 bins. Try to use only 10 bins or so, so it’s easy to check by eye.

The ndof = 48 looks like you obtained a DataHist with 50 bins, and it should be called "data", and two of the bins are maybe emtpy. I find that surprising, since as I said the default should be 100, but 50 also sounds like a typical “default value” that could be defined somewhere in case the user didn’t specify the number of bins.
Try something like

 frame->Print("V")

to print all objects that are inside the frame. Maybe there’s a dataHist that you didn’t expect. You can then retrieve it from the frame, and get some details.
Edit:
One more idea: Maybe the histogram has indeed 100 bins, but you have only integer numbers in the dataset, so every second bin is empty. That would make it 50 effective bins, and maybe two of the integer bins are empty on top. I guess you have to check the things that actually get plotted.

If this doesn’t help, note the following:
As explained in the thread I linked in my first post, RooChi2Var computes:

chi2_i = (probability at bin centre * expectedEvents * bin volume - nData)^2 / (asymmetric Poisson error)^2

It then sums all bins using Kahan summation, skipping bins where data = pdf = 0.

RooPlot computes as follows:

chi2_i = (nData - <approximate integral over post-fit curve>)^2 / (asymmetric errors as returned by data hist)^2

Then it sums without Kahan summation, and divides by ndf-fitParam.

  • Unless probabilities in bins are drastically different, the [Kahan vs no Kahan] shouldn’t make much of a difference, I guess you can check that only if you don’t find anything else.
  • The [integral over bin vs bin centre] might make a difference if the fit function has a large 2nd derivative in certain bins. In this case, approximating the integral of the curve doesn’t yield the same as taking the value in the centre of the bin. I don’t have a plot of the function, so have a look if that’s maybe causing the difference. If the function is wiggly, the problem might go away be using less bins. Maybe that’s why in your 30-bin example the values get close.
7

Hello Stephan,

Thank you so much for the help!

I checked and I found that the ndof which RooPlot::chisquare() uses also only takes non-empty bins into consideration , and now I got consistent expected numbers of non-empty bins calculated by manipulating chi2 (from chisquare()) and counting directly from the plot. (For this case, 120 bins in total, 72 are empty and 48 are non-empty, which is what we got in my last post). So the calculation on this side should be OK. The raw chi2 should be 23.207

The number given by RooChi2Var is 24.94, but since you said that two methods determine the expected number of events in a certain bin in different ways, do you think it can explain the difference on raw chi2 values? Do you know how can we check if this is the case somehow?
(I just checked several other binnings, the difference is not that significant but still noticeable. Plotting range is 600-3000, if nbins = 240, RooChi2Var:48.95, chisquare():41.84; nbins = 60, RooChi2Var:17.38, chisquare():15.95; nbins = 30, RooChi2Var:9.34, chisquare():8.37;)

And from your point of view, which method would you recommend to use, or do you think calculating chi2 with rougher binning to reduce the number of 0-entry bins would be better?

Ah some additional thoughts, numbers given by RooChi2Var are always larger than those given by chisquare(), do you think RooChiVar in fact only skips bins with expected = 0 && observed = 0? (Could you take a look here? https://root.cern.ch/doc/master/RooChi2Var_8cxx_source.html#l00287), which in my example expected entries are always larger than 0 (the smallest would be ~0.01 but well, it is not 0). If so, the difference on raw chi2 maybe can be explained since RooChi2Var still sums every bins when chisquare() skips bins with 0 entry?

Thanks a lot for your help!

Best,
Sebastian

Thanks you so much!

Best,
Sebastian

8

It should make a difference only if the 2nd derivative is large. Do you see strong changes of the curvature?

For what exactly? If you just want to fit, the absolute value of chi2 is irrelevant, you only need relative changes. Zero bins shouldn’t be a problem.

Yes, both data and model need to be zero for Chi2Var to skip the bin.

9

Hi Stephan,

Thanks for your reply!

Sorry that I have not been clear on what I am doing. So I am trying to use chi2 as a methods of GoF test, so I think the absolute value is needed here.

And yes I found that a very large part of the discrepancy can be taken care of if we exclude the empty bins from ndof. After that, two methods gave me extremely close results. But I also noticed that this is only the case for unweighted samples. For weighted samples, do you have suggestions on how to choose between those two?

Thanks!

Best,
Sebastian

10

My first idea is that there shouldn’t be a big difference if the samples are weighted. As long as data and prediction are zero (irrespective of the weight), the bins are skipped. The weights obviously have an impact on the uncertainties of the bin contents, but this is taken into account when you switch the errors to SumW2. See here for the switch:
https://root.cern.ch/doc/master/classRooDataHist.html#a00a476834b0f2d246f85da34a915fcd9

11

Hi Stephan,

Ah yes but the problem is that RooChi2Var with DataError(RooAbsData::SumW2) would not work if weighted data still have 0-entries. The plotting is fine to choose SumW2 but I think RooChi2Var only works with DataError(RooAbsData::Poisson) and DataError(RooAbsData::Expected) when we have 0-entry bins.

Best,
Xuli

12

I don’t think so. SumW2 definitely works to retrieve errors. Did you try to run it? What’s the output? There might be a programming error in the Chi2Var, but empty bins should just be skipped if both data and prediction are empty. If only the error is zero, there’s obviously a problem, but with SumW2 the error can only be zero if the prediction is zero.

13

Hi Stephan,

What I got is: [#0] ERROR:Eval – RooChi2Var::RooChi2Var(chi2) INFINITY ERROR: bin 41 has zero error.

So it seems like RooChi2Var just won’t skip bins with 0 errors. I am very sure that my predictions on that bin is non-zero (it is like 0.8 events are expected, from the plot) but observed data is 0. I am a little bit confused, did I misinterpret the actual meaning of “prediction is 0”? Does that mean prediction is smaller than 1 or some threshold? I mean, well, from the plot my fitted function is non zero across the whole range under study.

Thank you!

Best,
Sebastian

14

Hi @Xuli,

the infinity error you get when dataerror = 0 (which happens when data = 0), but the PDF prediction is > 0. It’s true that this doesn’t happen when you use poisson errors, because the poisson error of a bin with zero entries is still well defined. For SumW2 errors it’s unfortunately zero.
If you need weighted events and SumW2, you have to make sure that all data bins are not empty, unless the PDF is also zero in all of these bins.

When I said prediction zero / non zero, I mean that the PDF evaluates to zero / to something larger than zero.

15

Hi Stephan,

Ah okay so I think I am clear on the differences between those two methods! Thank you so much for your patient help! Really appreciate it:)

All the best!

Sebastian

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.