Minimized NLL not corresponding to best chi2

@Jonas If I wasn’t clear having the new NLL definition in the new root release would be fantastic…

I can build from the nighties if required (or head, let me know when available).

Alright I’ll let you know! The NLL definition with the offset will be available soon.

In the meantime, I have realized there is another problem in the way that RooFit defines likelihoods.

I will elaborate on it here before opening a GitHub issue for this, because here I can use LaTeX formulas which is quite nice.

If you have a continuous function f that you use to model some binned data, the non-extended NLL with the correct predicted probabilities p_i would be:

l = \sum_i{n_i\log{p_i}} = -\sum_i{n_i\log\left(\frac{1}{A}\int_{a_i}^{b_i} f dx\right)},

where n_i is the number of observations in each bin, \int_{a_i}^{b_i} f dx is the integral of f over the bin, and A = \int f dx is the normalization factor (integral over the full domain of x, the union of all bins).

Because the integration for each bin is inefficient, one usually just takes the value of f at the bin center, let’s call it \tilde{f}_i. The actual bin integral is this value times an arbitrary bias: b_i\tilde{f}_i.

The correct NLL is therefore:

l = - \sum_i n_i{\log{\left(\frac{1}{A}b_i\tilde{f}_i\right)}} = - \sum_i n_i{\log{\tilde{f}_i}} - \sum_i n_i \log{b_i} + N\log{A},

where N = \sum_i n_i.

Moving on to the RooFit likelihood. The biases are unknown, and the likelihood is approximated by dropping them.

l_\text{RooFit} = - \sum_i n_i{\log{\tilde{f}_i}} + N\log{A}.

Note for later that no approximation was done for the normalization integral!

Let’s go back to the correct NLL, but this time also express the integral A as a sum with biases for the binning effects, A = \sum_i b_i \tilde{f}_i:

l = - \sum_i n_i{\log{\tilde{f}_i}} - \sum_i n_i \log{b_i} + N\log{\sum_i b_i \tilde{f}_i}

We now split the bias into an average bias B times a bin-depended part \epsilon_i, i.e., b_i = B\epsilon_i:

l = - \sum_i n_i{\log{\tilde{f}_i}} - \sum_i n_i \log{B\epsilon_i} + N\log{\sum_i B\epsilon_i \tilde{f}_i}

If you pull the mean bias B out of the sum and the log, the terms with B cancel:

l = - \sum_i n_i{\log{\tilde{f}_i}} - \sum_i n_i \log{\epsilon_i} + N\log{\sum_i \epsilon_i \tilde{f}_i}.

Now it becomes a bit more vague. For most analytical PDF shapes, the constant bias is more important than the bin-variation of the bias. Think for example of an exponential decay: the binning effect bias is a flat multiplicative factor. Therefore RooFit looses quite some fit precision by doing the normalization integral analytically over the full domain: the cancellation of the mean bias term doesn’t happen.

In conclusion: it would also help if the NLL in RooFit for binned fits is reformulated such that the normalization integral is only approximated by evaluating at the bin centers. Like this, the average binning effects are cancelling out because they appear both in the numerator and denominator. Only bin-dependent binning effects are remaining then.

Besides reducing binning effects, this would improve numerical fit stability because no numeric integration of the PDF function needs to be done in the integral is not known analytically.

Sorry for the long text! It’s mainly for myself to not forget what I should do after the Christmas break.

The associated GitHub issue is this one:

Hi @jonas , I’m wondering when this update is planned to be included, is it the 6.28/00 release which is planned for May? If it is, is there any possibility of this being added to a nightly sooner?

Thanks for your work on this,
Jack

Hi @jonas

We are using the RooBinSamplingPDF to deal with some of the issues with the approximation of using the bin centre. Hopefully the above problem is handled correctly in this case…

Hi @JackLindon,

the implementation of the more stable likelihood definition has been in the nightlies for a few weeks now, but I haven’t reported on it yet. I first wanted to also write a tutorial to explain all of that.

The tutorial is proposed here:

You can try it with the ROOT nightlies, otherwise the Offset("bin") feature is not there yet.

Can you try if it works for you? Also your feedback on the tutorial is greatly appreciated!

Hi @jonas , great thanks very much :slight_smile: . I will try to run with a recent nightly and use it, will let you know how it goes.

@Jonas,

Thanks, this looks great. I have run some test using my standard fits to the dijet mass spectrum. So far the new Offset("bin") method works really well for the resulting fit parameters.

I have compared results with my own stand alone Minuit fitting code and get the same fit parameters within a small variation which is probably due to choices of integrator (the NLL value differs by ~0.1 out of 31.8).

I am using this in conjunction with RooBinSamplingPdf to get the integration over the bin.

One problem I am having is in making a pull distribution from a RooPlot.

In this case the resulting pull distribution is quite different from the true pull distribution I obtain by evaluating the resulting fit directly using the resulting fit parameters.

The RooPlot distribution (first plot) and my alternative plots are attached.

In conclusion the fitting appears to be working perfectly but the pdf that is stored in association with the fit has issues. If it is using a weighted bin centre rather than the integrated pdf this may be causing it.I have seen this before and will see if I can dig out the fix.

I can send my code privately or attach here. It is long and complicated so probably does not belong here.