Difference in the ratio of negative log likelihoods between the simultaneous fit and fit to the combined dataset

[danabo]

Dear Roofit experts,

I generated three toy datasets (representing three years of data taking) for 3400-3800 mass spectra, please find them attached.

I wanted to compare the results of the simultaneous fit to those three datasets with the fit to the combined spectra, in particular the test statistic which I compute as the ratio of likelihoods for the background only and signal plus background models.

I noticed that the results between the simultaneous and combined fits are consistent for the significant excess (which is the injected signal at the mass of around 3620.), however they are incompatible for the rest of the spectrum. To provide a direct comparison, I fixed the mass of the fits to 3576. and compared the results for both fits - please find attached scripts for both the simultaneous fit and the fit to the combined spectra. I also fitted all years separately (in the simultaneous fit script) and extracted the likelihoods from the results.
The results for the min negative log likelihoods for the bkg and bkg+sig hypotheses are following:

2016 bkg model minNll is 255985.469663
2016 sig+bkg model minNll is 255985.469636
2017 bkg model minNll is 266767.125233
2017 sig+bkg model minNll is 266767.125234
2018 bkg model minNll is 313803.973182
2018 sig+bkg model minNll is 313801.213808
-------------------------------------------------
sum of 16/17/18 bkg model minNll is 836556.568078
sum of 16/17/18 sig+bkg model minNll is 836553.808678
simultaneous fit bkg model minNll is 990127.192555
simultaneous fit sig model minNll is 990124.464513
combined fit bkg model minNll is 836556.575171
combined fit sig+bkg model minNll is 836556.070665

As you can see, the combined fit for the bkg gives the same NLL as the sum of the 16, 17 and 18 NLLs. However, it is different for the sig+bkg model - and if you compute the test statistic from those numbers, you get better agreement between the simultaneous fit and sum of the 16, 17 and 18 NLLs. So I am a bit puzzled here - shouldn’t I get the same result for the test statistic from both simultaneous fit and fit to the combined spectra?

I did the same study with the extended likelihood fits, but the result for the test statistic is the same. The posted examples are for the non-extended ones so it is as simple as possible.
Many thanks in advance for your time and help!

Cheers,
Dana

FinalMassFitToysSimple.C (1.9 KB) ToyDataset_2017.root (321.9 KB) ToyDataset_2016.root (310.4 KB) FinalMassFitToysSimultaneousSimple.C (6.0 KB) ToyDataset_2018.root (375.8 KB)

[couet]

May be @moneta has an idea.

[danabo]

Described difference was eventually understood. If one setup the simultaneous fit in a following way:

  RooAddPdf model16("model16", "Signal plus background PDF",  signal16, bkg16, sigFrac16);
  RooAddPdf model17("model17", "Signal plus background PDF",  signal16, bkg17, sigFrac16);
  RooAddPdf model18("model18", "Signal plus background PDF",  signal16, bkg18, sigFrac16);

so the signal fraction is the same in all samples (meaning that the signal yield is constrained to be of the same size in all years - taking into account the size of the data sample), DLL (difference in NLL between the background-only and signal+background hypotheses) from the simultaneous fit agrees with the result from the combined fit, which is the quantity that is expected to be the same between the two.

[moneta]

Thank you for letting us know. What you describe makes sense to me.
Best regards
Lorenzo