Dear Roofit experts,
I generated three toy datasets (representing three years of data taking) for 3400-3800 mass spectra, please find them attached.
I wanted to compare the results of the simultaneous fit to those three datasets with the fit to the combined spectra, in particular the test statistic which I compute as the ratio of likelihoods for the background only and signal plus background models.
I noticed that the results between the simultaneous and combined fits are consistent for the significant excess (which is the injected signal at the mass of around 3620.), however they are incompatible for the rest of the spectrum. To provide a direct comparison, I fixed the mass of the fits to 3576. and compared the results for both fits - please find attached scripts for both the simultaneous fit and the fit to the combined spectra. I also fitted all years separately (in the simultaneous fit script) and extracted the likelihoods from the results.
The results for the min negative log likelihoods for the bkg and bkg+sig hypotheses are following:
2016 bkg model minNll is 255985.469663 2016 sig+bkg model minNll is 255985.469636 2017 bkg model minNll is 266767.125233 2017 sig+bkg model minNll is 266767.125234 2018 bkg model minNll is 313803.973182 2018 sig+bkg model minNll is 313801.213808 ------------------------------------------------- sum of 16/17/18 bkg model minNll is 836556.568078 sum of 16/17/18 sig+bkg model minNll is 836553.808678 simultaneous fit bkg model minNll is 990127.192555 simultaneous fit sig model minNll is 990124.464513 combined fit bkg model minNll is 836556.575171 combined fit sig+bkg model minNll is 836556.070665
As you can see, the combined fit for the bkg gives the same NLL as the sum of the 16, 17 and 18 NLLs. However, it is different for the sig+bkg model - and if you compute the test statistic from those numbers, you get better agreement between the simultaneous fit and sum of the 16, 17 and 18 NLLs. So I am a bit puzzled here - shouldn’t I get the same result for the test statistic from both simultaneous fit and fit to the combined spectra?
I did the same study with the extended likelihood fits, but the result for the test statistic is the same. The posted examples are for the non-extended ones so it is as simple as possible.
Many thanks in advance for your time and help!