RooFitResult: fit parameter Error does not match to the error from the covariance matrix

Roofit and RooStats
1

Dear experts,

I am doing the likelihood fit of data histograms with 6 templates.
Fit parameters are the normalization factors for all templates which are not allowed to be negative.

When I print the RooFitResult I see that one of the fit parameter is close to zero and have some uncertainty which does not match to the square root from the corresponding element in the covariance matrix.

Here is the printout of RooFitResult:

  RooFitResult: minimized FCN value: -280.946, estimated distance to minimum: 4.91725e-08
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                   #mu    1.4571e+00 +/-  1.92e-01
                #mutop    6.1421e+00 +/-  3.23e+00
                 #mutw    1.3732e-07 +/-  4.79e+00
                 #muvv    2.0913e+01 +/-  4.52e+00
                  #muw    1.0742e+00 +/-  2.94e-02
                  #muz    2.4179e+00 +/-  2.83e-01

and here is the printout of covarianceMatrix:

6x6 matrix is as follows

     |      0    |      1    |      2    |      3    |      4    |
----------------------------------------------------------------------
   0 |    0.03676     -0.1251  -6.379e-07      -0.443    0.001705 
   1 |    -0.1251       10.48  -1.961e-06      -4.054    -0.01756 
   2 | -6.379e-07  -1.961e-06   2.674e-06  -1.432e-05   3.131e-08 
   3 |     -0.443      -4.054  -1.432e-05       20.51    -0.04668 
   4 |   0.001705    -0.01756   3.131e-08    -0.04668   0.0008639 
   5 |   0.001916     -0.1181   6.124e-08     -0.2566   -0.004253 


     |      5    |
----------------------------------------------------------------------
   0 |   0.001916 
   1 |    -0.1181 
   2 |  6.124e-08 
   3 |    -0.2566 
   4 |  -0.004253 
   5 |    0.07985

here we can see that 3rd fit parameter (#mutw) error is printed to be 4.79, while square root from corresponding element in the covariance matrix is very different:
sqrt(2.674e-06)=0.0016

For other fit parameters the errors agree with square root from corresponding element in the covariance matrix.

I was expecting the fit errors are determined from the covariance matrix, but it does not seem to be so.

Can you please tell me why we have this mismatch between the fit parameter error and sqrt from the corresponding covariance matrix element?

Best regards,
Archil

2

Dear @Archil ,

Thanks for posting on the forum! I believe @moneta could help with your question.
Cheers,
Vincenzo

3

Hello,

This is very strange, to understand it better, I would need to see the full log of the fit. Can you increase also the print level, by using the RooFit command line option RooFit::PrintLevel(1) ?

Cheers

Lorenzo

4

Dear @vpadulan and @moneta,

thanks a lot for your replies.

unfortunately setting the RooFit::PrintLevel(1) did not increase the details of the printout.
I am not sure how to provide the full log of the fit so I have attached the root file containing the RooFitResult object:
test.root (8.7 KB)

Best regards,
Archil

5

Hello,

I would like to see the full printout of the Minimization, if you can attach that it would be helpful. If you don’t see a difference it is probably because you have already a verbose logging.
From the RooFitResult I can see that your third parameter has a fit value close to its lower limit. This can cause a wrong estimation of the error. You would need to run Minos, or if you can lower the bound a little bit allowing the fitted parameter to become negative

Lorenzo

6

Dear @moneta,

I am using the TRExFitter for the fits so I am not sure how to print the full log of the fit.

I have tried to run the minimization using the RooWorkspace created by the TRExFitter, and here is the printed output:

[#1] INFO:Minimization -- p.d.f. provides expected number of events, including extended term in likelihood.
RooAbsTestStatistic::initSimMode: creating slave calculator #0 for state CR1_BDTG_WjZj (10 dataset entries)
RooAbsTestStatistic::initSimMode: creating slave calculator #1 for state CR2_BDTG_WjVV (10 dataset entries)
RooAbsTestStatistic::initSimMode: creating slave calculator #2 for state CR3_BDTG_WjST (10 dataset entries)
RooAbsTestStatistic::initSimMode: creating slave calculator #3 for state CR4_BDTG_tt (8 dataset entries)
RooAbsTestStatistic::initSimMode: creating slave calculator #4 for state SR_BDTG_tt (8 dataset entries)
[#1] INFO:Fitting -- RooAbsTestStatistic::initSimMode: created 5 slave calculators.
[#1] INFO:Minimization --  Including the following constraint terms in minimization: (lumiConstraint,alpha_DummyConstraint)
[#1] INFO:Minimization -- The following global observables have been defined and their values are taken from the model: (nominalLumi,nom_alpha_Dummy)
[#1] INFO:Minimization --  The following expressions have been identified as constant and will be precalculated and cached: (ttbar_CR1_BDTG_WjZj_shapes,Wjets_CR1_BDTG_WjZj_shapes,Zjets_CR1_BDTG_WjZj_shapes,Top_CR1_BDTG_WjZj_shapes,tW_CR1_BDTG_WjZj_shapes,VV_CR1_BDTG_WjZj_shapes)
[#1] INFO:Minimization --  The following expressions have been identified as constant and will be precalculated and cached: (ttbar_CR2_BDTG_WjVV_shapes,Wjets_CR2_BDTG_WjVV_shapes,Zjets_CR2_BDTG_WjVV_shapes,Top_CR2_BDTG_WjVV_shapes,tW_CR2_BDTG_WjVV_shapes,VV_CR2_BDTG_WjVV_shapes)
[#1] INFO:Minimization --  The following expressions have been identified as constant and will be precalculated and cached: (ttbar_CR3_BDTG_WjST_shapes,Wjets_CR3_BDTG_WjST_shapes,Zjets_CR3_BDTG_WjST_shapes,Top_CR3_BDTG_WjST_shapes,tW_CR3_BDTG_WjST_shapes,VV_CR3_BDTG_WjST_shapes)
[#1] INFO:Minimization --  The following expressions have been identified as constant and will be precalculated and cached: (ttbar_CR4_BDTG_tt_shapes,Wjets_CR4_BDTG_tt_shapes,Zjets_CR4_BDTG_tt_shapes,Top_CR4_BDTG_tt_shapes,tW_CR4_BDTG_tt_shapes,VV_CR4_BDTG_tt_shapes)
[#1] INFO:Minimization --  The following expressions have been identified as constant and will be precalculated and cached: (ttbar_SR_BDTG_tt_shapes,Wjets_SR_BDTG_tt_shapes,Zjets_SR_BDTG_tt_shapes,Top_SR_BDTG_tt_shapes,tW_SR_BDTG_tt_shapes,VV_SR_BDTG_tt_shapes)
[#1] INFO:Fitting -- RooAddition::defaultErrorLevel(nll_simPdf_obsData_with_constr) Summation contains a RooNLLVar, using its error level
[#1] INFO:Minimization -- RooAbsMinimizerFcn::setOptimizeConst: activating const optimization
Minuit2Minimizer: Minimize with max-calls 3500 convergence for edm < 1 strategy 1
Info in <Minuit2>: MnSeedGenerator Computing seed using NumericalGradient calculator
[#1] INFO:Minimization -- RooNLLVar::evaluatePartition(CR1_BDTG_WjZj) first = 0 last = 10 Likelihood offset now set to 7766.13
[#1] INFO:Minimization -- RooNLLVar::evaluatePartition(CR2_BDTG_WjVV) first = 0 last = 10 Likelihood offset now set to 1422.32
[#1] INFO:Minimization -- RooNLLVar::evaluatePartition(CR3_BDTG_WjST) first = 0 last = 10 Likelihood offset now set to 1469.18
[#1] INFO:Minimization -- RooNLLVar::evaluatePartition(CR4_BDTG_tt) first = 0 last = 8 Likelihood offset now set to 1014.87
[#1] INFO:Minimization -- RooNLLVar::evaluatePartition(SR_BDTG_tt) first = 0 last = 8 Likelihood offset now set to 1003.17
Info in <Minuit2>: MnSeedGenerator Initial state: FCN =     -0.4647080266 Edm =      -1290.472608 NCalls =     29
Info in <Minuit2>: NegativeG2LineSearch Doing a NegativeG2LineSearch since one of the G2 component is negative
Info in <Minuit2>: MnSeedGenerator Negative G2 found - new state: 
  Minimum value : -207.0313734
  Edm           : 555.0768522
  Internal parameters:	[     -1.370461484    -0.8368124947     -1.013527059    -0.9392015524     -1.370461484     -1.370461484                0]	
  Internal gradient  :	[     -385.3448371     -126.9227712     -73.89901187     -130.1838595     -7356.334857     -837.4527755                0]	
  Internal covariance matrix:
[[  0.00030535353              0              0              0              0              0              0]
 [              0    0.011531951              0              0              0              0              0]
 [              0              0    0.021671137              0              0              0              0]
 [              0              0              0    0.038365674              0              0              0]
 [              0              0              0              0  4.6955167e-06              0              0]
 [              0              0              0              0              0   0.0013781427              0]
 [              0              0              0              0              0              0    0.080000106]]]
Info in <Minuit2>: MnSeedGenerator Initial state  
  Minimum value : -207.0313734
  Edm           : 555.0768522
  Internal parameters:	[     -1.370461484    -0.8368124947     -1.013527059    -0.9392015524     -1.370461484     -1.370461484                0]	
  Internal gradient  :	[     -385.3448371     -126.9227712     -73.89901187     -130.1838595     -7356.334857     -837.4527755                0]	
  Internal covariance matrix:
[[  0.00030535353              0              0              0              0              0              0]
 [              0    0.011531951              0              0              0              0              0]
 [              0              0    0.021671137              0              0              0              0]
 [              0              0              0    0.038365674              0              0              0]
 [              0              0              0              0  4.6955167e-06              0              0]
 [              0              0              0              0              0   0.0013781427              0]
 [              0              0              0              0              0              0    0.080000106]]]
Info in <Minuit2>: VariableMetricBuilder Start iterating until Edm is < 0.001 with call limit = 3500
Info in <Minuit2>: VariableMetricBuilder    0 - FCN =      -207.0313734 Edm =       555.0768522 NCalls =    165
Info in <Minuit2>: VariableMetricBuilder    1 - FCN =       -271.818145 Edm =       35.44425486 NCalls =    187
Info in <Minuit2>: VariableMetricBuilder    2 - FCN =      -285.1427466 Edm =       1.645456416 NCalls =    203
Info in <Minuit2>: VariableMetricBuilder    3 - FCN =      -288.5749699 Edm =      0.9908530248 NCalls =    223
Info in <Minuit2>: VariableMetricBuilder    4 - FCN =      -289.8092066 Edm =      0.1492127473 NCalls =    240
Info in <Minuit2>: VariableMetricBuilder    5 - FCN =      -290.0558553 Edm =     0.09165506029 NCalls =    256
Info in <Minuit2>: VariableMetricBuilder    6 - FCN =      -290.2124467 Edm =     0.04249761954 NCalls =    272
Info in <Minuit2>: VariableMetricBuilder    7 - FCN =      -290.2762099 Edm =     0.02926391403 NCalls =    289
Info in <Minuit2>: VariableMetricBuilder    8 - FCN =      -290.3999072 Edm =     0.01734148502 NCalls =    307
Info in <Minuit2>: VariableMetricBuilder    9 - FCN =       -290.419307 Edm =    0.002677926088 NCalls =    323
Info in <Minuit2>: VariableMetricBuilder   10 - FCN =      -290.4211681 Edm =    0.000779832486 NCalls =    339
Info in <Minuit2>: VariableMetricBuilder   11 - FCN =      -290.4225457 Edm =   7.215752011e-05 NCalls =    355
Info in <Minuit2>: VariableMetricBuilder After Hessian
Info in <Minuit2>: VariableMetricBuilder   12 - FCN =      -290.4225457 Edm =   9.938653785e-05 NCalls =    405
Minuit2Minimizer : Valid minimum - status = 0
FVAL  = -290.422545690361346
Edm   = 9.93865378540374582e-05
Nfcn  = 405
#mu	  = 1.45643	 +/-  0.174111	(limited)
#mutop	  = 8.38406	 +/-  3.12646	(limited)
#mutw	  = 5.58026e-06	 +/-  2.406	(limited)
#muvv	  = 15.5462	 +/-  4.10719	(limited)
#muw	  = 1.07678	 +/-  0.0286011	(limited)
#muz	  = 2.36066	 +/-  0.277938	(limited)
alpha_Dummy	  = 0	 +/-  0.993348	(limited)

indeed, the fit value for the 3rd parameter is close to its lower limit.
Minos gives the upper error close to the one printed above:

Info in <Minuit2>: MnMinos end of Minos scan for up interval for parameter #mutw
Minos:  Parameter : #mutw  is at Lower limit; error is -5.58026e-06
Minos: Lower error for parameter #mutw  :  -5.58026e-06
Minos: Upper error for parameter #mutw  :  2.24958
Warning in <Minuit2>: RunMinosError Lower error for parameter #mutw is at the Lower limit!

If I allow the 3rd parameter to be negative, then it becomes #mutw -1.2886e+01 +/- 7.85e+00 and the error matches to the sqrt from corresponding element in the covariance matrix.

Since the negative value of fit parameters are unphysical, I set the lower bounds at zero, and then I guess only the upper error from minos makes sense.

Best regards,
Archil

7

Thank you for the log. It is already in verbose mode. What you observe is expected, since at the border the non-linearity of the parameter transformation used by Minuit is noticeable. Due to this transformation to move from internal (unbounded) parameters to bounded one the estimated error at the border will not be equal to the square root of the covariance matrix. As I said the covariance matrix will not be in this case reliable and you should use Minos.
Actually, I would argue that the correct uncertainty will be the one when you allow the fitted parameter to be negative. It is the true parameter that cannot be negative and not the measured one.

Best,

Lorenzo

8

Thanks a lot Lorenzo, you helped me to understand better these things.

Best regards,
Archil

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.