Parameter Uncertainties by RooAbsPdf::fitTo for weighted data depend on Minos option if SumW2Error is set too

Roofit and RooStats
1

Description of the Problem

Dear Rooters,

fitting weighted data and enabling Minos as well as SumW2Error(kTRUE) in RooAbsPdf::fitTo(), a warning is given:

[#0] WARNING:InputArguments -- RooAbsPdf::fitTo(pdfExtVggSig) WARNING: sum-of-weights correction does not apply to MINOS errors

This implies, that you can use Minos, but it will not improve the corrected result. However, if I run my fit with and without Minos one parameter error in my example changes by a factor of 10. With Minos enabled the error is smaller and does not agree with the square root of the covariance matrix element.
Here is the result of the fit with and without the Minos flag, taken from my example macro:

----- COMPARE THE FIT RESULTS  -----

Have look at the error of vggSigStdDevFit in both cases
WITH MINOS AND SUMW2ERROR ENABLED:

  RooFitResult: minimized FCN value: -94226, estimated distance to minimum: 53813.2
                covariance matrix quality: Unknown, matrix was externally provided
                Status : MINIMIZE=0 HESSE=0 HESSE=0 MINOS=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
  nSigFit                 2.0184e+04 +/-  1.18e+02
  vggSigMeanFit           1.2002e+02 +/-  3.37e-02
  vggSigStdDevFit         4.9183e+00 +/-  7.96e-03

Covariance for nSigFit:            29052.3      Sqrt(cov):   170.447   
Covariance for vggSigMeanFit:      0.00171763   Sqrt(cov):   0.0414443 
Covariance for vggSigStdDevFit:    0.00779129   Sqrt(cov):   0.0882683 
=> Parameter error and Sqrt of covariance matrix element do NOT agree

WITH MINOS DISABLED AND SUMW2ERROR ENABLED:

  RooFitResult: minimized FCN value: -94221.3, estimated distance to minimum: 53718.2
                covariance matrix quality: Unknown, matrix was externally provided
                Status : MINIMIZE=0 HESSE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
  nSigFit                 2.0184e+04 +/-  1.70e+02
  vggSigMeanFit           1.2002e+02 +/-  4.14e-02
  vggSigStdDevFit         4.9181e+00 +/-  8.83e-02

Covariance for nSigFit:            29052.3      Sqrt(cov):   170.447   
Covariance for vggSigMeanFit:      0.00171748   Sqrt(cov):   0.0414425 
Covariance for vggSigStdDevFit:    0.00778953   Sqrt(cov):   0.0882583 
=> Paramter error and Sqrt of covariance matrix element do agree

If Minos is run, the error is seemingly taken from the modified covariance matrix needed to caclulate the SumW2Error correction. See Line 211 to 228 in the attached log file and compare them to the result in RooFitResPtr (Line 378-380):
log.txt (18.6 KB)

Is this behavior intended? If not could it be fixed, or the user be warned, that he should not use the fit result if SumW2Error and Minos are enabled simultaneously?

On the Example

  • 2 Variables:
    • Discriminating variable mass (Exponential Background, Gaussian Signal)
    • Control variable (Gaussian Background, Gaussian Signal)
  • The Configurator class holds the Parameter values. The model is then build via the ModelBuilder class. But only the main macro CheckIfMinosChangesParrErr.C should be of interest.
  • To run the macro youself

ROOT Version

  • ROOT version: 6.16.00 , 6.20.00
2

Hi @thucking,

my first guess is that minos just runs without understanding that weights have an effect on the errors. It just scans the likelihood, and returns a value.
@moneta might know details.

I understand the warning as follows:
“The user asked for a correction of errors in the presence of weights, but we cannot apply this to minos errors.”

Regarding

I guess the authors of the warning intended to say that there is no mathematically sound way to correct minos errors in the presence of weights, and that the request cannot be handled.

I see a few options:

  1. We rephrase the warning message to make it more understandable.
  2. Maybe we let fitTo error out when a user asks for minos and weights correction.
  3. Something can be done for minos with weights, and we try to implement.

BTW:
You might be interested in the new asymptotically correct approach. It’s also correcting errors, but it’s much more accurate in certain conditions. It is included since v6.20. (edited)
https://root.cern.ch/doc/master/rf611__weightedfits_8C.html

3

Hello

When doing weighted likelihood fits you should apply the error correction which works only for Hessian errors, not Minos errors. MINOS does not work in this case, this is the sense of the warning and it should not be used.
If you have very non-parabolic likelihood then there is no real solution, since also there is no clear procedure how to resample the data with pseudo-experiments when you have weights, since there is often no distribution for the weights.

Lorenzo

4

Hi @StephanH,

thanks a lot for your fast and detailed reply.

Regarding

I can agree with that statement. But I would expect one of the following things to happen:

  1. Minos in overwriting the error calculated by SumW2Errror
  2. The Error from SumW2Error is kept, but asymmetric errors are overwritten (those accessed by RooFitResult::getErrorHi() and RooFitResult::getErrorLo()
  3. Minos overwrites everything done by SumW2Error

But none of this is happening. The error is taken from intermediate execution of Hesse, to calculate the C matrix Vcorr=VC^{−1}V.

Therefore I would personally prefer option 2 over option 1 from the solutions proposed. Option 3 would be of course preferable over option 2 ;), but I am not sure if it is possible.

Thanks a lot. This is really interesting.

5

Hello @moneta,

thanks for you reply.

If Sum2Error and Minos should not be used together it would be most save to let fitTo error out. I thought I would only waste CPU time, but not change the result of SumW2Error (as mentioned in my last post). I would be happy if this pit fall could be closed. Personally I was looking at some pulls for few month and after dropping Minos they changed drastically. I would like to spare other people that :wink:

6

Hi,
I agree with you. Until we don’t have a way to do the third option that @StephanH mentioned, we will implement the second one.

Lorenzo

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.