BatchMode gives wrong errors with SumW2Errors

BatchMode gives different errors for fits to weighted data if SumW2Errors is active. Reproducer below.


# %%
import ROOT as r

ws = r.RooWorkspace("workspace")
x = ws.factory("x[-10, 10]")
sig = ws.factory("Gaussian::sig(x, mu[-1, 1], s[0.1, 5])")
bkg = ws.factory("Chebychev::bkg(x, {c1[0.1, -1, 1]})")
shp = ws.factory("SUM::shp(Nsig[0, 20000] * sig, Nbkg[0, 20000] * bkg)")
_data = shp.generate(r.RooArgSet(x))
wFunc = ws.factory(f'FormulaVar::w("@0 * @0 + 10", {{{x.GetName()}}})')
w = _data.addColumn(wFunc)
data = r.RooDataSet(_data.GetName(), _data.GetTitle(), _data, _data.get(), "", w.GetName())
datahist = r.RooDataHist("datahist", "datahist", data.get(), data)

# %%
print("with BatchMode and with SumW2Error:")
yy = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(True),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(True),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
print("without BatchMode and with SumW2Error:")
ny = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(True),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(False),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
print("with BatchMode and without SumW2Error:")
yn = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(False),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(True),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
print("without BatchMode and without SumW2Error:")
nn = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(False),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(False),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)

# %%
print("with BatchMode and with SumW2Error:")
yy.Print()
print("without BatchMode and with SumW2Error:")
ny.Print()
print("with BatchMode and without SumW2Error:")
yn.Print()
print("without BatchMode and without SumW2Error:")
nn.Print()

Relevant output:

with BatchMode and with SumW2Error:

  RooFitResult: minimized FCN value: -4.42332e+06, estimated distance to minimum: 0.000130498
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    2.0000e+04 +/-  5.89e-04
                  Nsig    2.0000e+04 +/-  1.05e-03
                    c1    2.8927e-01 +/-  2.89e-03
                    mu   -1.0000e+00 +/-  1.98e-07
                     s    5.0000e+00 +/-  1.34e-07

without BatchMode and with SumW2Error:

  RooFitResult: minimized FCN value: -4.42332e+06, estimated distance to minimum: 1239.4
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    2.0000e+04 +/-  2.87e-08
                  Nsig    2.0000e+04 +/-  1.02e-07
                    c1    2.8928e-01 +/-  2.64e-02
                    mu   -1.0000e+00 +/-  5.50e-10
                     s    5.0000e+00 +/-  6.94e-10

with BatchMode and without SumW2Error:

  RooFitResult: minimized FCN value: -4.42332e+06, estimated distance to minimum: 1.66733e-07
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    2.0000e+04 +/-  3.13e-02
                  Nsig    2.0000e+04 +/-  4.24e-02
                    c1    2.8928e-01 +/-  2.89e-03
                    mu   -1.0000e+00 +/-  1.27e-04
                     s    5.0000e+00 +/-  2.23e-05

without BatchMode and without SumW2Error:

  RooFitResult: minimized FCN value: -4.42332e+06, estimated distance to minimum: 1.55437e-07
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    2.0000e+04 +/-  3.13e-02
                  Nsig    2.0000e+04 +/-  4.24e-02
                    c1    2.8928e-01 +/-  2.89e-03
                    mu   -1.0000e+00 +/-  1.27e-04
                     s    5.0000e+00 +/-  2.23e-05

Hi,
Which ROOT version are you using ? I think you would need to use the ROOT master version, having this latest fix: [RF] Implement SumW2 correction in new BatchMode with RooFitDriver by guitargeek · Pull Request #10229 · root-project/root · GitHub

Lorenzo

6.26/00. I have now tried it with 6.26/02, and the problem persists, though the exact values are different. I do not have a good way to test the master build right now.


Relevant output using 6.26/02:

with BatchMode and with SumW2Error:
without BatchMode and with SumW2Error:
with BatchMode and without SumW2Error:
without BatchMode and without SumW2Error:

  RooFitResult: minimized FCN value: -4.42332e+06, estimated distance to minimum: 2.53409e+06
                covariance matrix quality: Full matrix, but forced positive-definite
                Status : MINIMIZE=0 HESSE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    2.0000e+04 +/-  4.07e-03
                  Nsig    2.0000e+04 +/-  1.87e-07
                    c1    2.8927e-01 +/-  3.70e-02
                    mu   -1.0000e+00 +/-  6.62e-06
                     s    5.0000e+00 +/-  2.09e-06


  RooFitResult: minimized FCN value: -4.42332e+06, estimated distance to minimum: 1239.41
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    2.0000e+04 +/-  4.03e-09
                  Nsig    2.0000e+04 +/-  1.47e-05
                    c1    2.8928e-01 +/-  2.64e-02
                    mu   -1.0000e+00 +/-  2.90e-09
                     s    5.0000e+00 +/-  2.28e-10


  RooFitResult: minimized FCN value: -4.42332e+06, estimated distance to minimum: 1.57384e-07
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    2.0000e+04 +/-  3.13e-02
                  Nsig    2.0000e+04 +/-  4.24e-02
                    c1    2.8928e-01 +/-  2.89e-03
                    mu   -1.0000e+00 +/-  1.27e-04
                     s    5.0000e+00 +/-  2.23e-05


  RooFitResult: minimized FCN value: -4.42332e+06, estimated distance to minimum: 1.55437e-07
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    2.0000e+04 +/-  3.13e-02
                  Nsig    2.0000e+04 +/-  4.24e-02
                    c1    2.8928e-01 +/-  2.89e-03
                    mu   -1.0000e+00 +/-  1.27e-04
                     s    5.0000e+00 +/-  2.23e-05

Hi @mwilkins , @jonas might be able to help, let’s ping him.

Cheers,
Enrico

Hi,
The fix mentioned above it is applied only in the ROOT master version

Lorenzo

Hi! Actually the problem with the batch mode is fixed also in 6.26.02, but in your example there is another problem: your fit just doesn’t converge because the fit function is not appropriate. You can already see this for your fit results that are fishy, because you have the parameter limits as final values.

You are creating a datahist according to a RooFit pdf, and then you weight the data depending on x, completely distorting the shape. Then you attempt to fit the original model, which doesn’t fit anymore. If you have a model that fits the data, the SumW2 correction should work also with the BatchMode.

One other advice for these comparison studies: after each call to fitTo, you should reset the initial parameter values and errors to make sure the fit results are not different because of the starting conditions. You can use RooRealVar::setVal(<initial value of your choice>) and RooRealVar::setError(0.0) for this.

Hi, @jonas,

If I replace the weight function with a fixed value, the problem persists, suggesting that fit quality is not the issue. Changing wFunc to

wFunc = ws.factory('w[0.1]')

results in

with BatchMode and with SumW2Error:
without BatchMode and with SumW2Error:
with BatchMode and without SumW2Error:
without BatchMode and without SumW2Error:

  RooFitResult: minimized FCN value: -7504.22, estimated distance to minimum: 1.64472e-05
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    9.8896e+02 +/-  2.80e+01
                  Nsig    1.0112e+03 +/-  2.84e+01
                    c1    9.0614e-02 +/-  1.99e-02
                    mu   -6.6719e-02 +/-  4.05e-02
                     s    2.5354e+00 +/-  4.31e-02


  RooFitResult: minimized FCN value: -7504.22, estimated distance to minimum: 1.32625e-05
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    9.8896e+02 +/-  1.84e+01
                  Nsig    1.0112e+03 +/-  1.85e+01
                    c1    9.0614e-02 +/-  1.99e-02
                    mu   -6.6719e-02 +/-  4.05e-02
                     s    2.5354e+00 +/-  4.29e-02


  RooFitResult: minimized FCN value: -7504.22, estimated distance to minimum: 0.000132676
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    9.8896e+02 +/-  5.83e+01
                  Nsig    1.0112e+03 +/-  5.85e+01
                    c1    9.0614e-02 +/-  6.29e-02
                    mu   -6.6719e-02 +/-  1.28e-01
                     s    2.5354e+00 +/-  1.36e-01


  RooFitResult: minimized FCN value: -7504.22, estimated distance to minimum: 0.000132675
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    9.8896e+02 +/-  5.83e+01
                  Nsig    1.0112e+03 +/-  5.85e+01
                    c1    9.0614e-02 +/-  6.29e-02
                    mu   -6.6719e-02 +/-  1.28e-01
                     s    2.5354e+00 +/-  1.36e-01


I plot the fit for completeness:
test.pdf (23.9 KB)

As an aside, I’m not sure why you say the fit doesn’t converge in my original formulation. Yes, the fit looks bad, but you can see that MINIMIZE and HESSE return successes.


Full reproducer:

# %%
import ROOT as r

ws = r.RooWorkspace("workspace")
x = ws.factory("x[-10, 10]")
sig = ws.factory("Gaussian::sig(x, mu[-1, 1], s[0.1, 5])")
bkg = ws.factory("Chebychev::bkg(x, {c1[0.1, -1, 1]})")
shp = ws.factory("SUM::shp(Nsig[0, 20000] * sig, Nbkg[0, 20000] * bkg)")
_data = shp.generate(r.RooArgSet(x))
wFunc = ws.factory('w[0.1]')
w = _data.addColumn(wFunc)
data = r.RooDataSet(_data.GetName(), _data.GetTitle(), _data, _data.get(), "", w.GetName())
datahist = r.RooDataHist("datahist", "datahist", data.get(), data)


# %%
inVals = {v.GetName(): v.getVal() for v in ws.allVars()}
def resetVals():
    for k, v in inVals.items():
        ws.var(k).setVal(v)
        ws.var(k).setError(0)

# %%
print("with BatchMode and with SumW2Error:")
resetVals()
yy = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(True),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(True),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
yy.Print()
print("without BatchMode and with SumW2Error:")
resetVals()
ny = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(True),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(False),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
ny.Print()
print("with BatchMode and without SumW2Error:")
resetVals()
yn = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(False),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(True),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
yn.Print()
print("without BatchMode and without SumW2Error:")
resetVals()
nn = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(False),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(False),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
nn.Print()

# %%
c = r.TCanvas()
fr = x.frame()
datahist.plotOn(fr)
shp.plotOn(fr)
fr.Draw()
c.Draw()
c.SaveAs("test.pdf")

# %%

Using the nightlies (reported as 6.27/01), I get the following:

with BatchMode and with SumW2Error:
  RooFitResult: minimized FCN value: -7504.22, estimated distance to minimum: 0.000132715
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    9.8896e+02 +/-  5.82e+01
                  Nsig    1.0112e+03 +/-  5.84e+01
                    c1    9.0614e-02 +/-  6.30e-02
                    mu   -6.6719e-02 +/-  1.28e-01
                     s    2.5354e+00 +/-  1.36e-01

without BatchMode and with SumW2Error:
  RooFitResult: minimized FCN value: -7504.22, estimated distance to minimum: 1.32625e-05
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    9.8896e+02 +/-  1.84e+01
                  Nsig    1.0112e+03 +/-  1.85e+01
                    c1    9.0614e-02 +/-  1.99e-02
                    mu   -6.6719e-02 +/-  4.05e-02
                     s    2.5354e+00 +/-  4.29e-02

with BatchMode and without SumW2Error:
  RooFitResult: minimized FCN value: -7504.22, estimated distance to minimum: 0.000132676
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    9.8896e+02 +/-  5.83e+01
                  Nsig    1.0112e+03 +/-  5.85e+01
                    c1    9.0614e-02 +/-  6.29e-02
                    mu   -6.6719e-02 +/-  1.28e-01
                     s    2.5354e+00 +/-  1.36e-01

without BatchMode and without SumW2Error:
  RooFitResult: minimized FCN value: -7504.22, estimated distance to minimum: 0.000132675
                covariance matrix quality: Full, accurate covariance matrix
                Status : MINIMIZE=0 HESSE=0 

    Floating Parameter    FinalValue +/-  Error   
  --------------------  --------------------------
                  Nbkg    9.8896e+02 +/-  5.83e+01
                  Nsig    1.0112e+03 +/-  5.85e+01
                    c1    9.0614e-02 +/-  6.29e-02
                    mu   -6.6719e-02 +/-  1.28e-01
                     s    2.5354e+00 +/-  1.36e-01

You can see that the errors with BatchMode and SumW2Error are nearly identical to those without SumW2Error.


Full reproducer:

# %%
import ROOT as r

ws = r.RooWorkspace("workspace")
x = ws.factory("x[-10, 10]")
sig = ws.factory("Gaussian::sig(x, mu[-1, 1], s[0.1, 5])")
bkg = ws.factory("Chebychev::bkg(x, {c1[0.1, -1, 1]})")
shp = ws.factory("SUM::shp(Nsig[0, 20000] * sig, Nbkg[0, 20000] * bkg)")
_data = shp.generate(r.RooArgSet(x))
wFunc = ws.factory('w[0.1]')
w = _data.addColumn(wFunc)
data = r.RooDataSet(_data.GetName(), _data.GetTitle(), _data, _data.get(), "", w.GetName())
datahist = r.RooDataHist("datahist", "datahist", data.get(), data)


# %%
inVals = {v.GetName(): v.getVal() for v in ws.allVars()}
def resetVals():
    for k, v in inVals.items():
        ws.var(k).setVal(v)
        ws.var(k).setError(0)

# %%
print("with BatchMode and with SumW2Error:")
resetVals()
yy = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(True),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(True),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
yy.Print()
print("without BatchMode and with SumW2Error:")
resetVals()
ny = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(True),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(False),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
ny.Print()
print("with BatchMode and without SumW2Error:")
resetVals()
yn = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(False),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(True),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
yn.Print()
print("without BatchMode and without SumW2Error:")
resetVals()
nn = shp.fitTo(
    datahist,
    r.RooFit.Extended(),
    r.RooFit.Save(),
    r.RooFit.SumW2Error(False),
    r.RooFit.Strategy(1),
    r.RooFit.BatchMode(False),
    r.RooFit.Minimizer("Minuit2", "migrad"),
)
nn.Print()

# %%
c = r.TCanvas()
fr = x.frame()
datahist.plotOn(fr)
shp.plotOn(fr)
fr.Draw()
c.Draw()
c.SaveAs("test.pdf")

Hi, you are right, there was an error! It is fixed now in ROOT master and the patch will be also applied to the next 6.26 patch release, i.e. 6.26.04.

By the way, with “the fit doesn’t converge” I meant that it doesn’t converge to a meaningful value but to the parameter boundaries Sorry I wasn’t precise there.

Thanks again for reporting!

Jonas

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.