One sigma error in the parameter is defined as the value which change the chi2 value by one. N-sigma error change the value of the chi2 by N^2, because around the minimum the chi2 can be approximated as a parabola.
SetErrorDef defines the upper value of the chi2 function, which we use to compute the sigma.
For the contours, in reality, we should not speak in term of sigma, but of probability content of the two parameters considered in the contour. There is a simple formula to relate the upper value of the chi2 to the desired probability content:
Up = chisquare_quantile (prob, 2)
This is due to the fact that, if we consider 2 parameters, the chi2 of the fit is distributed as a chi2 distribution with 2 degree of freedom. You can find more information on this in the F. James statistics book, or if you don’t have in this note: seal.web.cern.ch/seal/documents/ … nerror.pdf
So what exactly is that I plot when I plot the points from:
(ROOT::Math::Minimizer) minimizer->Contour(0, 1, numPoints, xs, ys) using say SetErrorDef(4)?
for SetErrorDef(4) you will plot the contour corresponding to a confidence level of 86.5 %
(the value is obtained from the cumulative chi2 distribution function for 2 degree of freedom at the value 4,
ROOT::Math::chisquared_cdf(4,2) )
OK. I want to be certain on what the confidence level means.
From what I understand the 86.5% confidence level means that the probability that the true/real value of the “parameter vector” lies in the confidence region (“interval”) is 86.5%.
In other words, fitting 100 datasets the “parameter vector” would lie in the confidence region approximatelly 87 times.
This is wrong !!!
Here we are considering confidence intervals using frequentist statistics. In frequentist statistics you can not make any statement on the probability of the true parameter value, but only on the probability of observing data given a a parameter value.
A confidence interval is derived then from then the probability of estimating a certain parameter M, given a true parameter value M_true and it is defined at a confidence level or coverage probability.
The correct statement is, in an ensemble experiments repeated with the same conditions, 86.5% of the time, the obtained interval will contain the true parameter values.
Statements on true parameter probability given the data can be made only in Bayesian statistics, but in this case a prior distribution on the true parameters must be assumed. In this case, one estimates an interval from the posterior probability which is called “credible interval”.
OK. I read the Wikipedia article.
Now is this right:
From an experiment we get a parameter estimate [a, b] with 86.5% confidence. That is, there is 86.5% chance the estimate is right. In other words, if we repeat the experiment 100 times we get 100 estimates (intervals), but only 86.5% of them will be right (will contain the true parameter).
Yes this is correct if a Chi2 minimization is done.
In case of a likelihood fit you have a factor of 2: 2UP(p1, p2) = 2 ( L(p1,p2) - L_min ) is distributed as a Chi2(2) ( L is the likelihood function )
MINOS errors are just the extreme points of the contour (in case of two parameters) or hypervolume (in case of more parameters). Right?
They depend on the UP value, the larger UP the larger errors. Therefore the MINOS errors should not be quoted without also noting the UP value (1-sigma error for UP=1 etc.)
This is correct, but to be precise:
MINOS estimates errors only for one parameter at a time. It finds the extreme value x1 such
f(x1) = f_MINIMUM + UP.
If the function depends on more than one parameter, the other value are not kept fixed, but their best estimate is found, by minimizing the function in n-1 parameters.
You find more information on Minos in seal.web.cern.ch/seal/documents/ … nerror.pdf and in
paragraph 1.5.3 of seal.web.cern.ch/seal/documents/ … sguide.pdf.
The Contour method (MnContour) uses the same algorithms but for estimating the points (x1,y1) such that
f(x1,y1) = f_MINIMUM + UP.
There is no method in Minuit for estimating hypervolume contours for the case of more that 2 parameters.
The Minos error will depend then of UP, but normally, if UP is not quoted, it is assumed that the value corresponding to the 1-sigma error has been used.
In case of contour, I think, it should be always quoted the probability content (the confidence level of the contour).
Shouldn’t be then the MINOS errors called confidence intervals for individual parameters?
[quote=“moneta”]This is correct, but to be precise:
MINOS estimates errors only for one parameter at a time. It finds the extreme value x1 such
f(x1) = f_MINIMUM + UP.
If the function depends on more than one parameter, the other value are not kept fixed, but their best estimate is found, by minimizing the function in n-1 parameters.
You find more information on Minos in seal.web.cern.ch/seal/documents/ … nerror.pdf and in
paragraph 1.5.3 of seal.web.cern.ch/seal/documents/ … sguide.pdf.
The Contour method (MnContour) uses the same algorithms but for estimating the points (x1,y1) such that
f(x1,y1) = f_MINIMUM + UP.
There is no method in Minuit for estimating hypervolume contours for the case of more that 2 parameters.
The Minos error will depend then of UP, but normally, if UP is not quoted, it is assumed that the value corresponding to the 1-sigma error has been used.
In case of contour, I think, it should be always quoted the probability content (the confidence level of the contour).
I am still not entirely sure how to interpret the MINOS errors. They look to me as confidence intervals. (They are generated in exatly the same way as confidence contours)
Yes, the confidence interval for a parameter p is the interval
[p0-LOWER_MINOS_ERROR, p0 + UPPER_MINOS_ERROR]
where p0 is the best fit value.
I hope it is clear now, otherwise I would recommend to look also in a statistical book.
A very good one is the latest edition of “Statistical Methods in Experimental Physics”, by F. James
(see for example chapter 9 on Interval estimation) or “Statistical Data Analysis” by G. Cowan.
Is there a reason why I should be getting slightly different confidence intervals from MATLAB and MINUIT at the same level (68%, UP=1)?
The data are the same and the best fit parameters too.
I don’t know how Matlab estimates the confidence interval. I guess it just uses the covariance matrix from the fit, which is correct if the chi2 function (or log-likelihood) can be represented as a parabola around the minimum.
I am confused, let say we have a 4 parameter fit. But we want to obtain 68% CL for only 2 parameters. Then the value of UP should be 4.72 or 2.3?
What I mean is that, the value of UP, to obtain 68%CL is decided by total number of fitting parameters or by the number of parameters for which we wish to obtain the contour?
Just to be Sure, for a four parameter fit – if I want to obtain the contours for two parameters at 68%CL, should I set
UP=2.3, right? And this is true for even for n-parameter fit ?
