GSLMinimizer only varies one parameter during fit

mcgowan · November 12, 2010, 7:24pm

Hi,

I’m using SteepestDescent method on a 7 parameter function. Clipped code below:

ROOT::Math::Minimizer* min = ROOT::Math::CreateMinimizer(“GSLMultiMin”, “SteepestDescent”);
ROOT::Math::Functor f(this,&TF::total3viewEnergy,7);
min->SetLimitedVariable(0,“energyPerPlane”,variable[0], step[0], m_minTrackEnergyPerPlane, m_maxTrackEnergyPerPlane );
min->SetLimitedVariable(1,“vtx_x”, variable[1], step[1], m_boxXmin, m_boxXmax );
min->SetLimitedVariable(2,“vtx_y”, variable[2], step[2], m_boxYmin, m_boxYmax );
min->SetLimitedVariable(3,“vtx_z”, variable[3], step[3], zUpstream-m_zAddBox, zDownstream+m_zAddBox );
min->SetLimitedVariable(4,“end_x”, variable[4], step[4], m_boxXmin, m_boxXmax );
min->SetLimitedVariable(5,“end_y”, variable[5], step[5], m_boxYmin, m_boxYmax );
min->SetLimitedVariable(6,“end_z”, variable[6], step[6], zUpstream-m_zAddBox, zDownstream+m_zAddBox );

double TF::total3viewEnergy( const double *par )

When I monitor the log file, I find that GSL is only changing one parameter (the [0] out of the vector of 7). Out of curiosity, I changed the assignment of par[0] to a different variable (vtx_x) and it is now the only one being varied:

----------> Iteration 0
FVAL = 1561675.36144889938
X Values : energyPerPlane = 2 vtx_x = 1 vtx_y = 0 vtx_z = 7619.61 end_x = 0 end_y = 0 end_z = 9088.08
…
----------> Iteration 4
FVAL = 1561380.44705637125
X Values : energyPerPlane = 2 vtx_x = 0.966 vtx_y = 0 vtx_z = 7619.61 end_x = 0 end_y = 0 end_z = 9088.08
…

I also see the same behavior with ConjugateFR in GSL.

I originally tried to use Minuit2 (so that I could use my parameter boundaries) but it crashes with the following:
minuit2/src/MnUserTransformation.cxx:381: unsigned int ROOT::Minuit2::MnUserTransformation::Index(const char*) const: Assertion `ipar != fParameters.end()’ failed.

I understand that I can utilize the Minuit2::SinParameterTransformation class to implement my own parameter boundaries with GSL, but first I would simply like to know why it will only vary one of my parameters! Is there some setting I don’t know about that is causing GSL to treat my problem as 1-dimensional?

Thanks!

moneta · November 13, 2010, 9:16am

Hi,

can you please attach your full script and tell me which ROOT version are you using ?
Thank you
Lorenzo

mcgowan · November 14, 2010, 2:03pm

Hi Lorenzo,

This is part of a Gaudi Tool + Algorithm running inside of an LHCb-based framework, and takes a ~150M root file as input. I am running ROOT v5.18.00f, compiled under slc4_ia32_gcc34. I don’t know a straightforward way to let you duplicate the Minuit2 runtime error (MnUserTransformation.cxx:381: unsigned int ROOT::Minuit2::MnUserTransformation::Index(const char*) const: Assertion `ipar != fParameters.end()’ failed.), and I don’t think I can pare this down to a simple ROOT script. I will attach my code (even though it won’t compile outside of my environment) and perhaps you might see what is causing this behavior.

I worry that I’ve done something wrong with handling the fit parameters, since the Minuit2 crash complains about a parameter iterator, and GSL only varies one of my parameters. The function being minimized is double Tf::total3viewEnergy( const double *par ) in Tf.cpp. The minimization occurs in the function StatusCode Tf::findTrack( Track &uniqueTrack, double boxZrange[2] ) in Tf.cpp. This should be all the code that handles the fit parameters.

Thanks!
GSLproblem.tar (40 KB)

moneta · November 14, 2010, 3:19pm

Hi,

you are using a very old version of ROOT (5.18) . This version is of 3 years ago. Setting limited parameters was not supported for GSL at that time.
However Minuit2 should have been working. Could you check that min > max values in SetLimitedParameters ?
I would however recommend you to upgrade ROOT version, at least to 5.26, which is the official version supported by LHCb.

Best Regards

Lorenzo

mcgowan · November 14, 2010, 8:52pm

Hi Lorenzo,
I’ll try to run with 5.26 installed. Do you mean that GSL respects parameter boundaries in ROOT 5.26?
I will also look into the source code MnUserTransformation.cxx to try to see what this Minuit2 error means. I’ll let you know if either GSL or Minuit2 behaves better in 5.26.
Thanks!

moneta · November 14, 2010, 10:14pm

Yes, since ROOT 5.24, you can use limited parameters with the GSL minimizer.
It is also possible that in ROOT 5.26 the problem you observe in Minuit2 is fixed. If this is not the case, please let me know and we can investigate it. In this case a back trace obtained with gdb would be useful,

Best Regards

Lorenzo

mcgowan · November 15, 2010, 8:47pm

Hi Lorenzo,

There are some complications to safely upgrading our Gaudi/ROOT installations without disturbing the rest of our framework. I assume it will happen at some point in the future, but for now I am doing what I can with ROOT 5.18.

I have added the following print statements that I think help to understand what is going on with the variable assignment:
bool a = min->SetLimitedVariable(0,“energyPerPlane”,variable[0], step[0], lower[0], upper[0] );
bool b = min->SetLimitedVariable(1,“vtx_x”, variable[1], step[1], lower[1], upper[1] );
bool c = min->SetLimitedVariable(2,“vtx_y”, variable[2], step[2], lower[2], upper[2] );
bool d = min->SetLimitedVariable(3,“vtx_z”, variable[3], step[3], lower[3], upper[3] );
bool e = min->SetLimitedVariable(4,“end_x”, variable[4], step[4], lower[4], upper[4] );
bool ff = min->SetLimitedVariable(5,“end_y”, variable[5], step[5], lower[5], upper[5] );
bool g = min->SetLimitedVariable(6,“end_z”, variable[6], step[6], lower[6], upper[6] );
if (a) info() << "set energy per plane: " << a << endmsg;
if (b) info() << "set vtx_x: " << b << endmsg;
if © info() << "set vtx_y: " << c << endmsg;
if (d) info() << "set vtx_z: " << d << endmsg;
if (e) info() << "set end_x: " << e << endmsg;
if (ff) info() << "set end_y: " << ff << endmsg;
if (g) info() << "set end_z: " << g << endmsg;

When I select any GSL option, the log demonstrates that SetLimitedVariable does indeed give a return value:
ToolSvc.TrackFinder INFO Using minimization scheme GSLMultiMin, ConjugateFR
ToolSvc.TrackFinder INFO Allowing for 1000 iterations
ToolSvc.TrackFinder INFO set energy per plane: 1
ToolSvc.TrackFinder INFO set vtx_x: 1
ToolSvc.TrackFinder INFO set vtx_y: 1
ToolSvc.TrackFinder INFO set vtx_z: 1
ToolSvc.TrackFinder INFO set end_x: 1
ToolSvc.TrackFinder INFO set end_y: 1
ToolSvc.TrackFinder INFO set end_z: 1

However, when I select any Minuit2 option, the log shows that SetLimitedVariable is failing on the first call:
ToolSvc.TrackFinder INFO Using minimization scheme Minuit2, Migrad
ToolSvc.TrackFinder INFO Allowing for 1000 iterations
SystemTestsApp.exe: minuit2/src/MnUserTransformation.cxx:381: unsigned int ROOT::Minuit2::MnUserTransformation::Index(const char*) const: Assertion `ipar != fParameters.end()’ failed.

So, Minuit2 is failing to set any variables for minimization, which is why the parameter counter in MnUserTransformation.cxx initializes at the end of fParameters:

unsigned int MnUserTransformation::Index(const std::string & name) const {
// get index (external) corresponding to name
std::vector::const_iterator ipar =
std::find_if(fParameters.begin(), fParameters.end(), MnParStr(name));
assert(ipar != fParameters.end());
// return (ipar - fParameters.begin());
return (*ipar).Number();
}

I do not understand why GSL successfully calls SetLimitedVariable and Minuit2 does not.

The odd behavior in GSL persists: only the first parameter is varied:
----------> Iteration 0
FVAL = 1485317.06681756792
X Values : energyPerPlane = 1.9 vtx_x = 0 vtx_y = 0 vtx_z = 7619.61 end_x = 0 end_y = 0 end_z = 9088.08
----------> Iteration 1
FVAL = 1442306.18008259218
X Values : energyPerPlane = 1.7 vtx_x = 0 vtx_y = 0 vtx_z = 7619.61 end_x = 0 end_y = 0 end_z = 9088.08

moneta · November 16, 2010, 8:21am

Hi,

when you use a GSL minimizer in 5.18 a limited variable is considered un-limited. FOr this the function returns true. All the other variables should be varied. Have you tried with more iterations ?
I don’t understand your assert in Minuit2. Can you send me the back trace obtained from gdb ?

Lorenzo

mcgowan · November 16, 2010, 3:27pm

Hi Lorenzo,

Here is the gdb output at the point where Minuit2 fails (the call to SetLimitedVariable), along with the backtrace. This behavior is identical with all five minimization schemes within Minuit2:

SystemTestsApp.exe: minuit2/src/MnUserTransformation.cxx:381: unsigned int ROOT::Minuit2::MnUserTransformation::Index(const char*) const: Assertion `ipar != fParameters.end()’ failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 4154526240 (LWP 9914)]
0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x0064c825 in raise () from /lib/tls/libc.so.6
#2 0x0064e289 in abort () from /lib/tls/libc.so.6
#3 0x00645da1 in __assert_fail () from /lib/tls/libc.so.6
#4 0xec17a858 in ROOT::Minuit2::MnUserTransformation::Index () from /grid/fermiapp/minerva/software_releases/v8r0_DEV/lcg/external/root/5.18.00f/slc4_ia32_gcc34/root/lib/libMinuit2.so
#5 0xec175906 in ROOT::Minuit2::MnUserParameters::Index () from /grid/fermiapp/minerva/software_releases/v8r0_DEV/lcg/external/root/5.18.00f/slc4_ia32_gcc34/root/lib/libMinuit2.so
#6 0xec175cf6 in ROOT::Minuit2::MnUserParameterState::Index () from /grid/fermiapp/minerva/software_releases/v8r0_DEV/lcg/external/root/5.18.00f/slc4_ia32_gcc34/root/lib/libMinuit2.so
#7 0xec1532a9 in ROOT::Minuit2::Minuit2Minimizer::SetVariable () from /grid/fermiapp/minerva/software_releases/v8r0_DEV/lcg/external/root/5.18.00f/slc4_ia32_gcc34/root/lib/libMinuit2.so
#8 0xec1531d1 in ROOT::Minuit2::Minuit2Minimizer::SetLimitedVariable () from /grid/fermiapp/minerva/software_releases/v8r0_DEV/lcg/external/root/5.18.00f/slc4_ia32_gcc34/root/lib/libMinuit2.so
#9 0xf2e66585 in TrackFinder::findTrack () from /minerva/app/users/mcgo0109/cmtuser/Minerva_v8r0_DEV/InstallArea/slc4_ia32_gcc34/lib/libElasticTrackerLib.so

The behavior is also similar if I use SetVariable() instead of SetLimitedVariable():

SystemTestsApp.exe: minuit2/src/MnUserTransformation.cxx:381: unsigned int ROOT::Minuit2::MnUserTransformation::Index(const char*) const: Assertion `ipar != fParameters.end()’ failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 4154526240 (LWP 17799)]
0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x0064c825 in raise () from /lib/tls/libc.so.6
#2 0x0064e289 in abort () from /lib/tls/libc.so.6
#3 0x00645da1 in __assert_fail () from /lib/tls/libc.so.6
#4 0xec17a858 in ROOT::Minuit2::MnUserTransformation::Index () from /grid/fermiapp/minerva/software_releases/v8r0_DEV/lcg/external/root/5.18.00f/slc4_ia32_gcc34/root/lib/libMinuit2.so
#5 0xec175906 in ROOT::Minuit2::MnUserParameters::Index () from /grid/fermiapp/minerva/software_releases/v8r0_DEV/lcg/external/root/5.18.00f/slc4_ia32_gcc34/root/lib/libMinuit2.so
#6 0xec175cf6 in ROOT::Minuit2::MnUserParameterState::Index () from /grid/fermiapp/minerva/software_releases/v8r0_DEV/lcg/external/root/5.18.00f/slc4_ia32_gcc34/root/lib/libMinuit2.so
#7 0xec1532a9 in ROOT::Minuit2::Minuit2Minimizer::SetVariable () from /grid/fermiapp/minerva/software_releases/v8r0_DEV/lcg/external/root/5.18.00f/slc4_ia32_gcc34/root/lib/libMinuit2.so
#8 0xf2e66571 in TrackFinder::findTrack () from /minerva/app/users/mcgo0109/cmtuser/Minerva_v8r0_DEV/InstallArea/slc4_ia32_gcc34/lib/libElasticTrackerLib.so

GSL does not complain about setting the variables, and goes forward to start minimization, which is why this is so perplexing. Eventually, GSL fails with status 27 because it is only changing one variable and cannot move towards the minimum.

mcgowan · November 16, 2010, 10:22pm

Interestingly enough, GSLSimAn does manage to vary all of my parameters, but because it will not respect parameter limits, it decides wildly huge values and sends the minimization far away from the global minimum. I may try to implement the sine transformation that Minuit uses in combination with GSLSimAn, since that is the only combination that I can imagine working in 5.18 with my code to limit my fit parameters and also get the Minimizer to vary all parameters.

moneta · November 17, 2010, 10:38am

Hi,

The crash happens if you are calling more than one time SetVariable (or SetLimitedVariable) for the same variable (parameter). Can you check you are not doing this ?

For implementing the limited variable in GSL have a look at the new classes in ROOT (from 5.24):

root.cern.ch/viewvc/trunk/math/m … iew=markup

and

root.cern.ch/viewvc/trunk/math/m … iew=markup

Lorenzo

mcgowan · November 17, 2010, 3:08pm

Hi Lorenzo,

Thanks for the response. I rearranged the print statements to demonstrate that the crash is not happening due to a second call of SetVariable for the same variable:

  info() << "arrived at variable setting block " << endmsg;
  bool a = min->SetLimitedVariable(0,"energyPerPlane",variable[0], step[0], lower[0], upper[0] );
  info() << "set energy per plane: " << a << endmsg;
  bool b = min->SetLimitedVariable(1,"vtx_x", variable[1], step[1], lower[1], upper[1] );
  info() << "set vtx_x: " << b << endmsg;
  ...

As you can see from the log, the crash occurs during the first call to SetLimitedVariable (no print statement for “set energy per plane”, the first variable, appears):

ToolSvc.TrackFinder            INFO arrived at variable setting block
SystemTestsApp.exe: minuit2/src/MnUserTransformation.cxx:381: unsigned int ROOT::Minuit2::MnUserTransformation::Index(const char*) const: Assertion `ipar != fParameters.end()' failed.

Thanks for the references for implementing limited variables in GSL.

I remain curious about the problems with Minuit2 assigning variables for multi-dimensional minimization in ROOT 5.18. Do any working examples exist that I could study?

moneta · November 17, 2010, 3:25pm

Hi,

yes I have examples working with 5.18. See for example

root.cern.ch/viewvc/trunk/math/m … iew=markup

Lorenzo

moneta · November 18, 2010, 12:52pm

Hi,

looking back at the old code of 5.18, I have understood what is causing your assert in Minuit2. The
length of the parameter name should be <= 10 characters. In 5.18 longer names were not supported, while they are supported with all the newer versions.

Best Regards

Lorenzo

mcgowan · November 18, 2010, 6:09pm

Hi Lorenzo,

Thank you for specifying the restriction on the length of the variable name (my first name was 14 characters). Minuit2 now runs properly and I will likely adopt this for use in my application.

Thanks again!