ROOT (or at least Minuit2) runs extremely slow when `soversion` is disabled

septi · May 17, 2023, 9:12am

We observed an order of magnitude difference in the computing speed (wall time) for one of our processing pipelines when using the exact same ROOT version (6.22.06). It turned out that the only difference is the shared library option. If this option is enabled, the pipeline is roughly 10 times faster compared to the ROOT installation without soversion.

Is there any reason why e.g. Minuit2 (which we use in our pipeline) would run that much slower without soversion?

We tried to track it down and reduce it to a MWE and what we have now is minuit2FitBench2D.C (taken from the ROOT fit tutorials (ROOT: tutorials/fit/minuit2FitBench2D.C File Reference).

If executed with the soversion enabled, this is the output (look at the RT and Cpu times)

   ------------------------------------------------------------------
  | Welcome to ROOT 6.22/06                        https://root.cern |
  | (c) 1995-2020, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on May 16 2023, 16:18:00                 |
  | From tags/v6-22-06@v6-22-06                                      |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

root [0] .x minuit2FitBench2D.C
 FCN=7196.63 FROM MIGRAD    STATUS=CONVERGED     169 CALLS         170 TOTAL
                     EDM=3.06507e-09    STRATEGY= 1  ERROR MATRIX UNCERTAINTY   2.0 per cent
  EXT PARAMETER                                   STEP         FIRST
  NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE
   1  p0           5.28670e+01   2.67227e-01   1.71567e-03   1.41591e-04
   2  p1           2.00562e+00   9.99521e-03  -6.72962e-05  -5.71944e-03
   3  p2          -1.02279e+00   1.35440e-02   3.22465e-05   2.77025e-03
   4  p3           2.89928e+00   7.94099e-03  -9.38703e-05  -4.55754e-03
   5  p4           3.86656e+00   1.13330e-02   1.06568e-05  -2.11128e-03
Minuit, npass=0  : RT=  0.306 s, Cpu=  0.230 s

****************************************
Minimizer is Fumili
Chi2                      =      7196.63
NDf                       =         7366
NCalls                    =           30
p0                        =      52.8672   +/-   0.262932
p1                        =      2.00562   +/-   0.00992527
p2                        =     -1.02279   +/-   0.0135821
p3                        =      2.89927   +/-   0.00781808
p4                        =      3.86655   +/-   0.011235
Fumili, npass=0  : RT=  0.234 s, Cpu=  0.230 s

****************************************
Minimizer is Minuit2
Chi2                      =      7196.63
NDf                       =         7366
Edm                       =  4.35808e-08
NCalls                    =          176
p0                        =       52.867   +/-   0.264728
p1                        =      2.00562   +/-   0.00995152
p2                        =      -1.0228   +/-   0.0137398
p3                        =      2.89928   +/-   0.00798634
p4                        =      3.86656   +/-   0.0112306
Minuit2, npass=0  : RT=  0.147 s, Cpu=  0.140 s

****************************************
Minimizer is Minuit2 / Fumili
Chi2                      =      7196.63
NDf                       =         7366
Edm                       =  2.49343e-07
NCalls                    =           99
p0                        =      52.8669   +/-   0.265504
p1                        =      2.00562   +/-   0.00992463
p2                        =      -1.0228   +/-   0.0135834
p3                        =      2.89928   +/-   0.00792724
p4                        =      3.86656   +/-   0.0113104
Fumili2, npass=0  : RT=  0.143 s, Cpu=  0.140 s

When running without the soversion, we see a factor 4+ difference:

root [0] .x minuit2FitBench2D.C
Warning in <TCanvas::Constructor>: Deleting canvas with same name: c1
 FCN=7196.63 FROM MIGRAD    STATUS=CONVERGED     169 CALLS         170 TOTAL
                     EDM=3.06507e-09    STRATEGY= 1  ERROR MATRIX UNCERTAINTY   2.0 per cent
  EXT PARAMETER                                   STEP         FIRST
  NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE
   1  p0           5.28670e+01   2.67227e-01   1.71567e-03   1.41591e-04
   2  p1           2.00562e+00   9.99521e-03  -6.72962e-05  -5.71944e-03
   3  p2          -1.02279e+00   1.35440e-02   3.22465e-05   2.77025e-03
   4  p3           2.89928e+00   7.94099e-03  -9.38703e-05  -4.55754e-03
   5  p4           3.86656e+00   1.13330e-02   1.06568e-05  -2.11128e-03
Minuit, npass=0  : RT=  0.991 s, Cpu=  0.990 s

****************************************
Minimizer is Fumili
Chi2                      =      7196.63
NDf                       =         7366
NCalls                    =           30
p0                        =      52.8672   +/-   0.262932
p1                        =      2.00562   +/-   0.00992527
p2                        =     -1.02279   +/-   0.0135821
p3                        =      2.89927   +/-   0.00781808
p4                        =      3.86655   +/-   0.011235
Fumili, npass=0  : RT=  0.447 s, Cpu=  0.440 s

****************************************
Minimizer is Minuit2
Chi2                      =      7196.63
NDf                       =         7366
Edm                       =  4.35808e-08
NCalls                    =          176
p0                        =       52.867   +/-   0.264728
p1                        =      2.00562   +/-   0.00995152
p2                        =      -1.0228   +/-   0.0137398
p3                        =      2.89928   +/-   0.00798634
p4                        =      3.86656   +/-   0.0112306
Minuit2, npass=0  : RT=  0.810 s, Cpu=  0.810 s

****************************************
Minimizer is Minuit2 / Fumili
Chi2                      =      7196.63
NDf                       =         7366
Edm                       =  2.49343e-07
NCalls                    =           99
p0                        =      52.8669   +/-   0.265504
p1                        =      2.00562   +/-   0.00992463
p2                        =      -1.0228   +/-   0.0135834
p3                        =      2.89928   +/-   0.00792724
p4                        =      3.86656   +/-   0.0113104
Fumili2, npass=0  : RT=  0.663 s, Cpu=  0.670 s

ROOT Version: 6.22.06
Platform: Linux
Compiler: GCC

vpadulan · May 17, 2023, 12:46pm

Hi @septi ,

I am not sure what could be the issue, maybe @moneta or @axel might have a clue? Can you perhaps also try running the same with a more recent ROOT version, ideally the latest 6.28/04?

Cheers,
Vincenzo

pcanal · May 17, 2023, 1:38pm

Check that the debug/optimize mode did not change inadvertently between the 2 builds:

grep CMAKE_BUILD_TYPE CMakeCache.txt

septi · May 17, 2023, 1:55pm

The slower one (without soversion) has no entry there:

$ grep CMAKE_BUILD_TYPE CMakeCache.txt
CMAKE_BUILD_TYPE:STRING=

I don’t have the CMakeCache.txt for the new one, unfortunately.

@vpadulan I did not try with more recent versions. We also have to stick to 6.22.06 with our data processing.

pcanal · May 17, 2023, 2:17pm

I recommend you retry the experiment for clarity:

Create 2 distinct build directory
Configure and build them in 2 distinct shell
Run the test a couple of time in both shell to verify the different is due to the build and not to something else that happened to happen at the same time on the machine.

then you/we can investigate further if there is other issue/differences.

septi · May 18, 2023, 7:47am

I did, two parallel installations on the same machine, with the only difference of soversion on and off.

Wile_E_Coyote · May 18, 2023, 7:49am

Attach both CMakeCache.txt files for inspection.
Also, for both distributions, execute: ldd $(which root.exe) $(which hist2workspace)

system · June 1, 2023, 7:49am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.