Memory allocation issues when opening multiple files

luminociter · May 24, 2020, 7:08pm

Dear experts, I am trying to collect my analysis results from different root files in a single plot. In order to do that I have to open the files sequentially, access some tree variables (and potentially do a quick selection) and then plot then on top of each other. My files are organized in different folders with different number of files in each folder, so i have a part of the code that does a recursive search to find all available root files in the given directory and sub directories and access them.
The issue is that I am facing quite a weird crash on lxplus after the 4th iteration on a 20 root file array, pertaining to a memory corruption:

*** Error in `/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/bin/root.exe': malloc(): memory corruption: 0x0000000005e5edc0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x82aa6)[0x7fcafca3aaa6]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7fcafca3d6fc]
/cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/lib64/libstdc++.so.6(_Znwm+0x18)[0x7fcafd5566e8]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCore.so(_ZN8TStorage11ObjectAllocEm+0x9)[0x7fcafdaa3de9]
/afs/cern.ch/work/e/egkougko/private/RecoveringData_C.so(main+0x12f2)[0x7fcaef3947e2]
[0x7fcafe144031]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so(_ZNK5cling19IncrementalExecutor14executeWrapperEN4llvm9StringRefEPNS_5ValueE+0x269)[0x7fcaf8231169]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so(_ZN5cling11Interpreter11RunFunctionEPKN5clang12FunctionDeclEPNS_5ValueE+0x93)[0x7fcaf81c8d43]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so(_ZN5cling11Interpreter16EvaluateInternalERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_18CompilationOptionsEPNS_5ValueEPPNS_11TransactionEm+0x15a)[0x7fcaf81ce7ca]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so(_ZN5cling11Interpreter7processERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS_5ValueEPPNS_11TransactionEb+0x111)[0x7fcaf81cedb1]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so(_ZN5cling13MetaProcessor7processEN4llvm9StringRefERNS_11Interpreter17CompilationResultEPNS_5ValueEb+0x1bd)[0x7fcaf828540d]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so(+0x3e28bc)[0x7fcaf81538bc]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCling.so(+0x3f7936)[0x7fcaf8168936]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCore.so(_ZN12TApplication11ProcessLineEPKcbPi+0xb6c)[0x7fcafda5574c]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libRint.so(_ZN5TRint13ProcessLineNrEPKcS1_Pi+0xc2)[0x7fcafdf09dc2]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libRint.so(_ZN5TRint15HandleTermInputEv+0x233)[0x7fcafdf0a133]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCore.so(_ZN11TUnixSystem16CheckDescriptorsEv+0x8b)[0x7fcafdb8b09b]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCore.so(_ZN11TUnixSystem16DispatchOneEventEb+0x558)[0x7fcafdb8ca58]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCore.so(_ZN7TSystem3RunEv+0xe1)[0x7fcafdab7eb1]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libCore.so(_ZN12TApplication3RunEb+0x1f)[0x7fcafda528ff]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libRint.so(_ZN5TRint3RunEb+0x456)[0x7fcafdf0b626]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/bin/root.exe[0x400a1a]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fcafc9da555]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/bin/root.exe[0x400a5f]
======= Memory map: ========
00400000-00401000 r-xp 00000000 00:38 633579750                          /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/bin/root.exe
00600000-00601000 r--p 00000000 00:38 633579750                          /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/bin/root.exe
00601000-00602000 rw-p 00001000 00:38 633579750                          /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/bin/root.exe
0252f000-06073000 rw-p 00000000 00:00 0                                  [heap]
7fcae0000000-7fcae0021000 rw-p 00000000 00:00 0
7fcae0021000-7fcae4000000 ---p 00000000 00:00 0
7fcae778e000-7fcae7889000 r--p 00000000 00:38 638548017                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/fonts/FreeSans.otf
7fcae7889000-7fcae78e8000 r-xp 00000000 00:38 633577302                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTGpadv7.so
7fcae78e8000-7fcae7ae8000 ---p 0005f000 00:38 633577302                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTGpadv7.so
7fcae7ae8000-7fcae7aea000 r--p 0005f000 00:38 633577302                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTGpadv7.so
7fcae7aea000-7fcae7aeb000 rw-p 00061000 00:38 633577302                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTGpadv7.so
7fcae7aeb000-7fcae7aef000 rw-p 00000000 00:00 0
7fcae7aef000-7fcae7b01000 r-xp 00000000 00:38 633577267                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTGraphicsPrimitives.so
7fcae7b01000-7fcae7d01000 ---p 00012000 00:38 633577267                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTGraphicsPrimitives.so
7fcae7d01000-7fcae7d02000 r--p 00012000 00:38 633577267                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTGraphicsPrimitives.so
7fcae7d02000-7fcae7d03000 rw-p 00013000 00:38 633577267                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTGraphicsPrimitives.so
7fcae7d03000-7fcae7d04000 rw-p 00000000 00:00 0
7fcae7d04000-7fcae7d17000 r-xp 00000000 00:38 633576983                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTHistDraw.so
7fcae7d17000-7fcae7f17000 ---p 00013000 00:38 633576983                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTHistDraw.so
7fcae7f17000-7fcae7f18000 r--p 00013000 00:38 633576983                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTHistDraw.so
7fcae7f18000-7fcae7f19000 rw-p 00014000 00:38 633576983                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.18.04-c767d/x86_64-centos7-gcc8-opt/lib/libROOTHistDraw.so%

Does anyone have an ideRecoveringData.C (10.3 KB) a of what this could be? I am also uploading the macro that I wrote which i run afrter loading with .L .C+

Thanks in advance, every hint would be helpful!!

ROOT Version: 6.18/04
Platform: lxplus
Compiler: AClick

amadio · May 25, 2020, 12:02pm

I cannot run your macro because it accesses files in your home in EOS, and I have no permission. What I suggest is for you to just take a list of files as input with argc,argv, compile your code rather than run it as a macro, and use the shell to pass in the list of files with something like */*.root. You can also try to run your macro with +g to get better trace of what the error is (i.e. with debugging info enabled). Just looking at the code, it’s hard to tell what’s wrong, but the part where you modify dirs and increase hep does make it look like you could run over the end of the vector and read invalid memory.

luminociter · May 26, 2020, 10:42am

Thank you for your answers. I do not believe the issue is there since it seems to be able to go through the recursive file definition with no issue. Besides, the same function integrated in an analysis framework i created works as expected. The macro crashes after having already gone through some of the files and having done the requested rebining or plotting for at least two or three of therm.

At this point I believe that either the pointer vectors I am using for the TH1D , TCanvas and TAxis* and TFile* objects are not correctly initialized. I would expect the resize() command to take case of that but perhaps I have to explicitly call the new operator?

Another more probable answer would be the point of attachment of the created histograms, which would be one level up with respect to the accessed file, so the interpreter would not be able to locate the pointers at the current working directory.

Perhaps the most correct approach would be to open and close the files instead of keeping them open throughout the process?

pcanal · May 26, 2020, 3:02pm

To improve the diagnosis, you can compile the macro in debug mode with .L RecoveringData.C+g (or ++g if the .so is still around).

Then you can even run with valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp .... .

luminociter · May 26, 2020, 4:43pm

Thank you both, I was able to finally track it down to a typo in an index of an array that refers to another array… If i was running in compiled mode I guess the compiler would have complained tat the variable i am using as array index is out of scope. I assume that in interpreter mode, even if I define an index within a loop, it remains valid outside of it…