Your zip file only contains two shell scripts and an empty directory. Can you provide a minimal reproducer, or a snapshot of the code, with the relevant part? Did you try to use valgrind on your project to detect the memory leaks? Something like:
valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp ./yourprogram
My mistake, I forgot to add the “-r” flag when I zipped the archive together. I updated the initial post so that the code is included. Sorry about that.
I did initially try the following valgrind command:
valgrind --num-callers=30 --suppressions=$ROOTSYS/etc/valgrind-root.supp --log-file=valgrind-testMapPointing-part_54.log --leak-check=full --show-leak-kinds=all -v ./buld/testMapPointing 54
But given the version of valgrind I was precompiled, I eventually ended up with the following error at the end:
--69715:0: aspacem Valgrind: FATAL: VG_N_SEGMENTS is too low.
--69715:0: aspacem Increase it and rebuild. Exiting now.
I tried first a macro version of the function, and then a binary. But in both instances I got the output above. It is why I’m having a colleague look at it on their computer network.
But if someone here could recognize the problem before then, that would be great!
@bellenot, so I tried valgrind again. It still reports running out of memory. With the command
valgrind --num-callers=30 --suppressions=$ROOTSYS/etc/valgrind-root.supp --log-file=valgrind-testMapPointing-part_54.log --leak-check=full --show-leak-kinds=all -v ./build/testMapPointing 54
I got the attached output:
valgrind-testMapPointing-part_54.txt (32.8 KB)
Maybe i am too naive here, but have you tried to declare the function using
MakeXXX( TFile & file)
Or
MakeXXX( TFile & &)
Or inlining the function?
Also i think you can also return directly a TH2D from your functions and use the TH1::SetDirectory calls to associate a th2d to a file or a directory. (at least i usually do this when i need to shuffle and associate writing of histos to a tfile). Apologize if my comments are not going to help in your case
Why? How many files is there? Do you have TTrees? How many histograms? How many (total) number of bins?
Thank you, @RENATO_QUAGLIANI, but the TFiles are referenced in functions found under src/eventCorrelator.cc. The binary testMapPointing refers to these functions, but doesn’t have TFile input of its own.
When I tried deleting the references to the files, I also tried when initially opening them to use TH1::SetDirectory as suggested here, but that didn’t seem to work?
Hi @pcanal,
To clarify your questions, the files being referenced are the ones produced by the binaries makeAntSphericalCosineProductsFiles and makeDeltaTMapFiles. Each of these produces two files containing TH2D histograms, not in TTrees. Two files are produced because one is more finely binned. The more coarsely-binned version of the files contains histograms of dimension 180 x 100 bins, while the more finely binned version of the files contains histograms of dimension 760 x 240 bins.
In the files produced by makeDeltaTMapFiles, there are 672 TH2D histograms. In the files produced by makeAntSphereicalCosineProductFiles, there are 768 TH2D histograms.
The size of the produced files are as follows:
85M deltaTMapCoarse.root
859M deltaTMapFine.root
46M antSphericalCosineProductsCoarse.root
449M antSphericalCosineProductsFine.root
Working with one of my collaborators to run a smaller job so that valgrind doesn’t run out of memory, we were able to glean the following output:
valgrind_results.zip (146.1 KB)
In an effort to expedite resolution to the memory leak problem, I have also uploaded the project to GitHub:
Updates from November 4th were motivated by the output from the valgrind log file. However, it looks like the changes didn’t help resolve the issue.
From the valgrind file, what suggestions do people have on how to resolve the issue?
Hi @jwruss,
did you run valgrind with --suppressions=$ROOTSYS/etc/valgrind-root.supp? It seems that many of the warnings come from the interpreter, but those are harmless and should be suppressed by the above suppression file.
Also there are several ROOT warnings and errors being printed, including missing input files and unwritable output files – although they should not cause memory leaks/hogging, it is often trickier to debug problems if they are entangled with other issues.
Most importantly: you should use valgrind --tool=massif rather than the default valgrind’s tool (memcheck) to get a log of where memory is allocated.
Unfortunately without the input files we cannot try to reproduce the problem on our side. But maybe you can reproduce the problem running on the same file many times, and just share that one file?
Cheers,
Enrico
Hi Enrico,
Thank you for your response. When trying to run valgrind on my end, from the project directory I have tried the commands
valgrind --num-callers=30 --suppressions=$ROOTSYS/etc/valgrind-root.supp --log-file=valgrind-testMapPointing-part_54.log --leak-check=full --show-leak-kinds=all -v ./build/testMapPointing 54
and
valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp --log-file=valgrind-testMapPointing-part_54.log ./build/testMapPointing 54
However, using machines with CentOS on them, valgrind is pulled from a repository precompiled. Valgrind ends up exiting saying it is “out of memory”.
I have a collaborator who doesn’t appear to have this problem with valgrind trying to help me diagnose the problem from their end. It seems they have been running valgrind like in the second command
valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp --log-file=valgrind-testMapPointing-part_54.log ./build/testMapPointing 54
I have asked them to try running it like the first command to see if any more information can be gleaned. I have also just forwarded along your message. Hopefully our suggestions will help resolve this. If we figure anything out, I will post it here.
Best,
JohnHi Enrico,
Although not ideal, I have found a way to try and debug in parallel with my collaborator. I was able to download the relevant files to my work machine, but when it runs any of the above commands that is all it can run. I have tried running it this way using the command
valgrind --num-callers=30 --suppressions=$ROOTSYS/etc/valgrind-root.supp --log-file=valgrind-testMapPointing-part_54.log --leak-check=full --show-leak-kinds=all -v ./build/testMapPointing 54
but the output is not as informative as I had hoped. I tried uploading a zipped version of it here for reference, but it was still too big. The problem is that when it reports lines where problems could be occurring, the lines are too high level, not directing to lines in functions called by other functions. For example:
==3717== 2,388,940 (9,664 direct, 2,379,276 indirect) bytes in 4 blocks are definitely lost in loss record 31,618 of 31,797
==3717== at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3717== by 0x50F078D: UCorrelator::fillStrategyWithKey(FilterStrategy*, char const*) (UCFilters.cc:74)
==3717== by 0x486E613: makePeakUnnormalizedFlatInterferometricMap(int, double, double, AnitaPol::EAnitaPol, TString, int, int) (eventCorrelator.cc:1178)
==3717== by 0x10CE9B: addPolTree(int, bool) (testMapPointing.cxx:110)
==3717== by 0x10E341: main (testMapPointing.cxx:290)
==3717==
==3717== 2,652,760 (9,664 direct, 2,643,096 indirect) bytes in 4 blocks are definitely lost in loss record 31,619 of 31,797
==3717== at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3717== by 0x50F078D: UCorrelator::fillStrategyWithKey(FilterStrategy*, char const*) (UCFilters.cc:74)
==3717== by 0x486AF5C: makePeakUnnormalizedInterferometricMap(int, double, double, AnitaPol::EAnitaPol, bool, TString, int, int) (eventCorrelator.cc:743)
==3717== by 0x10D0C6: addPolTree(int, bool) (testMapPointing.cxx:133)
==3717== by 0x10E341: main (testMapPointing.cxx:290)
==3717==
I am looking to run now with the above command adding --tool=massif to it, but I don’t think that will automatically go further in depth into which lines of code the leaking can be traced back to.
From this thread it looks like I might want to use --track-origins=yes, or from this thread I would want to use --trace-children=yes? They seem like they may also be applicable to wanting to see more information, but I’m not sure. Have you had experience using these flags before?
Hi,
valgrind seems fairly convinced that an allocation happening at UCFilters.cc:74 leaks. What happens at that line?
--track-origins=yes is useful in case of accesses to uninitialized memory, which is not your case.
--tool=massif is a different valgrind tool – rather than looking for leaks or bad memory accesses, it records where all memory allocations come from, and its report tells you exactly what fraction of the memory used was allocated by what line.
If you think there should be more…ROOT in those callstacks, maybe it would help to switch to a debug build of ROOT/your program.
Cheers,
Enrico
Hi Enrico,
I will bring up UCFilters.cc:74 to my collaborator who manages the repository this source file is in. It looks to be a new declaration which may not be deleted?
To switch to a debug build of ROOT/my program, I presume I will have to update the CMakeLists.txt in this repository. Any suggestions on how to update this file to do that?
Best,
John
Apart from UCFilters.cc:74 I bet there’s memory hogging, not a memory leak. If you’re lucky we can quickly diagnose this using the following trick:
echo 'Root.ObjectStat: 1' > .rootrc
Then edit the sources of your program such that it calls gObjectTable->Print() regularly (#include "TObjectTable.h"). Build it, run it, share the output!
Cheers, Axel.
And about debug symbols: for code you build with cmake, you need to change cmake .. (or the corresponding cmake configuration command) to cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo (or Debug instead of RelWithDebInfo if you want more “precise” debugging and don’t care about performance). You can also get debug builds of ROOT e.g. from the LCG releases on lxplus.
What @Axel suggests might be a good alternative to valgrind --tool=massif.
Cheers,
Enrico
Thank you for these suggestions! I will implement them and get back to you.
Best,
John
Hi Enrico,
I’m working on implementing @Axel’s suggestion, but I wanted to check with you if the debug setting has already been set in my CMakeLists.txt. Near the top of the file, I have the line set(CMAKE_BUILD_TYPE Debug). That should mean that when I run cmake .., it should be equivalent to cmake -DCMAKE_BUILD_TYPE=Debug ..?
Best,
John
Yes,
that should be equivalent!