How to find suspect memory leak

Hello,
I am using pyROOT within the CMS framework, and it does a great job :slight_smile:
However, when I read a series of trees in different files (~30 files, 20k events each),
I see the the memory usage increasing continuously. This goes on until
at some point the interpreter crashes with error messages like

or errors about reading the root file. I suspect this might be a memory
leak, but I do not know how to investigate it; I tried simply to monitor
the garbage collector:

and I see that this is increasing as well.
The code looks like:

from ROOT import *
from cmstools import *
import os

gSystem.Load("libFWCoreFWLite.so")
AutoLibraryLoader.enable()
for phi in phi_s[:-1] :
    for eta  in eta_s :
             <...>
             fileSim = TFile.Open(fname)           
            #get the tree
            simEvents = fileSim.Get("Events")
            nSimEvents = simEvents.GetEntries()


            # read the branches
            hcalSimHitBranch = simEvents.GetBranch("PCaloHits_g4SimHits_HcalHits_T.obj")
            hcalSimHit = std.vector(PCaloHit)()
            hcalSimHitBranch.SetAddress(hcalSimHit)
            for ev in range(nEvents): 
                hcalSimHitBranch.GetEntry(ev)
                for simHit in all(hcalSimHit):
                   <do>
            fileSim.Close()

The code works perfectly fine when I run on few (~1k) events per file, but
gives problems when running on more events (this seems to be a clear hint
to an eventual memory leak…).
Please, could you give me any suggestion about how I could debug this? Did anyone experience anything like this when reading a large number of events with pyROOT ?
Thank you very much in advance,

davide

Davide,

the easiest way of finding memory leaks is through the use of valgrind (http://www.valgrind.org), but this is an easy one. This:hcalSimHit = std.vector(PCaloHit)()causes the creation of a new vector on the heap, and this:hcalSimHitBranch.SetAddress(hcalSimHit) sets it to be maintained by the C++ side, to prevent Python ref-counting from removing it from underneath the application.

To solve, you can either set the memory model used by PyROOT to “strict” (see: http://cern.ch/wlav/pyroot/memory.html), or simple call clear() on the vector rather than creating new ones and SetAddress()-ing them (is probably faster, too).

Note that, depending on the ROOT release that you are using, you don’t need SetAddress, but can just access hcalSimHit in your tree directly, as if it were an attribute of the tree.

Cheers,
Wim

Hello Wim,
thank you very much for your quick reply. Unfortunately I wasn’t able to get any of your suggestions to work (BTW, I am using root 5.12/00e, Python 2.4.2).
If I try to access directly the branch:

            fileSim = TFile.Open(fname)           
            #get the tree
            simEvents = fileSim.Get("Events")
            hcalEnergy = 0.
            #loop on simHits
            for ev in range(simEvents.GetEntries()):
                simEvents.GetEntry(ev)
                for simHit in all(simEvents.PCaloHits_g4SimHits_HcalHits_T.obj):
                    hcalEnergy = hcalEnergy + simHit.energy()

I get the following error:

If I try to create the vector outside the files-loop, clear() it and SetAddress()-ing it after reading each file, I get the following:

Warning in <TClass>: no dictionary for class __gnu_cxx::__normal_iterator<const PCaloHit*,vector<PCaloHit> > is available Traceback (most recent call last): File "test.py", line 47, in ? hcalEnergy = hcalEnergy + simHit.energy() ValueError: NULL result where temporary expected

If I try to set the deletion policy to strict I get

Traceback (most recent call last): File "test.py", line 12, in ? ROOT.SetMemoryPolicy( ROOT.kMemoryStrict ) AttributeError: type object 'ROOT' has no attribute 'SetMemoryPolicy'

If I try to delete manually the vector (as suggested in http://wlav.web.cern.ch/wlav/pyroot/memory.html ), I get the following:

File "test.py", line 99, in ? hcalSimHit.IsA().Destructor( hcalSimHit ) AttributeError: 'vector<PCaloHit>' object has no attribute 'IsA'

I also tried creating the vector and then use “del”, but without any success.
I woudl be very grateful if you have any suggestion, meanwhile I will try with valgrind.
Thank you very much,

davide

Davide,

yes, 5.12 has issues with access to TTree branches as attributes, but in your case the branch name is “PCaloHits_g4SimHits_HcalHits_T.obj”, no? If so, the proper call would be getattr( simEvents, "PCaloHits_g4SimHits_HcalHits_T.obj" ) to prevent the translation of the ‘.’ into object member access.

As for: ValueError: NULL result where temporary expected again a 5.12 limitation … w/o a dictionary for the STL iterator that you want to use, it will not iterate into the vector (5.13 has emulation code, even as it would still be better to not rely on that, but just provide the dictionary). Easiest way around: v = getattr( simEvents, "PCaloHits_g4SimHits_HcalHits_T.obj" ) for i in xrange(v.size()): simHit = v[i] which also gives better performance.

Then this: ROOT.SetMemoryPolicy( ROOT.kMemoryStrict ) The problem here is that you access “ROOT” the C++ namespace, not “ROOT” the python module. Therefore, either SetMemoryPolicy( kMemoryStrict ) after your “from ROOT import *” or do an “import ROOT” first, so that you can access the python module.

Finally, this: hcalSimHit.IsA().Destructor( hcalSimHit ) could be made to work by using gROOT.GetClass( hcalSimHit.__class__.__name__ ).Destructor( hcalSimHit ) The IsA() method is a TObject thing, and simply returns the appropriate TClass. I could probably add an IsA() to all pythonized classes, just for consistency, but that won’t help you here and now. :slight_smile:

Cheers,
Wim