Memory leak using ROOT.std.map('<Int_t,Float_t>')()

[Tom_Ruf]

I am seeing a memory leak using the following simple code:

import ROOT
allChannels = ROOT.std.map(’<Int_t,Float_t>’)()
allChannels[3]=5.
N=0
for n in range(10000000):
for c in allChannels:
N+=1

ROOT version 6.25/01
Ubuntu 20.04

This is at the moment prohibiting me to run over large statistics. Within seconds, it consumes GB of memory. Is this an user error, or an issue with pyroot?

[etejedor]

Hello,
Thanks for the report, I do see a memory increase as well while the program runs but it’s not as fast as what you see. Looks related to the iteration on the map, I’ll investigate!

[Tom_Ruf]

I see more than one Gbyte in about 60seconds.

[etejedor]

In the meantime, as a workaround to unblock you, you could try to do the looping in C++. You can encapsulate the code in a function that accepts the map and loops over it.

ROOT.gInterpreter.Declare("""
void your_function(std::map<int,float>& m)
{
   ...
}
""")

ROOT.your_function(allChannels)

[etejedor]

I created this issue about it:

[PyROOT] Memory leak when iterating over std::map from Python

[Tom_Ruf]

I am afraid that putting the python logic of the loop into a C++ function is not really a solution and straightforward. I would need to copy all the parameters used in the loop to the wrapper function. And I would have to do it differently for all the different loops.

Is there any other possibility to convert std::map<int,float> to something else in the wrapper function which could be returned to the python world without memory explosion?

allChannels = aHit.GetAllSignals() # where GetAllSignals() returns std::map<int,float>

for c in allChannels:

channel = barnSiPMsnSides + c[0]

rc = h[‘hit_’+str(s)+str(l)].Fill( int(channel))

rc = h[‘bar_’+str(s)+str(l)].Fill(bar)

if s==2 and smallSiPMchannel(c[0]) : rc = h[‘sigS_’+str(s)+str(l)].Fill(c[1])

elif c[0]<nSiPMs: rc = h[‘sigL_’+str(s)+str(l)].Fill(c[1])

else : rc = h[‘sigR_’+str(s)+str(l)].Fill(c[1])

rc = h[‘sig_’+str(s)+str(l)].Fill(c[1])

[truf]

OK, found a solution. Splitting map into two std::vector which are returned to the python world.

[truf]

Found a new memory leak with cppyy.gbl.std.vector<genfit::TrackPoint*> object

points = aTrack.getPoints()
for n in range(1000000):
    for p in points:
             n+=1

unfortunately this can only be reproduced within our software environment.

[etejedor]

If I create an iterator for an std::vector of e.g. ints in a hot loop I don’t see the same leak as for std::map (I do see it for an std::list too, for example). So it must be related to the way the iterators are implemented for those std containers (vector has its own implementation). I’ll report when I have news.

[etejedor]

Hello,

This PR fixes the leak, thank you again for reporting!

[PyROOT] Fix memory leak in iterators

[etejedor]

This will be included in the forthcoming 6.26 release.