Advice to navigate through root file directories in PyROOT

[Quentin_Buat]

Dear experts,

I’m trying to navigate through a 300MB root file with a fairly deep arborescence using pyROOT:

level 0 - category (64 TDirectory):
level 1 - samples (~66 TDirectory):
level 2 - histograms (~170 TH1F)

To do so, I dump the content of the root file into a python dictionary by doing something along these lines:

# fill dict
my_dict = {}
for k in rfile.GetListOfKeys():
   cat = k.ReadObj()
   if isinstance(cat, ROOT.TDirectoryFile):
      my_dict[k.GetName()] = {}
      for kk in cat.GetListOfKeys():
         samp = kk.ReadObj()
         if isinstance(samp, ROOT.TDirectoryFile):
            my_dict[k.GetName()][kk.GetName()] = {}
            for kkk in samp.GetListOfKeys():
               hist = kkk.ReadObj()
               if isinstance(samp, ROOT.TH1):
                  my_dict[k.GetName()][kk.GetName()][kkk.GetName()] = hist
# use dict
...
#

Everything seems to be working fine except that my python run is hanging forever at the end.
I’ve tried to:

• delete the dictionary by hand (using python del or reallocating each hist to None,
• close the rootfile manually (calling rfile.Close()
• loop over the TDirectoryFile in the root file and close folders one by one

all the attempts were unsuccessful :-/. I’ve also monitored the memory consumption and indeed when I build my dictionary it seems very memory hungry. When I run on smaller root files (with a similar structure), my program ends smoothly.

To me it points to something wrong I do in the handling of the root file but I can not figure out what.

Thanks in advance for your help!

Cheers,
Quentin

[etejedor]

Hi Quentin,

It might be a problem deleting the histograms at the end of your program. Can you add either the following line before the loop:

ROOT.TH1.AddDirectory(False)

or, alternatively, right after reading every histogram:

hist.SetDirectory(0)

For reference (“Creating histograms” section):
https://root.cern.ch/doc/master/classTH1.html

[Quentin_Buat]

Hi @etejedor,

Thanks for the suggestion!
I tried the second option and it did allow my program to end.

In the meantime, I’ve also found on another thread (Pyroot hangs on TFile.Close() method):

 ROOT.gROOT.GetListOfFiles().Remove(rfile_0)

This also seems quite efficient at releasing the memory.
I am not sure I fully understand what the latter instruction does though…

Cheers,
Quentin

[etejedor]

Hi Quentin,

Good to hear it works!

It looks like that second option you mention removes the file from a list of “files to cleanup”, so that the objects they contain are not cleaned at the end.

But @pcanal will be able to give a more accurate comment, since he proposed it in the other ticket.