PyROOT: Walking through a ROOT-file gets slower the longer you "walk"

ROOT
,
6

Hi @etejedor,

inspired from your answer in PyROOT loop over files, huge memory leak I think I found the solution to my problem: using ROOT.SetOwnership(obj, True).

It was one the first things that I tried originally, but only on the TEfficiency objets itself. As it turns out, doing that for the TDirectory objects as well that I am “walking” through is the crucial bit.

Now the code looks like this:

#!/usr/bin/env python
import ROOT
import time
import os.path



_print_interval = 1000
_last_time = time.time()
_obj_cnt = 0
def print_time_past():
    global _print_interval, _last_time, _obj_cnt
    _obj_cnt += 1
    if _obj_cnt%_print_interval == 0:
        ms_past = (time.time()-_last_time)*1000.
        print(f"Took {ms_past:.0f}ms for the last {_print_interval} objects.")
        _last_time = time.time()

def walk_scurves(root_file, in_dir="", basepath=""):
    tdir = root_file.GetDirectory(in_dir)
    ROOT.SetOwnership(tdir, True) #IMPORTANT!
    basepath = os.path.join(basepath, in_dir)

    for key in tdir.GetListOfKeys():
        name = key.GetName()
        classname = key.GetClassName()
        if classname == "TEfficiency":
            yield os.path.join(basepath, name)
        elif classname.startswith("TDirectory"):
            for scurve in walk_scurves(tdir, name, basepath):
                yield scurve

def main():
    fname = "scurves.root"

    root_file = ROOT.TFile(fname,"READ")
    for scurve in walk_scurves(root_file):
        obj = root_file.Get(scurve)
        ROOT.SetOwnership(obj, True) #IMPORTANT!
        #Do something with object
        print_time_past()

    root_file.Close()


main()

and the output:

Took 719ms for the last 1000 objects.
Took 91ms for the last 1000 objects.
Took 89ms for the last 1000 objects.
Took 90ms for the last 1000 objects.
Took 89ms for the last 1000 objects.
Took 88ms for the last 1000 objects.
Took 89ms for the last 1000 objects.
Took 88ms for the last 1000 objects.
Took 89ms for the last 1000 objects.
Took 89ms for the last 1000 objects.
Took 83ms for the last 1000 objects.
Took 72ms for the last 1000 objects.
Took 76ms for the last 1000 objects.
Took 74ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 72ms for the last 1000 objects.
Took 74ms for the last 1000 objects.
Took 76ms for the last 1000 objects.
Took 75ms for the last 1000 objects.
Took 74ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 74ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 74ms for the last 1000 objects.
Took 76ms for the last 1000 objects.
Took 78ms for the last 1000 objects.
Took 78ms for the last 1000 objects.
Took 76ms for the last 1000 objects.
Took 75ms for the last 1000 objects.
Took 75ms for the last 1000 objects.
Took 75ms for the last 1000 objects.
Took 74ms for the last 1000 objects.
Took 76ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 78ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 72ms for the last 1000 objects.
Took 74ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 75ms for the last 1000 objects.
Took 71ms for the last 1000 objects.
Took 73ms for the last 1000 objects.
Took 72ms for the last 1000 objects.
Took 79ms for the last 1000 objects.
Took 74ms for the last 1000 objects.
Took 75ms for the last 1000 objects.
Took 80ms for the last 1000 objects.
Took 78ms for the last 1000 objects.
Took 75ms for the last 1000 objects.
Took 75ms for the last 1000 objects.
Took 76ms for the last 1000 objects.

Thanks a lot for your support!

Cheers,
Daniel

PS: Please let me know if you think that this is not the proper solution to the issue