Memory leak when looking on histograms

[wiso]

Describe the bug

When reading histograms from a TFile in a loop memory is not released.

Expected behaviour

In Python I expect when an object goes out of scope, its reference counter decreases and when reaching zero is marker to be deleted, releasing the memory.

I expect very few memory to be used by the script.
I also expect all the memory to be free when the file is closed.

To Reproduce

This script is very similar to the tutorial: ROOT: tutorials/pyroot/pyroot005_tfile_context_manager.py File Reference

import resource
import ROOT

def print_memory_usage(message):
    print(f"{message:50} {resource.getrusage(resource.RUSAGE_SELF).ru_maxrss}")

print_memory_usage("start")
histogram_names = open("histogram_names.txt").read().splitlines()
print_memory_usage("read histogram names")

fn = "NTUP_PHYSVAL.40023485._000001.pool.root.1"
with ROOT.TFile.Open(fn) as f:
    print_memory_usage("open ROOT file")
    for i, histogram_name in enumerate(sorted(histogram_names)):
        h = f[histogram_name]
        h.SetDirectory(0)  # doing nothing
        if i % 1000 == 0:
            print_memory_usage(f"read {i+1} histograms")
    print_memory_usage("read all histograms")
print_memory_usage("outside context maneger (closing ROOT file)")

The ROOT file (60MB) can be downloaded from CERNBox
The txt file is attached
histogram_names.txt (1.4 MB)

The output I get

start                                              537736
read histogram names                               541668
open ROOT file                                     556796
read 1 histograms                                  559996
read 1001 histograms                               562300
read 2001 histograms                               887816
read 3001 histograms                               1049608
read 4001 histograms                               1534472
read 5001 histograms                               1607304
read 6001 histograms                               1625864
read 7001 histograms                               1639048
read 8001 histograms                               1654408
read 9001 histograms                               1666312
read 10001 histograms                              1682312
read 11001 histograms                              1699208
read 12001 histograms                              1711368
read 13001 histograms                              1726728
read 14001 histograms                              1739528
read 15001 histograms                              2191056
read 16001 histograms                              2258512
read all histograms                                2260048
outside context maneger (closing ROOT file)        2260048

• Adding h.SetDirecotory(0) (already in the script) is not helping.
• Adding ROOT.TH1.AddDirectory(False) is not helping.
• Adding gc.collect() is not helping
• Adding h.Delete() is solving the issue, but I don’t see why it is needed.

Setup

ROOT v6.32.02
Built for linuxx8664gcc on Jul 08 2024, 09:53:51
From heads/master@tags/v6-32-02
With
Binary directory: /home/turra/micromamba/envs/mamba-python3.11/bin
python: 3.11.8

[silverweed]

Hello @wiso, thanks for the question and the reproducer!

The behavior you’re observing is indeed strange. I suspect it’s a bug on the python bindings of TDirectoryFile::Get which is not properly taking ownership of the object.
I am doing some tests locally to verify my hypothesis with the help of @vpadulan and @jonas.
I’ll be back to you briefly.

[silverweed]

Alright, after some testing I found that there are 2 things at play here:

firstly, you’re using the dictionary getattr version f[histogram_name] to retrieve the histograms. This way of getting the TObject causes it to be cached by pyroot for future calls, keeping the memory allocated until the program terminates.

The way to get a TObject from a TFile without it being cached is by using the Get method:

h = f.Get(histogram_name) # no caching happens under the hood

However, I discovered that this method is currently buggy and will actually leak the object, causing the exact same memory profile of the other one. Once this is fixed, using f.Get(histogram_name) yields the following:

start                                              411820
read histogram names                               411820
open ROOT file                                     464016
read 1 histograms                                  464016
read 1001 histograms                               464016
read 2001 histograms                               464016
read 3001 histograms                               464016
read 4001 histograms                               464864
read 5001 histograms                               465376
read 6001 histograms                               465376
read 7001 histograms                               465376
read 8001 histograms                               465376
read 9001 histograms                               465376
read 10001 histograms                              465540
read 11001 histograms                              465540
read 12001 histograms                              465540
read 13001 histograms                              465540
read 14001 histograms                              465540
read 15001 histograms                              491364
read 16001 histograms                              491876
read all histograms                                492132
outside context maneger (closing ROOT file)        492132

Hopefully we can merge the fix by 6.34. Once that’s done, remember to change your code to use Get if you don’t want the caching behavior and the issue should be fixed

Thanks for helping us spot the bug!

[wiso]

Thank you very much for the prompt solution. Can you point me to the merge request?

In addition, why in the last line, after closing the file the memory is not dropping?

[silverweed]

Hi @wiso,
unfortunately my simple fix breaks some unit tests, so we need to investigate a bit to understand why. In principle it seems that granting python ownership on the object should be the correct solution, but there are probably some subtlelties to iron out.
I opened an issue to keep track of it, so you can follow its progress.

In addition, why in the last line, after closing the file the memory is not dropping?

I’m not 100% sure, but consider that some memory is allocated by ROOT and cppyy for bookkeeping, or by the cling interpreter (not to mention by the python runtime itself), so I wouldn’t expect it to drop to the initial value.