Memory reduction when loading ROOT RDataFrame in python

Newbie
,
1

Please read tips for efficient and successful posting and posting code

ROOT Version: Not Provided
Platform: lxplus
*Compiler:*gcc 13.1

When working with python and RDataFrame one often has the problem that a large amount of memory is used. In particular, when using cling and C++ functions that are not precompiled. In this contest I was wondering why a simple ROOT import does already take 600 Mbyte and if there is a way to reduce it. I run my jobs on the grid where I read in a tree and create histograms. So I do not need any graphics etc.. Together with MakexAODDataFrame I start with 1 Gbyte of memory just loading the libraries. This is prohibitive for grid jobs since often the sites have only 2 Gbyte.
Could one have a root light version for such applications ?

The attached files is a simple example of how much memory is used using ROOT in python
Thank you for your help.

test.py (994 Bytes)

2

Maybe @vpadulan or @Danilo can comment on this.
And BTW, FWIW, I see less memory usage on Windows:

C:\root-dev\rootdev>python test.py
print_memory:  print_memory  Starting: Print memory consumption:
print_memory:  Resident memory: 294.953125 MB
print_memory:  Virtual memory : 168.81640625 MB
3

Just a clarification.

On lxplus vanilla
print_memory: print_memory Starting: Print memory consumption:
print_memory: Resident memory: 285.37890625 MB
print_memory: Virtual memory : 384.734375 MB

After
setupATLAS
asetup AnalysisBase,25.2.45

print_memory: print_memory Starting: Print memory consumption:
print_memory: Resident memory: 455.78515625 MB
print_memory: Virtual memory : 564.23046875 MB

The larger memory was using a docker image of AnalysisBase on my laptop

4

Dear @tancredi,

Thanks for reaching out to the forum!

One possible source of discrepancy between the two environments is that in the ATLAS environment many more libraries are loaded by default by AnalysisBase, because they are needed for the rest of ATLAS processing. One starting point to look into this could be running ROOT.gSystem.ListLibraries() in both environments and see the differences.

I see the potential issue with this, clearly we always strive to be as lightweight as possible, Iā€™m sure that improvements are always possible.

Cheers,

Vincenzo

1 Like
5

Yes, the ATLAS libraries add as well. We probably should work on both. However, for ROOT is there a way to have a light weight version, e.g. that does not use graphic libraries ?

I also noticed that when writing in-time compiled code in python/RDataFrame the memory usage increases a lot. Here, I could make use of pre-compiled code. I am still wondering why this is so memory-heavy.
Tancredi