Home | News | Documentation | Download

JIT Compilation Memory Hogs in RDataFrame

Hello,

We are encountering this RDF JIT memory hog issue that has been discussed at length in other forum posts. Our use case is ntuple production postprocessing in ATLAS analysis. We define an RDataFrame for each of our ntuples (when we process systematics, that’s ~115 ntuples per file) and apply some Filters and Defines, and save the output in a Snapshot. When processing systematics, we run into the memory hog issue and causes a crash when running production on the grid. Release 6.22 is not available on the grid at the moment, but local tests do reflect an improvement in memory management advertised by the ROOT team (though the runaway behavior of memory hogging still exists, see attached plot).

We have implemented the suggested workaround with python’s multiprocess module, as suggested by eguiraud in other forum posts (e.g. this post from a few weeks ago). We trigger each of the RDF event loops in a separate Process and it solves the memory hog issue.

This is all to say that 1) we are another of a growing number of user groups who are impacted by this JIT compilation behavior and 2) while developers work to fix it, the suggested workaround with the multiprocess module works for PyROOT users.

Cheers!

RDF_MemoryLeak_R6.22.pdf (103.5 KB)
RDF_Processes_R6.22.pdf (82.1 KB)


ROOT Version: 6.20.02
Platform: x86_64-centos7
Compiler: gcc8-opt


1 Like

Hi @BrendonB ,
thank you very much for the feedback, this is good to know!

Cheers,
Enrico

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.