Hello,
We are encountering this RDF JIT memory hog issue that has been discussed at length in other forum posts. Our use case is ntuple production postprocessing in ATLAS analysis. We define an RDataFrame for each of our ntuples (when we process systematics, that’s ~115 ntuples per file) and apply some Filters and Defines, and save the output in a Snapshot. When processing systematics, we run into the memory hog issue and causes a crash when running production on the grid. Release 6.22 is not available on the grid at the moment, but local tests do reflect an improvement in memory management advertised by the ROOT team (though the runaway behavior of memory hogging still exists, see attached plot).
We have implemented the suggested workaround with python’s multiprocess module, as suggested by eguiraud in other forum posts (e.g. this post from a few weeks ago). We trigger each of the RDF event loops in a separate Process and it solves the memory hog issue.
This is all to say that 1) we are another of a growing number of user groups who are impacted by this JIT compilation behavior and 2) while developers work to fix it, the suggested workaround with the multiprocess module works for PyROOT users.
Cheers!
RDF_MemoryLeak_R6.22.pdf (103.5 KB)
RDF_Processes_R6.22.pdf (82.1 KB)
ROOT Version: 6.20.02
Platform: x86_64-centos7
Compiler: gcc8-opt