Trigger several times event loops with RDataFrame

ROOT
#1

Dear ROOT experts,
I have a code which was written like this

TChain *chain = GetChainFromSomewhere() ; 
for( slot_computation : CONFIGURATIONS_SETTINGS){
    ROOT::EnableImplicitMT( 8) ; 
    ROOT::RDataFrame df( *chain); 
    RNode lastNode = df.Define("DUMMY","1>0");
    for( xx: SEVERAL SUBSETUPS){ 
      lastNode = lastNode.XXX() ; 
   } 
   *df.Count(); 
   //Store results for this configuration
   ROOT::DisableIMplicitMT() ; //Neede to store in a sorted way with RDataFrame some results i produced
}

With this configuration of the code , after i am iterating for 1-2 slot_computation , the code breaks and it cnsumes an enourmous amount of memory.

If i refactor the code to do this :

TChain *chain = GetChainFromSomewhere() ; 
ROOT::RDataFrame df( *chain); 
for( slot_computation : CONFIGURATIONS_SETTINGS){
    ROOT::EnableImplicitMT( 8) ; 
    RNode lastNode = df.Define("DUMMY","1>0");
    for( xx: SEVERAL SUBSETUPS){ 
      lastNode = lastNode.XXX() ; 
   } 
   *df.Count(); 
   //Store results for this configuration
   ROOT::DisableIMplicitMT() ; //Neede to store in a sorted way with RDataFrame some results i produced
}

The code consumes much less memory and it completes successfully.

In practice, i know i would need to refactor the code to “bookkep” everything once, and just do 1 event loop, but it would require quite some time to refactor containers of results to have an extra bookkeping level in my code.

I wonder if this behaviour is expected.
Cheers
Renato

Please read tips for efficient and successful posting and posting code

_ROOT Version: 6.18/04
Platform: x86_64-centos7-gcc8-opt
Compiler: linuxx8664gcc

#2

You should probably take a look at what is consuming the memory. You can run your program under valgrind --tool=massif and check its output for instance.

But it also runs on a single thread, right? EnableImplicitMT should be called before the RDF object is constructed (with more recent ROOT versions you should get a warning).
Also I’m not sure what purpose it serves to EnableImplicitMT and DisableImplicitMT in a loop, in the end all event loops run multi-threaded (i.e. they shuffle entries).

You should see a reduced amount of memory used if you refactor this way, which runs a single event loop:

TChain *chain = GetChainFromSomewhere() ; 
ROOT::EnableImplicitMT( 8) ; 
ROOT::RDataFrame df( *chain); 
std::vector<ROOT::RDF::RResultPtr<ULong64_t>> counts;
for( slot_computation : CONFIGURATIONS_SETTINGS){
    RNode lastNode = df.Define("DUMMY","1>0");
    for( xx: SEVERAL SUBSETUPS){ 
      lastNode = lastNode.XXX() ; 
   } 
   counts.emplace_back(df.Count()); 
   //Store results for this configuration
   ROOT::DisableIMplicitMT() ; //Neede to store in a sorted way with RDataFrame some results i produced
}

counts[0].GetValue();  // trigger the event loop

If that’s not an option, in 6.22 you should still see a reduced memory usage if you do it like this:

TChain *chain = GetChainFromSomewhere() ; 
std::vector<ROOT::RDF::RResultPtr<ULong64_t>> counts;
for( slot_computation : CONFIGURATIONS_SETTINGS){
    ROOT::EnableImplicitMT( 8) ; 
    ROOT::RDataFrame df( *chain); 
    RNode lastNode = df.Define("DUMMY","1>0");
    for( xx: SEVERAL SUBSETUPS){ 
      lastNode = lastNode.XXX() ; 
   } 
   counts.emplace_back(df.Count()); 
   //Store results for this configuration
   ROOT::DisableIMplicitMT() ; //Neede to store in a sorted way with RDataFrame some results i produced
}

// trigger all event loops
for (auto &count : counts)
  *count;

and in the upcoming 6.24, even if you have different RDataFrames like in the last example you can run all of their separate event loops concurrently substituting

std::vector<ROOT::RDF::RResultPtr<ULong64_t>> counts;
...
for (auto &count : counts)
  *count;

with

std::vector<ROOT::RDF::RResultHandle> counts;
...
ROOT::RDF::RunGraphs(counts);

but that does not necessarily reduce memory consumption.

Still, best thing is to measure what’s taking up memory.
Cheers,
Enrico

#3

I do this because after i disable it i collect
epsNumerator =df.Filter(x).Take(RVec coulumn)
epsDenominator=df.Filter(y).Take(RVec column2)

I know in advance that my each single column vector has 100 entries, and i snapshot to disk a new ntuple using RDataframe and i want the entires to match the [i] of the entries column i extracted. If i don’t disable it, the indexes get shuffled and this is something i need to avoid (without entering in details, i bootstrap efficiencies results, and i expect to estimate correlations, so i need to keep indexes ordered l

#4

The main reason why i cannot refactor better the code is that i have not yet managed to implement something allowing to do a Sum<RVec>( vectorcolumn)

I.e if i have a 100 variation of my weight attached to the ntuple i want to extract the sum[i] of each. I wrote the code like this because i am currently Taking the vector column and summing by hand. I will post here the helper i will implement for that.

#5

I can refactor my code avoiding to Take<RVec<double>> columns in each loop using what i posted in https://root-forum.cern.ch/t/sum-for-each-rvec-column-returning-an-rvec/42259.
Thanks @eguiraud for the suggested refactoring, i will go in that direction, but i need to have the

RVec<double> sumWColumns = df.Sum< RVec<double> >("vectorColumn");

Implemented before doing it.

#6

Refactoring the code as you suggested and using the Reduce calls to gather sum of vector column as vector of double, helped to get rid of many Take<> i had for an overall factor 100 speed up. Indeed triggering event loops only once and work out a solution where one can avoid to Take columns for postprocessing is much faster.

#7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.