RDataFrame MT performance running on remote files

[RENATO_QUAGLIANI]

Can you share the code where you do all the operations?

I had in the past a huge slow down when i was doing a huge amount of :

Filter().Define("weight", wEpression).Histo1D() 

I don’t know if that’s the case for you as well, the solution i found to make this as fast as possible was to refactor my code to have the Weight Definition only once with different bookkeped names if the weights were changing, and given that i was cutting several time my dataframe, and most of the time in the same way, i created a map< TString, RNode> to keep the nodes form Filters in a sort of cached place, to don’t redo the computation and compilation of the cut several time .
I have never run profilers for RDataFrame applications… but sometimes refactor the nodes building helped me to get much faster event loops…

Just for reference, we do observed several issues related to eos instabilities and RDataFrame in our analysis as well. Our code was not running with the same performance speed-up from a remote machine or from lxplus reading the files at /eos. That also might well be the reason of what you see.

[zhubacek]

This might hijack the original performance issue, but I tried to create a graph with what I’m trying to achieve

From the main frame, I have a couple of Defines and simple Filters, then I would like to split the events into N categories and in these categories to M bins, where N,M I don’t know in the beginning/would like to set them up in a configuration file.
I have the code working unfortunately it is not efficient - going from 1 to 2 categories basically doubles the processing time. I do it maybe in a similar way as you do, only instead of a map, I keep a vector (size N) of DataFrames/Nodes pointers after I split them with a Filter and use these pointers then to split them into M bins (2D vector NxM), then I have a 2D vector (also NxM) of TH1DModels which are filled from each of the frame

1. Any idea where this becomes inefficient?

2. Some of the Defines in my code are ‘constants’ used in some later cuts/filters - I put those as new columns in the DataFrame instead of capturing these variables in a lambda. This was mainly that I could write each Filter as a full function which I could put into a separate file instead of writing a lambda in the middle of the code - is this a good idea or should I try to avoid unnecessary new columns? Most of the Defines are ints/floats or RVec, RVec, not complicated objects.

Thanks,
Zdenek

[jblomer]

What might help is a system-wide flame graph to see what the CPU is doing when its utilization drops. It would be interesting to know if it is waiting for more data or if it is it stuck in a global lock. @eguiraud might have more ideas.

[eguiraud]

Hi,
It would be great to be able to run this ourselves to investigate properly, but from the plots these are my best guesses:

• the first phase, that lasts about the same independently of where data is stored, is just-in-time compilation. String Filter/Defines such as Filter("x > 0") as well as non-templated actions such as Histo1D("x") (as opposed to Histo1D<float>("x")) need to be just-in-time compiled right before starting the event loop. I very much hope those are not really 3000 seconds, 3 seconds would be a more appropriate runtime for this phase
• the drop in CPU usage in the second phase when reading for EOS is most probably due to too-slow network I/O: CPUs need to wait for the data
• the increase in memory usage after the first phase is due to RDataFrame actually creating the histograms and all other storage necessary for the analysis at that point, right before starting the data processing. Running the program under valgrind --tool=massif should confirm that’s the case.

Do these statements sound reasonable?

Performance tips:

• avoid unnecessary columns (e.g. for those constants that you mentioned), they have a cost and RDF does not know they are constants so actually re-evaluates the expressions at every entry
• to split events into categories, I suppose you are using one Filter and one histogram per category? That is much less efficient than e.g. using a single higher-dimensional histogram, with its “slices” representing categories, or even a dedicated custom action (we should really provide such an action or at least an helper, but I could never quite put my finger on the scenarios we need to cover)

Hope this helps!
Cheers,
Enrico

[zhubacek]

Hi @eguiraud,
Thanks a lot! There are several good suggestions which I will try!

• the first phase, that lasts about the same independently of where data is stored, is just-in-time compilation. String Filter/Defines such as Filter("x > 0") as well as non-templated actions such as Histo1D("x") (as opposed to Histo1D<float>("x")) need to be just-in-time compiled right before starting the event loop. I very much hope those are not really 3000 seconds, 3 seconds would be a more appropriate runtime for this phase

Is there any counter available how many objects are compiled? I don’t know if in my case it gets compiled for each category? It also seems that the first part scales with the number of events in my case?

Does the Histo1D(“x”) work also for RVec<float> ? I will try it for saving on the histogramming part

Performance tips:

• avoid unnecessary columns (e.g. for those constants that you mentioned), they have a cost and RDF does not know they are constants so actually re-evaluates the expressions at every entry

Ok, thanks! I will remove what I don’t really need. Do you have a good style suggestion - I use these constants in Filters, having them as columns, I could define the cut function in a separate header file - can I write a Filter function that takes both columns and other non-column variables as parameters? I could capture the parameter in a lambda but that would mean that I need to write the lambda somewhere in the middle of the code?

• to split events into categories, I suppose you are using one Filter and one histogram per category? That is much less efficient than e.g. using a single higher-dimensional histogram, with its “slices” representing categories, or even a dedicated custom action (we should really provide such an action or at least an helper, but I could never quite put my finger on the scenarios we need to cover)

Thanks for the suggestion. It’s ultimately more than one histogram (like a vector of configured histograms) for each category (so M x N x O 3D array of TH1DModels, M,N,O are relatively small <10), but I will think if I can rewrite this in TH3DModels instead

Hope this helps!
Cheers,
Enrico

Thanks!
Cheers
Zdenek

[eguiraud]

@zhubacek

Is there any counter available how many objects are compiled?

Not really, but every Filter(string), Define(string, string) and Histo*D(...) call has to be just-in-time compiled. Filter(callable), Define(string, callable) and Histo*D<ColType>(...) do not require just-in-time compilation.

@zhubacek

It also seems that the first part scales with the number of events in my case?

Ah that’s interesting, I don’t know what to make of it. As I mentioned it would be great to be able to run this ourselves to investigate properly. Could you provide a reproducer that we can run e.g. on lxplus?

Alternatively, as @jblomer suggested it would be interesting to look at a flamegraph of your application.

@zhubacek

Does the Histo1D(“x”) work also for RVec ?

Yes, and Histo1D<ROOT::RVec<float>>("x") as well.

@zhubacek

Do you have a good style suggestion - I use these constants in Filters, having them as columns, I could define the cut function in a separate header file - can I write a Filter function that takes both columns and other non-column variables as parameters?

If you would rather not define inline lambdas, I would suggest to create them from a lambda factory:

auto MakeCut(float constant) {
  return [constant](float x, float y) { return x + y > constant; };
}

Cheers,
Enrico

[zhubacek]

Hello Enrico,
thank you very much for your suggestions!
A few remarks on what I tried:

1. The lambda factories are nice! For my C++ experience (or lack of it, especially in functions returning functions ) it somewhat abuses the C++ syntax I’m used to, but it allowed to reduce the number of Defines

2. The templated Histo1D unfortunately hinders my efforts to be able to plot anything from a configuration, I tried to rewrite this as

if (frame.GetColumnType(variable) == "type1")
{
  result = frame.Histo1D<type1>(variable);
}
else if (frame.GetColumnType(variable)=="type2")
{
 result = frame.Histo1D<type2>(variable);
}
else
{
result = frame.Histo1D(variable)
}
...

but I don’t know if the GetColumnType()/comparison with strings is efficient here, I could in principle rewrite this to an integer comparisons as I could know the types

if (GiveMeType(variable) == 1)
{
result = frame.Histo1D<type1>(variable);
}
else if (GiveMeType(variable) == 2)
{
result = frame.Histo1D<type2>(variable);
}
...

but it’s still not elegant as I have to repeat this a few times in my code…

But generally 1 and 2 didn’t bring a significant improvement so
3) I think the most important would be to avoid the splitting into separate dataframes and use TH2D/TH3D and slice them in the end
However, I don’t know how to fill the 2D histogram for this case?
Let’s say I will have a TH2DModel, x-axis will be the variable of interest, y-axis will be the categories - it seems that Histo2D will only support filling x vs y or vector<x> for 1 category?

In a loop-style, I would do something like

for (auto x: vector<x>)
{
  for (auto category : categories)
   {
     my2Dhisto->Fill(x, category, weight);
   }
}

(event could be filled for several categories)
but is it possible to achieve this with Histo2D?

Thanks,
Zdenek

[eguiraud]

Hi,
sorry for the latency, I’ll take a look today/tomorrow!

[eguiraud]

@zhubacek

it seems that Histo2D will only support filling x vs y or vector<x> for 1 category?

vectorx vs vectory (possibly with vector_weights) is also supported.

I don’t understand the situation: for every event, x is an array (which represents different variables) and each element of the array could belong to one or more categories? What is weight? 0/1 depending

EDIT:
I might be missing the point, but how about a TH2D per variable, with the category on the x axis and the variable values on the y axis?

This works if the variable you ultimately want to fill histograms with is a scalar for every event:

// on_categories is an array with the bins of the active categories for this event
df_c = df.Define("on_categories", "{1,3,4,6}", {....});
for (auto &varname : varnames)
  df.Histo2D(..., {"on_categories", varname});

If the variable is an array per event, the default behavior of Histo2D is not what you want, so you probably need Aggregate:

struct CategoriesAndValues {
  RVec<int> on_categories;
  RVec<double> values;
};
df_c = df.Define("categories_and_values", "CategoriesAndValues{cats, xs}", {...});
auto fill = [] (TH2D &h, const CategoriesAndValues &c) {
  for (auto cat : c.on_categories)
    for (auto x : c.values)
      h.Fill(cat, x);
};
auto add_histos = [] (TH2D &h1, TH2D &h2) { h1->Add(h2); };
auto th2 = df.Aggregate(fill, add_histos, {"categories_and_values"});

[zhubacek]

Hello,
The EDIT part is what I meant! Thank you!

The scalar option is nice, unfortunately the variables of interests are vectors/arrays so I will have to go the second option. The weight is an event weight, so I think I would need to add it to the struct CategoriesAndValues as well.

I’m a bit worried about the performance since it means re-adding the same variable, but I’ll need to test it first

[eguiraud]

@zhubacek

I’m a bit worried about the performance since it means re-adding the same variable, but I’ll need to test it first

The computation graph should shrink by a lot, and that should help performance greatly.

But again: every time performance is a concern, you should first of all measure what’s taking time, e.g. by looking at a flamegraph as suggested above.

[zhubacek]

Hello,
I have now managed to rewrite my code and I’m testing it at the moment:
I couldn’t get the Define work with the initialization list, I changed it to a separate function

auto Build2DObservable(ROOT::RVec<float> values, ROOT::RVec<int> categories, double weight)
{
 My2DObservablestruct x{values,categories,weight};
 return x;
}

and

df2 = df.Define("categories_and_values", Build2DObservable, {...});

I had to change the merger function form to

auto add_histos = [] (TH2D h1, TH2D h2) { h1.Add(&h2); return h1;};

as with U=TH2D in my case, the merger has a signature U(U,U), the aggregate is auto fill = [] (TH2D &h, const My2DObservablestruct &c) - is that correct?

In the performance:

the psrecord graph shows a huge (~factor of 10) improvement in time.

I’m at the moment investigating, but I’m getting a crash in the end when I try to create the RDataFrame graph - is it possible that I have too many Defines?
In my case, I not only have to (re)Define each My2DObservablestruct but since the ‘values’ in it take RVec<float>, I also re-Define non-RVec<float> (only int, float, RVec<int> in this example) variables to RVec<float> if I want to fill them using the same code

Is is somehow possible to ‘hack’ or change the column type in RDataFrame? Let’s say column is an integer, but I would like the RDataFrame think that it is a vector<float> of size 1?

Cheers,
Zdenek

[eguiraud]

Hi,

@zhubacek

the merger has a signature U(U,U), the aggregate is auto fill = [] (TH2D &h, const My2DObservablestruct &c) - is that correct?

Sounds good.

@zhubacek

the psrecord graph shows a huge (~factor of 10) improvement in time

Great!

@zhubacek

I’m getting a crash in the end when I try to create the RDataFrame graph - is it possible that I have too many Defines?

No, that should not cause a crash. Do you get a stacktrace? Can you share a minimal self-contained reproducer so that we can debug the crash on our side? Otherwise I suggest you compile the code with debug symbols (-g) and inspect the point of crash with gdb.

Ah I see this is now Crash when creating RDataFrame graph

@zhubacek

I also re-Define non-RVec<float> (only int, float, RVec<int> in this example) variables to RVec<float> if I want to fill them using the same code

If that becomes a problem you can use templates to write a generic My2DObservablestruct and two fill aggregate functions, one for the scalar case and one for the RVec case:

template <typename T>
class My2DObservablestruct {
  T values;
  RVec<int> categories;
};

template <typename T>
My2DObservablestruct<T> Build2DObservable(const T& values, const RVec<int> &categories) {
  return My2DObservablestruct<T>{values, categories};
}

// scalar fill
template <typename T>
void fill(TH2D &h, const My2DObservablestruct<T> &c) {
  h.Fill(...);
}

// vector fill
template <typename T>
void fill(TH2D &h, const My2DObservablestruct<RVec<T>> &c) {
  for (...)
    h.Fill(...);
}

// use as:
df.Define("categories_and_values", Build2DObservable<RVec<float>>, {...}).
  .Aggregate(..., fill<RVec<float>>, ...);

@zhubacek

Is is somehow possible to ‘hack’ or change the column type in RDataFrame? Let’s say column is an integer, but I would like the RDataFrame think that it is a vector<float> of size 1?

Unfortunately not, the type of the column matters because it tells RDF how to read the bytes from disk.

P.S.
better to take RVecs as const& function arguments to avoid extra copies: void fun(const RVec<float> &values) rather than void fun(RVec<float> values).

[zhubacek]

Thanks! I originally tried the templated version, but couldn’t make it work. I will try again.
I think I still need something like

if (df.GetColumnType("variable")=="int") 
{
df.Define(...,Build2DObservable<int>,...).Aggregate(...,fill<int>,...);
}
else if (df.GetColumnType("variable")=="ROOT::VecOps::RVec<int>")
{
df.Define(...,Build2DObservable<ROOT::VecOps::RVec<int>>,...).Aggregate(...,fill<ROOT::VecOps::RVec<int>>,...)
}
...

Or can I resolve the type in a better way?
Zdenek

[zhubacek]

@eguiraud

// scalar fill
template <typename T>
void fill(TH2D &h, const My2DObservablestruct<T> &c) {
  h.Fill(...);
}

// vector fill
template <typename T>
void fill(TH2D &h, const My2DObservablestruct<RVec<T>> &c) {
  for (...)
    h.Fill(...);
}

Hello, I unfortunatelly, couldn’t get it resolve the correct form of fill for the scalar and vector forms?
I had to use different names

template <typename T>
void fillscalar(TH2D &h, const My2DObservablestruct<T> &c) {

template <typename T>
void fillvector(TH2D &h, const My2DObservablestruct<T> &c) { 
//and here change RVec<T> to just T

(it isn’t a problem in my case)

[eguiraud]

If you don’t know the types in advance, that’s one way to do it. You can easily measure whether the setup time becomes too large.

[eguiraud]

That works too!

[zhubacek]

Hello,
The question is more if RDataFrame is able to resolve the correct type of the templated aggregator?
It works on a c++ level - simple example is attached test1.C (720 Bytes) as the functions have same names but different parameters, but I couldn’t get it work in RDataFrame Aggregate
test2.C (1.5 KB)
(it’s supposed to run on a simple tree with two branches - int x and std::vector<int> vecypokus1.root (6.2 KB) )

 //this doesn't work
  //auto total1 = df.Aggregate(agg<int>,add_all,"x",all1);
  //auto total2 = df.Aggregate(agg<std::vector<int>>,add_all,"y",all1);
  //this works
  auto total3 = df.Aggregate(aggscalar<int>,add_all,"x",all1);
  auto total4 = df.Aggregate(aggvector<std::vector<int>>,add_all,"vecy",all1);

I get something like

/Applications/root_v6.22.02/include/ROOT/RDF/RInterface.hxx:2029:18: note: 
      candidate template ignored: couldn't infer template argument 'AccFun'
   RResultPtr<U> Aggregate(AccFun aggregator, MergeFun merger, std::stri...

[eguiraud]

Ah that’s tricky, and the error message is horrible, sorry about that.
It’s because agg<int> and agg<std::vector<int>> are in principle ambiguous (they could both refer to both overloads) and the right overload can only be resolved when you call the function but RDF tries to figure out the signature of the function before calling it…

Anyway, simple workaround, make it a functor:

template <typename T>                                                                                                   
struct Aggregator {                                                                                                     
   void operator()(std::vector<int> &a, const T &b)                                                                     
   {                                                                                                                    
      std::cout << "f1" << std::endl;                                                                                   
      a.push_back(b);                                                                                                   
   }                                                                                                                    
};                                                                                                                      
                                                                                                                        
template <typename T>                                                                                                   
struct Aggregator<std::vector<T>> {                                                                                     
   void operator()(std::vector<int> &a, const std::vector<T> &b)                                                        
   {                                                                                                                    
      std::cout << "f2"                                                                                                 
                << " " << b.size() << std::endl;                                                                        
      a.insert(a.end(), b.begin(), b.end());                                                                            
   }                                                                                                                    
};   

...

auto total1 = df.Aggregate(Aggregator<int>{},add_all,"x",all1);                                                       
auto total2 = df.Aggregate(Aggregator<std::vector<int>>{},add_all,"y",all1);     

Cheers,
Enrico