Bad_alloc issue in TH1 using RDataFrame

[imanol097]

Dear experts,

I am reading a quite big rootfile, 600M rows, and producing some histograms using RDataFrame.
When trying to manipulate the TH1 the issue that I have is the following:

bad_alloc: TH1D& ROOT::RDF::RResultPtr<TH1D>::operator*() =>
    bad_alloc: std::bad_alloc

Is this a memory error independent of RDataFrame or it is related to it ?
Can you guide me a little bit?

Cheers,
Imanol

---

Please read tips for efficient and successful posting and posting code

Please fill also the fields below. Note that root -b -q will tell you this info, and starting from 6.28/06 upwards, you can call .forum bug from the ROOT prompt to pre-populate a topic.

_ROOT Version: 6.28/04
Platform: Not Provided
Compiler: Not Provided

---

[bellenot]

It might be a memory issue related to e.g. the number of bins of your histogram, but in order to help more, we would need a minimal reproducer, or at least a view on your code. And I invite @vpadulan to give some more hints

[imanol097]

Hi @bellenot, thanks for answering.

The same code runs well with smaller data sets, so I don’t think the problem is the number of bins.

The code is very simple and basically does histograms in a loop and saves them into a dictionary:

for (b, m, sample, nch, eta_bin) in product(BeamConf, MagConf, samples, Nch_list, eta_bins):
    rdf          = RDataFrame(tree, file)
    filter_rdf = rdf.Filter(MultiplicityUpperRange + "&&" + MultiplicityLowerRange)
  
    variable_h1D[b][m][sample][nch][eta_bin]  = filter_rdf.Histo1D(h1D_pT_model, variable)
    variable_h1D[b][m][sample][nch][eta_bin].SetDirectory(0)

[Danilo]

Hello,

It depends very much on the nature of the datasets, but it’s not unreasonable to expect a slightly increased memory usage with larger datasets (because of memory buffers, for example), but the difference would not be huge, in absence of issues such as memory leaks in constructors of classes composing the data model.

However, allow me to be pragmatic: how many histograms is the code booking and how many bins does the model have?

Cheers,
Danilo