How to fill a histogram in parallel?

Newbie
1

So I have a TTree we’ll call it Data_A
and it has a branch named “Time” and there are millions to sometimes billions of entries.
My data analysis is based on the difference in the “Time” values between each entry.
And I want to do it with the difference between 2 different parameters

difference between consecutive time entries
difference between time entries with 1 separation inbetween

The code below is what it looks like currently as a single threaded process.
The problem I’m having is that some Data_A files are simply massive, and can take hours to process.
So I wanted to utilize tools to potentially cut down on this significantly.
I tried looking into multithreading tutorials but none of them really explain well what are the proper steps are for doing a task like this because they work with different variables within the same TTree entry. ROOT::EnableImplicitMT()" does not seem to improve my performance at all. So I want to do explicit multicore/threading.

There are two ways I am currently thinking about approaching this, and pointers on how I can achieve them would be greatly appreciated.
The first is to cut the TTree in segments of 50e6 entries, then process each of the files with a core.
The second is to just divide the Data_A file by “n_workers” number of segments then do the same.
Tips, tricks or suggestions please?

void main(TString fname)
{
    TFile *file_0 = new TFile(fname.Data());
    ULong64_t time{0};
    Data_A -> SetBranchAddress("Time",&time);
    TH1D* hdiff0 = new TH1D("diff0","", 1e2,0,1e6);
    TH1D* hdiff1 = new TH1D("diff1","", 1e2,0,1e6);

    for (Long64_t i = 0; i < Data_A->GetEntries()-1; i++)
    {
        Data_A->GetEntry(i);
        double time0 = time;
        double diff = 0;

        Data_A->GetEntry(i+1);
        diff = time - time0;
        hdiff0->Fill(diff);

        Data_A->GetEntry(i+2);
        diff = time - time0;
        hdiff1->Fill(diff);
    }

    hdiff0 ->Draw();
    hdiff1->Draw("same")
}

Please read tips for efficient and successful posting and posting code

ROOT Version: 6.24
Platform: Ubuntu
Compiler: Not Provided

2

Hmmm, I would suggest the following:

Load first your time column into an std::vector<ULong64_t>, using:
https://root.cern/doc/master/classROOT_1_1RDF_1_1RInterface.html#aa571f08b8e4d9300cacdea039a379a22

or by just doing:
vector<int> v; v.reserve(Data_A->GetEntries(); for(Long64_t i = 0; i< Data_A->GetEntries(); ++i){Data_A.GetEntry(i); v.emplace_back(time);}

or in Python:

Then follow this example:

#include <iostream>
#include <vector>
#include <numeric>

int main()
{
    std::vector<int> v {4, 6, 9, 13, 18, 19, 19, 15, 10};
    for(auto& r : v) std::cout << r << "\t"; std::cout << std::endl;
    //Calculate now the difference between adjacent elements
    std::adjacent_difference(v.begin(), v.end(), v.begin());//here, set execution policy to parallel
    v.erase(v.begin()); // pop_front
    for(auto& r : v) std::cout << r << "\t"; std::cout << std::endl;
    //Calculate now the adjacent sum of the adjacent difference, which equals the second-adjacent difference of the first vector
    std::adjacent_difference(v.begin(), v.end(), v.begin(), [](const int x, const int y) { return x+y; });//here, set execution policy to parallel
    v.erase(v.begin()); // pop_front
    for(auto& r : v) std::cout << r << "\t"; std::cout << std::endl;
    return 0;
}

which gives what you are looking for:

4       6       9       13      18      19      19      15      10
2       3       4       5       1       0       -4      -5
5       7       9       6       1       -4      -9

You just need to set the execution policy to parallel.

Then, you can fill the histograms in parallel, see ROOT: tutorials/multicore/mt201_parallelHistoFill.C File Reference

If your dataset is too big and does not fit on RAM, you can do it in batches of 1e8 events or something like that.

3

Hmmm that looks pretty good, I’ll give it a shot