Speeding up histogram process

So I have this large root file that is about 24GB size,

With a set of cut conditions, I will produce about 100 sets of histogram with corresponds do different cut condition.

For example, for a cut condition that range from 1 to 100 in 1.0 steps.

So to histogram it,

for (int i=1; i<101;i++){
//my code here
}

How it is exactly can this be done that I split this into 4 parallel process (4 core machine) with the 1st core handles the i=1 with second core with i=2 and so on?

I tried read through the documentation but I am confused how this can be done. apologies if this has been asked before.

Hello,

Can you say which version of ROOT you are using?
We recently new simpler ways to do this, so depending on the version the solution can be different.

G Ganis

…where, FYI, the optimal approach is to split the input data into chunks, not the histograms :slight_smile: ROOT does that for you almost automatically, depending on the ROOT version you use - see Gerri’s reply!

Are you sure you have the resource to read the disk at many different locations at once? If you don’t then its most likely to slow your histogram filling to do it paralelly. Btw. C++17 now provides std::execution::parallel_policy to help you running std::for_each for the jobs you want to perform.

I am doing it with root v5.34.32 on a server currently (may switch to rootv6.08 later once i figured out how to edit my .bashrc) I am interested in knowing both ways.

Hello,

Sorry for coming back only now.
Assuming that your file contains a TTree, your problem has two dimensions on which you could parallelize: the number of events in the TTree and the number of histogram.
I think you have first to understand what is your bottleneck.

The traditional way to parallelize is with Proof. This will work in root 5 and 6. By default you can drive Proof with the TTree, paralleling in the events in the TTree.
But you can also set it up to parallelize on the histogram id.

In all case you have to express you code in a TSelector, where the Process(Long64_t entry) method does the job of filling the histogram. (‘entry’ will be the TTree entry or the histogram id).

For the latest versions of Root 6, you have alternatives with Lambda functions. Have a look under tutorials/multicore of your root 6 installation. (Note however that this is still somehow experimental).

G Ganis

Dear Gerri,

Thank you for your kind reply. I suppose the tutorial/multicore/mt001_fillHistos.C is a good example of it.

As I am trying to run this in a remote server, The root version I am using is v6.04.02. So I have been using the default root that is installed by the server admin. I supposed that the admin who built root needs to allow multicore when building it? cause when I tried to run it, it said root failed to locate EnableThreadSafety. Correct me if I’m wrong.

As the raw data I am dealing with are of order of TB so I have been doing it on a remote server of my institution. Actual root files I dealing per script are few GBs.

Danny

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.