TTree Draw() and threadsafety with openMP

ROOT
1

Hi all,

I have some code which returns me the weighted yield and uncertainty from a TTree like so:

    TH1D *tmpWeightHisto = new TH1D("tmpWeightHisto","tmpWeightHisto",1,-3.0,3.0);
        tmpWeightHisto->Sumw2();
        if(*cut != ""){
                procNtuple->Draw(*cut+">>tmpWeightHisto","("+*cut+")*("+perEventWeightVar+")");
        }else{
                procNtuple->Draw("1>>tmpWeightHisto","("+perEventWeightVar+")");
        }
        *Entries = tmpWeightHisto->Integral();
        *d_Entries = sqrt(tmpWeightHisto->GetSumw2()->At(1));
        tmpWeightHisto->Delete();

In my use case, I perform this operation over several ntuples many times. It’s an ideal candidate for paralellisation, so I threw OMP at it. To my surprise the rest of the code works beautifully with OMP, except for this one part, which segfaults unless I put a critical directive around it like so:

        //Needed to make a unique histo per thread
        TString n("tmpWeightHisto_");
        #ifdef _OPENMP
        char uniq = 'a'+(omp_get_thread_num()%26);
        n+=uniq;
        n+=+"_";
        #endif

       TH1D *tmpWeightHisto = new TH1D(n,n,1,-3.0,3.0);
        tmpWeightHisto->Sumw2();
        if(*cut != ""){
                //Critical directives added here to prevent crashing. Literally only these two calls seem to cause crashes 
                #pragma omp critical(crasheswithoutme)
                {
                procNtuple->Draw(*cut+">>"+n,"("+*cut+")*("+perEventWeightVar+")");
                }
        }else{
                #pragma omp critical(crasheswithoutme)
                {
                procNtuple->Draw("1>>"+n,"("+perEventWeightVar+")");
                }
        }
        TH1D *tmpWeightHisto = (TH1D*)gDirectory->Get(n);
        *Entries = tmpWeightHisto->Integral();
        *d_Entries = sqrt(tmpWeightHisto->GetSumw2()->At(1));
        tmpWeightHisto->Delete();

Adding the critical directives means that now, my rather large codebase works multithreaded, even with TMVA running in each thread. Unfortunately, the Draw() is where the code spends most of its time, so performance is actually reduced by a factor of almost 2 when running with 4 threads on a 4-core machine compared to running single-threaded as the critical is adding a huge overhead. I have two questions:

1: Is this a dumb way to obtain a weighted yield from an ntuple with cuts? Ie: is there a more efficient and thread-safe method of applying a cut string and getting back a weighted yield +/- uncert?
2: If this is the best way to do it, is there a way I can make it thread safe without locking? I’m open to other multithreading options than OMP, but it’s the one I’m most familiar with and given that everything nearly works with only a couple of additional lines of pragma, I’d be happiest with a solution that still makes use of it.

Thanks!

2

Hi Conor,

we have an example in our set of Multicore tutorial for running in parallel over several ntuples:
root.cern.ch/doc/master/mt102__ … it_8C.html
the threading model is not important: we used std::thread for that particular code.
In general the strategy to be followed is:

  1. Enable ROOT thread safety with a call to ROOT::EnableThreadSafety()
  2. Treat one file and one tree per thread

Can your usecase expressed in a small-ish piece of code? It could be interesting to add it to the portfolio of the existing ROOT examples.

Cheers,
Danilo

3

Hi Danilo,

Thanks for the fast response!

I was under the impression from your comment in another thread that ROOT::EnableThreadSafety() doesn’t yet work (even though 6.06 release notes indicate otherwise):

I have been calling this and/or TThread::Initialise() already. My suspicion with the tutorial you linked to is that in my use case I am calling Draw() on a single file that is used in more than one thread. I have a dozen or so TFiles that are unique to each thread, and a single TFile that is shared across the threads. These are all only using the Draw command and are opened read-only. In the tutorial it looks like you’re acting only on unique files in each.

My use case relies on mountains of additional classes I’ve written over the years, so I’m not sure I can slim it down to a size that fits nicely in a tutorial, but I’ll have a think.

4

Hi Conor,

ROOT::EnableThreadSafety() works well but exists since ROOT 6.08 (the previous incarnation of it is TThread::Initialize()) :slight_smile: Since ROOT 6.08 the programming model is simpler since it’s possible to use directly any threading model.

What is still a requirement is to use a TFile per thread: if possible this should be changed in your setup. ROOT is then able to follow in a thread safe manner a different “stream of execution” for every thread.

This is understood. Thanks for giving it a think :slight_smile:

Cheers,
Danilo