Home | News | Documentation | Download

Proof in python

Hi
I want to implement proof in python through pyroot. Now I have problem with importing TPySelector:

If you have any idea about this, please let me know.
Thanks

_ROOT Version: 6.22
Platform: Not Provided
Compiler: Not Provided


Hello,

Instead of using TPySelector, you can directly inherit from TSelector in Python. For example, this should work (iirc you’ll need ROOT 6.24):

import ROOT
from ROOT import TDSet, TProof

begin = False
terminate = False

class MySelector(ROOT.TSelector):
    def Begin(self, tree):
        global begin
        begin = True

    def Terminate(self):
        global terminate
        terminate = True

dataset = TDSet( 'TTree', 'events' )
dataset.Add('simpleTree.root')

proof = TProof.Open('')

mysel = MySelector()

dataset.Process(mysel)

print(begin, terminate)

As a replacement for Proof, we are now working on the distributed version of RDataFrame, a high-level interface to do analysis in ROOT. If you are interested in trying it out we could help you.

Thank you very much for your advices, it works.
I have a little experience working with RDataFrame. I will be happy to know how can I use RDataFrame instead of proof. Are there any documentations In this case?

Hello,

Perhaps you can start with this documentation:

https://root.cern/doc/master/classROOT_1_1RDataFrame.html

You will notice that the interface is quite different from Proof, but this interface is what we are trying to promote now from the ROOT team for analysis. The idea is that you write your analysis as a series of transformations on a dataset and some actions to get final results (e.g. a histogram). The same analysis code can then be automatically parallelized on the cores of your machine or also on multiple nodes (this is where it replaces Proof).

Hello,
thanks a lot.

Hello etejedor,
I wrote a code based on the code you sent, but the Slave Begin, Process, Init functions don’t work! Do you have any idea about this problem? Actually, only Begin and Terminate functions were called.

import ROOT
from ROOT import TDSet, TProof, TChain, TTree

begin = False
terminate = False
slavebegin = False
init = False
process = False
class MySelector(ROOT.TSelector):
    
    def Begin(self, tree):
        global begin
        begin = True
        self.Info('Begin', '-'*60)
        
    def Init(self, tree):
        global init
        init = True
        self.pt_lep_2b = ROOT.std.vector('float')()
        self.pt_lep_2b = 0
        self.Info('Init', '-'*30)
        self.fChain.SetBranchAddress("pt_lep_2b", self.pt_lep_2b)


    def SlaveBegin(self, tree): 
        global slavebegin
        slavebegin = True
        self.Info('SlaveBegin', '-'*60)
        self.eventsProcessed = 0
        self.h_pt = ROOT.TH1F("pt_lep_2b","pt_lep_2b",100,0,200)
        self.outList=self.GetOutputList()
        self.outList.Add(self.h_pt)

    def Process(self, entry):
        global process
        process = True
        print ("Process")
#         self.fChain.GetEntry(entry)
        self.GetTree().GetEntry(entry)
#         for position in range(0, self.pt_lep_2b.size()):
        for position in range(0, 2):
#             self.h_pt.Fill(self.pt_lep_2b[position], 1)
            self.h_pt.Fill(10, 1)

        return True

    def SlaveTerminate(self):
        print ("Slave Terminate")


    def Terminate(self):
        global terminate
        terminate = True
        self.file = ROOT.TFile("histos.root","RECREATE");

#		for var in range(0, len(self.variable)):
#         self.outList.FindObject("pt_lep_2b").Write()
        self.file.Close()

dataset = TDSet( 'TTree', 'analysis' )
dataset.Add('/home/sima/jupy/hep_ml/tree_data.root')

proof = TProof.Open('')
mysel = MySelector()

dataset.Process( mysel)

print(begin, terminate, slavebegin, init, process)

The output :

True True False False False
 
Info in <TProofLite::SetQueryRunning>: starting query: 3
Info in <TProofQueryResult::SetRunning>: nwrks: 4
Info in <TSelector::Begin>: ------------------------------------------------------------
Looking up for exact location of files: OK (1 files)                 
Looking up for exact location of files: OK (1 files)                 
Info in <TPacketizer::TPacketizer>: Initial number of workers: 4
Validating files: OK (1 files)                 
[TProof::Progress] Total 129983 events	|====================| 100.00 % [14442556.0 evts/s, 20.6 MB/s, time left: 0.0 s]
 Query processing time: 0.0 s
Lite-0: all output objects have been merged   

Thanks
Sima

Hello,

I am not sure why Proof is not invoking all the methods you define.

But if I do the same in C++, the behaviour is the same:

class MySelector : public TSelector {
        public:

    void Begin(TTree *tree) { cout << "BEGIN" << endl; }

    void Init(TTree *tree) { cout << "END" << endl; }

    void SlaveBegin(TTree *tree) { cout << "SLAVEBEGIN" << endl; }

    void Process(long entry) { cout << "PROCESS" << entry << endl; }

    void SlaveTerminate() { cout << "SLAVETERM" << endl; }

    void Terminate() { cout << "TERMINATE" << endl; }
};


void tselforum () {
    TDSet dataset("TTree", "events");
    dataset.Add("/tmp/simpleTree.root");

    auto proof = TProof::Open("");
    MySelector mysel;

    dataset.Process(&mysel);
}

Only Begin and Terminate are called. So this means it is not a Python problem.

Perhaps @ganis knows what’s missing?

I reiterate my offering to help you convert this to RDataFrame if that’s an option for you.

Cheers,
Enric

Hello etejedor,

Because the code I want to execute with proof is not the analysis code, but solves some equation s for tree events and also calls several external libraries, so I thought it could not be done with RDataFrame.
about c++ code and proof:
already I wrote my Analysis code based on the samples in c++ and ran with proof. I have never seen command “cout” or “printf” work in SlaveBegin , Process and SlaveTerminate functions, but as your code shows above works for Begin and Terminate functions. I don’t know why. But I always define objects like TH1F(histogram) in SlaveBegin and add to OutputList in c++ codes, if “SlaveTerminate” function works, the TH1F objects must be seen in OutputList in Terminate, or we can write it to a root file. I don’t know what is missed in python one.

Thanks for your great helps
Sima

For the output of “cout” and “printf”, try to see into:
find ${HOME}/.proof -name "*.log" -type f -print

Hello,

As for the example we are discussing here, the behaviour from Python and C++ is the same (i.e. only Begin and Terminate are called). I can’t find any trace of evidence that says otherwise in $HOME/.proof either, neither for C++ nor for Python. We’ll need @ganis to shed some light on this.

Hi,

This sounds weird. The printouts of SalveBegin, Process, SlaveTerminate should be in the workers logs, as those methods are executed in there. Can you locate the workers logs under $HOME/.proof ?

This said, with respect to

RDataFrame is much more flexible than PROOF so I would be surprised if you cannot solve your equation in RDF. If you (@Sima_Bashiri ) could post an example of what you ant to do, I am sure that Enric or the other ROOT developers can help you in the translation.

G Ganis

Hi,

I ran again the C++ version of the program (see above), which spawns 8 workers (I guess as many as logical cores in my laptop). This is the output of one of the workers (worker-0.6.log):

11:25:14  9023 Wrk-0.6 | Info in <TProofServLite::Setup>: fWorkDir: /home/etejedor/.proof
11:25:14  9023 Wrk-0.6 | Info in <TProofServLite::SetupCommon>:  0 global package directories registered
11:25:15  9023 Wrk-0.6 | Info in <TProofServLite::HandleProcess>: selector obj for 'TSelector' found
11:25:15  9023 Wrk-0.6 | Info in <TProofServLite::HandleProcess>: calling fPlayer->Process() with selector object: TSelector
11:25:15  9023 Wrk-0.6 | Info in <TProofPlayerSlave::AssertSelector>: Processing via TSelector object
11:25:15  9023 Wrk-0.6 | Info in <TEventIter::TEventIter>: fPackets list 'ProcessedPackets_0.6' created
11:25:15  9023 Wrk-0.6 | Info in <TProofPlayerSlave::Process>: save partial results? 0  per-packet? 0
11:25:15  9023 Wrk-0.6 | SvcMsg in <TProofPlayerSlave::CheckMemUsage>: Memory 369648 virtual 200144 resident event 0
11:25:15  9023 Wrk-0.6 | SvcMsg in <TProofPlayerSlave::CheckMemUsage>: Memory 369648 virtual 200144 resident event 0
11:25:15  9023 Wrk-0.6 | Error in <TProofServLite::HandleSocketInput>: retrieving message from input socket
11:25:15  9023 Wrk-0.6 | Info in <TProofServLite::Terminate>: starting session termination operations ...
11:25:15  9023 Wrk-0.6 | Info in <TProofServLite::Terminate>: data directory '/home/etejedor/.proof/data/0.6/0.6-laptop-x1-1639131914-9023' has been removed
Terminate: termination operations ended: quitting!

There’s an error there but I don’t see the message I print, though.

Can it be that the “MySelector” class is not propagated to slaves?

Your slave says “selector object: TSelector” and I would expect “selector object: MySelector” (or something similar).

Hello,
I put the intended code here and also the code is simplified so that it could be converted to RDF faster. The “nuSolutions.py” file is imported in the main code. The code is written in Python 2.

import nuSolutions as nu
import ROOT as r
import numpy as np
import math
f = r.TFile.Open("file.root")
tree = r.gDirectory.Get('')
t = f.Get("tree")
entries = t.GetEntriesFast()
for jentry in xrange(50):
    t.GetEntry(jentry)
    bList = []
    lList = []
    metxList = []
    metyList = []
    pt_b = t.pt_b_ee
    eta_b = t.eta_b_ee
    phi_b = t.phi_b_ee
    e_b = t.e_b_ee
    pt_lep = t.pt_lep_ee
    eta_lep = t.eta_lep_ee
    phi_lep = t.phi_lep_ee
    e_lep = t.e_lep_ee
    ch_lep = t.ch_lep_ee
    pt_met = t.pt_met_ee
    phi_met = t.phi_met_ee

    for pt,eta,phi,e,charge in map(None, pt_lep, eta_lep, phi_lep, e_lep, ch_lep):
        l = r.TLorentzVector()
        l.SetPtEtaPhiE(pt,eta,phi,e)
        ll= l,charge
        lList.append(ll)
    for pt,eta,phi,e in map(None, pt_b, eta_b, phi_b, e_b):
        b = r.TLorentzVector()
        b.SetPtEtaPhiE(pt,eta,phi,e)
        bList.append(b)
    for pt, phi in map(None, pt_met, phi_met):
        mett = r.TLorentzVector()
        mett.SetPtEtaPhiE(pt,0,phi,0)
        px = mett.Px()
        py = mett.Py()
        metxList.append(px)
        metyList.append(py)
    if(len(bList) >= 2):
        metx = metxList[0]
        mety = metyList[0]
        try:
            solver = nu.doubleNeutrinoSolutions((bList[0],bList[1]),(lList[0][0],lList[1][0]),(metx,mety),(nu.mW)**2,(nu.mT)**2)
            Neu = solver.nunu_s
            print (Neu)
        except :
            continue
            

nuSolutions.py (8.4 KB)
file.root (1.4 MB)

By looking at your code, one fundamental change is the removal of the event loop: you will no longer need to loop over the tree entries and get the branches from the iterated tree. Instead, the event loop will run in RDataFrame code, in C++, while your code only applies per-event operations to the tree dataset.

You can, for example, create new branches for lList, bList, metxList and metyList with Define. The type of those branches can be a vector (RVec). You can use later bList in a Filter to decide whether you do the doubleNeutrinoSolutions computation for a particular event depending on the length of bList.

Perhaps the trickiest part here is invoking the code that you have in nuSolutions.py, which is in Python. RDataFrame allows you to compile with Numba a given Python function (e.g. your doubleNeutrinoSolutions), see example:

https://root.cern.ch/doc/master/pyroot004__NumbaDeclare_8py.html

with some limitations. You would need to try whether that works fine, otherwise the function would need to be implemented in C++.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.