TSelector in Python

[jstrube]

I have a Selector that is created from TTree::MakeSelector. I want to write a class in Python that inherits from this Selector and Process my tree with it in Python as well. (Actually I the derived tree is C++, but CINT doesn’t understand it, so I want to re-implement it in Python)
I have the following simple Program

from ROOT import gSystem
gSystem.Load('MySelector_C')
from ROOT import MySelector, TSelector

class MyPythonSelector(MySelector):
    def __init__(self):
        print dir(self)

    def SlaveBegin(self, tree):
        MySelector.SlaveBegin(self, tree)
        self.Init(tree)

    def Process(self, entry):
        print dir(self)
        return True

    def Terminate(self):
        pass

    def SlaveTerminate(self):
        pass

    def Init(self, tree):
        pass

I can instantiate the class, but tree->Process() dies with
(after it successfully completes the first iteration of Process())

 *** Break *** segmentation violation
 Generating stack trace...
 0x01022630 in PyROOT::TRootObjectConverter::SetArg(_object*, G__CallFunc*) + 0xa2 from /u1/local/root/lib/root/libPyROOT.so
 0x0102a1ed in PyROOT::TMethodHolder::SetMethodArgs(_object*) + 0xad from /u1/local/root/lib/root/libPyROOT.so
 0x0102a431 in PyROOT::TMethodHolder::operator()(PyROOT::ObjectProxy*, _object*, _object*) + 0x5b from /u1/local/root/lib/root/libPyROOT.so
 0x0102ad8f in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) + 0x173 from /u1/local/root/lib/root/libPyROOT.so
 0x0805bd60 in PyObject_Call + 0x1c from /u1/local/bin/python
 0x080a927f in <unknown> from /u1/local/bin/python
 0x080a8d37 in <unknown> from /u1/local/bin/python
 0x080a73b3 in PyEval_EvalFrame + 0x223b from /u1/local/bin/python
 0x080a7e6e in PyEval_EvalCodeEx + 0x40a from /u1/local/bin/python
 0x080aa2c4 in <unknown> from /u1/local/bin/python
 0x080a6977 in PyEval_EvalFrame + 0x17ff from /u1/local/bin/python
 0x080a7e6e in PyEval_EvalCodeEx + 0x40a from /u1/local/bin/python
 0x080a913d in <unknown> from /u1/local/bin/python
 0x080a8da7 in <unknown> from /u1/local/bin/python
 0x080a73b3 in PyEval_EvalFrame + 0x223b from /u1/local/bin/python
 0x080a7e6e in PyEval_EvalCodeEx + 0x40a from /u1/local/bin/python
 0x080a913d in <unknown> from /u1/local/bin/python
 0x080a8da7 in <unknown> from /u1/local/bin/python
 0x080a73b3 in PyEval_EvalFrame + 0x223b from /u1/local/bin/python
 0x080a91b9 in <unknown> from /u1/local/bin/python
 0x080a8da7 in <unknown> from /u1/local/bin/python
 0x080a73b3 in PyEval_EvalFrame + 0x223b from /u1/local/bin/python
 0x080a7e6e in PyEval_EvalCodeEx + 0x40a from /u1/local/bin/python
 0x080a913d in <unknown> from /u1/local/bin/python
 0x080a8da7 in <unknown> from /u1/local/bin/python
 0x080a73b3 in PyEval_EvalFrame + 0x223b from /u1/local/bin/python
 0x080a7e6e in PyEval_EvalCodeEx + 0x40a from /u1/local/bin/python
 0x080a913d in <unknown> from /u1/local/bin/python
 0x080a8da7 in <unknown> from /u1/local/bin/python
 0x080a73b3 in PyEval_EvalFrame + 0x223b from /u1/local/bin/python
 0x080a7e6e in PyEval_EvalCodeEx + 0x40a from /u1/local/bin/python
 0x080a913d in <unknown> from /u1/local/bin/python
 0x080a8da7 in <unknown> from /u1/local/bin/python
 0x080a73b3 in PyEval_EvalFrame + 0x223b from /u1/local/bin/python
 0x080a7e6e in PyEval_EvalCodeEx + 0x40a from /u1/local/bin/python
 0x080aa8ca in PyEval_EvalCode + 0x22 from /u1/local/bin/python
 0x080d22a5 in <unknown> from /u1/local/bin/python
 0x080d1b09 in PyRun_SimpleFileExFlags + 0x191 from /u1/local/bin/python
 0x080554cb in Py_Main + 0x56b from /u1/local/bin/python
 0x08054f5b in main + 0x17 from /u1/local/bin/python
 0x0012678a in __libc_start_main + 0xda from /lib/tls/libc.so.6
 0x08054eb9 in ldexp + 0x61 from /u1/local/bin/python
Aborted

Any help is greatly appreciated.
This is ROOT from CVS as of 12-Oct-2005

[wlav]

Jan,

an instance of a derived class implemented in python can not be passed as an argument where a C++ pointer/reference to the base is expected. The problem is that the C++ mechanism for calling overridden functions is an implementation detail (the vtable) that can not be accessed programmatically in a portable way.

There is a trick, which works by using a class written in C++, derived from the class you want to derive from in python, which forwards calls to the actual python instance. This proxy is then passed around through the pointer arguments, its overridden functions are called as appropriate (it’s all C++, after all), and it forwards these calls to python. I have no clear idea how to automate this procedure, though, as it requires a compilation step to be able to “modify” the vtable.

Best regards,
Wim

[jstrube]

Hi Wim.
thanks for your explanation.
I would be OK calling my already existing code from python, but CINT doesn’t understand it.
In principle I could just wrap it with any tool, right ?
So if I could wrap it with SWIG and then instantiate it in python and then pass that instance to Tree::Process, that should work in principle, right ?

Thanks,
Jan

[wlav]

Jan,

somehow I feel that I don’t completely understand the problem that you’re having. I’m going to make a few assumptions below, so please correct me when I write something wrong.

I would be OK calling my already existing code from python, but CINT doesn’t understand it.

So you have a piece of C++ code, that implements a MySelector, which derives from a TSelector in C++. However, you can’t use this code from PyROOT because CINT can’t parse it? Have you tried ACLiC? It is slower than CINT in the building step, but it generally gives better results.

In principle I could just wrap it with any tool, right ?

Unfortunately not, as each tool packs its C++ pointer to objects differently, and it isn’t always possible to write unpacking code for different tools without linking with these tools. I know interoperability with boost.python will be a hassle; I still have to look into SWIG. The easiest, of course, would be if tools could use the standard Python CObject interface, even as that would give some casting problems.

So if I could wrap it with SWIG and then instantiate it in python and then pass that instance to Tree::Process, that should work in principle, right ?

AFAIK, SWIG will give you the same problems when deriving a Python class from your C++ one and passing instances of that through base C++ pointers (boost.python has a half-automated solution: wrapper code that you write yourself which generates the proxy for forwarding).

It would be helpful if you can give some more detail as to what piece of code gives CINT problems? And what its importance is? I’m collecting information to see how much use there is for a direct Reflex - PyROOT connection, under the assumption that Reflex can produce dictionary information for a larger part of the C++ standard, like SWIG and Pyste do already today.

Cheers,
Wim

[jstrube]

Hi Wim,

I think you have fully understood my problem

Have you tried ACLiC

In fact, I have. Unfortunately, no go. ACLiC even tells me that I may have hit its limitations.
To be more precise about what I want to do:
I have a c++ class that inherits from TSelector. I have a couple of cut functions that I can put in a std::set and see if the given event passes or not.
I want to be able to quickly change the order of the cuts, so the respective cut function is passed as a pointer to member.
This allows me to arrange the cuts in the same order as my command-line arguments.
So far so good.
Now I have a c++ program that does all the interface to the user and the filesystem, like read the files from a certain location, parse the command-line arguments and is responsible for the output.
This is the part I would like to replace with python. Simple things like reading a directory are just a hassle in c++ compared to os.listdir().

I have found comments on the CINT pages that pointer to members are in principle not supported by either ACLiC or CINT and the solution to enable the preliminary support for it didn’t work for me.

I think your idea of a direct Reflex - PyROOT connection is a great idea ! I think it would be very helpful !

Cheers,
Jan

[wlav]

Jan,

still not totally sure that I understand it. Let’s take this one step at a time.

First, exposing a class to be used through a base pointer if that class itself can’t be parsed by CINT/ACLiC, can be done by writting a simple macro that contains:

and have the implementation of that function and the implementation of your MySelector in an ordinary shared/dynamic library. Load the library with gSystem->LoadLibrary(), create a dictionary for the above. All set.

Similarly, you can also register a python object the other way around and call it from C++.

The second step would be to expose the additional member functions of your MySelector class. But here I get confused: how do you expect to write your cut functions in python, given that they are pointer-to-member?
Or are you only selecting them in python? If so, any chance of doing that, too, through a simpler interface on top of your MySelector (cf. above)?

HTH,
Wim

[jstrube]

Hi Wim,

aha ! Now it gets interesting. Now I am not sure I understand what you mean

Load the library with gSystem->LoadLibrary()

You mean gSystem->Load(), right ?

create a dictionary for the above. All set.

This is the part I don’t understand. How can I create a dictionary ? I thought that’s what I need ACLiC for, no ? Loading the class that ACliC doesn’t understand with gSystem->Load() results in a dlopen error.

Similarly, you can also register a python object the other way around and call it from C++

In your tutorial you do that with TPython::LoadMacro(). Is that what you mean ?

[quote]The second step would be to expose the additional member functions of your MySelector class. But here I get confused: how do you expect to write your cut functions in python, given that they are pointer-to-member?
Or are you only selecting them in python? If so, any chance of doing that, too, through a simpler interface on top of your MySelector (cf. above)?
[/quote]
I would be OK just calling my cuts from python.
Otherwise I would have to somehow implement the whole Selector in Python. But I am sure that it is even easier to change the order of the cuts dynamically in python than it is in C++. I don’t want to do that, but now I am curious: With TPython::LoadMacro it would be possible to load anything that’s valid python into C++ ?

Thanks a lot for your explanations.
Cheers,
Jan

[wlav]

Jan,

let’s walk through a quick & dirty example (add the proper error handling, etc. yourself; sorry, little time and besides, writing this example out properly is a good exercise ):

MyClass.h[code]#include "TSelector.h"
class TTree;

class MyClass : public TSelector {
public:
virtual ~MyClass();
virtual void SlaveBegin( TTree* );

void RegisterCallBack( PyObject* );

private:
PyObject* m_callback;
};[/code]

MyClass.cxx:[code]#include “Python.h”
#include “MyClass.h”
#include

MyClass::~MyClass() {}

void MyClass::SlaveBegin( TTree* )
{
std::cout << “Hi from C++!” << std::endl;
PyObject* result = PyObject_CallFunction( m_callback, “” );
Py_XDECREF( result );
}

void MyClass::RegisterCallBack( PyObject* pyobject )
{
Py_INCREF( pyobject );
m_callback = pyobject;
}[/code]

MyHelpers.h:[code]class TSelector;
struct _object; typedef _object PyObject;

TSelector* CreateMyClass();
void RegisterCallBack( PyObject* );[/code]

MyHelpers.cxx:[code]#include “Python.h”
#include “MyClass.h”

static MyClass s_myClass;

TSelector* CreateMyClass() {
return &s_myClass;
}

void RegisterCallBack( PyObject* pyobject )
{
s_myClass.RegisterCallBack( pyobject );
}
[/code]

and finally, an example program, MyClass.py:[code]from ROOT import *
gSystem.Load( ‘libMyClass’ )
gROOT.LoadMacro( ‘MyHelpers.h+’ )

def SayHi():
print ‘Hi from python!’

s = CreateMyClass()
RegisterCallBack( SayHi );
s.SlaveBegin( NULL )[/code]

Build it, with something like this:
$ gcc --shared -I$ROOTSYS/include -I/usr/local/include/python2.4 MyClass.cxx MyHelpers.cxx -o libMyClass.so

and then run it like for example:

$ python MyClass.py

As you can see above, you can create and use a MyClass that doesn’t have an associated dictionary, as long as you are callling it only in C++ internally and/or refer outside of it only through its base class (TSelector), for which there is a dictionary.

Many variations on this theme are possible, of course. Is homework. Oh, and the TPython::LoadMacro() is still rather limited and loads python classes into CINT, not into C++ (for that, just use the Python C-API).

Sorry, gotta rush, more later if I’m still not quite understanding what you need.

HTH,
Wim

[jstrube]

Hi Wim,

this is good stuff !
Thanks a lot for the explanation. I had never looked at the C Extensions before. I should be able to get something going with these snippets.

Thanks again !
Cheers,
Jan