Hi, I have recently learned about many different options for C++ extensions in Python, where one can write a C++ function/class which is said to be faster, but then use it within Python as a regular function or object. Some of the options are: SWIG, boost::python, SIP, Py++, Pyplusplus, probably others…
In the context of trying to write C++/ROOT functions which are callable from a PyROOT program, with ROOT objects, which of these options would be the most appropriate?
Example use case: I have some Python program involving ROOT objects. I would like to perform some operation on each of these objects, but it is very slow in Python/PyROOT. I would then write the operation as a C++ function (faster in principle), but then I would want to import these into my Python program.
I’m asking before I try diving too deep into a particular framework.
I’d favor just creating a dictionary for them, if only b/c it’d allow use from both CPython and PyPy (as well as CINT and therefore calls to such things as TTree::Draw etc.). But then again, I’m biased.
In the past, I’ve used both SWIG and boost::python. They’ve both been around a long time (as has SIP) and aren’t dead yet, although boost:python has not seen any serious development since 2004 or so. The other two you mention are more recent, but rely on gccxml. My preference today would go out to SWIG, as it is more actively developed/maintained than boost::python, although the latter has an easier C++11 story. To call with (Py)ROOT objects, though, you’d have to extract and rebind the pointers (easiest with ROOT.AsCObject/BindObject and the other framework’s equivalent). See $ROOTSYS/tutorials/pyroot/qtexample.py for such an example using PyQT (i.e. SIP).
I apologize for opening up an old topic, but my question seems most relevant to this theme.
We are currently using ROOT’s dictionary generation of our software package to enable scripting in python. However, we would like to extend this to enable classes written in python and deriving from our C++ base classes to have their python functions called appropriately in C++, for example, in a processing framework. A good example of the functionality we would like to achieve is that of TPySelector (http://root.cern.ch/root/html/TPySelector.html), or more generally SWIG director classes http://www.swig.org/Doc1.3/Python.html#Python_nn34 that enable overloaded virtual base class functions in python.
# in python
class pyObject(MyObject):
def Print(self):
print "Py Derived"
anobj = pyObject()
printMyObject(anobj)
and have this print out:
Py Derived
Since we are only expecting to do this with a few classes, I had considered mirroring what TPySelector does. The problem is that it relies upon a number of internal PyROOT functions that are not exposed to the user in a normal ROOT build. (Are there ways to expose the internals to PyROOT? This would be nice…) A solution with SWIG (or something similar) would be nice because it wouldn’t require maintaining a separate class for each base class from which we wish to derive in python, but it seems difficult to get this to play well with PyROOT.
which internal functions are you missing? Several functions, such as extracting the ROOT object as a void* and rebinding are made available in the TPython interface (see $ROOTSYS/include/TPython). TPySelector does not go through that interface, but they are effectively the same methods. If any are missing, they can be added: TPython is grown organically on an as-needed basis.
As a general solution: that is for PyCling. I once prototyped something for CINT, using ACLiC to generate the intermediate class (i.e. the TPySelector equivalent), so I know it works, but I never put it in PyROOT proper. You can have the prototype code, if you’re interested, but it’s not production-level. In particular, it needs additional code to deal with updates and intermediate files. With Cling, I don’t need to deal with intermediate files, making for a much cleaner solution.
I’ve attached an example of what I would like to do, using the functions I would expect to use. (I haven’t put some checks in to ensure the class is being called in a python script, etc.) You may untar and type ‘make’ to give the example. In particular, we would need:
PyROOT::TMemoryRegulator::RetrieveObject, to get the python proxy object
PyROOT::MethodProxy_CheckExact, to ensure that a python function is returned.
In principle, accessing the underlying Python.h API would also be necessary for us and so it would be nice to be able to find out exactly which python was used to build the ROOT installation, though I think I can get at this sufficiently via the root-config --python-version command. Is there any easier way?
Sure, I’d love to see this, it would be very helpful.
Another set of functions that would be necessary (for us) would be access to the buffer wrappings in TPyBufferFactory as we deal with raw buffers in our processing framework. Should I open a feature request for these additions to TPython?
For the buffers, there are better alternatives in the memory view structs of python. At least, if you want to use numpy on the other side (is one more large TODO item to remove TBufferFactory, which doesn’t work as it should in p3 anyway).
attached is a revived version of the auto-generated proxies as I did some time ago. This’ll be so much easier with Cling. To use, make sure that the path to python is properly set in Derived.py, then simply run the script.
As for your code: I don’t think you need the TMemoryRegulator (that’s a TPySelector artefact b/c of creation order in the workers). An alternative is to simply pass “self” into the init of the base or through a set_proxy() as done in the automatic case.
The checking of the method type is to prevent circular calls. This is only necessary if the user may decide to NOT override the method (as is the case for TPySelector). The auto-generation works around this by not generating stubs for methods that are not overridden.
I thought about doing this, however, it seems I cannot pass the memory view object “back” into functions taking arrays. The conversion function, in particular PyROOT::Utility::GetBuffer does not handle this.
As an aside, I believe this should work if pyROOT was internally using memory views, but it is limited from doing so before python version 3. I believe this might be too strict, as memory views have been back-ported to 2.7. Would a configure check to see if the API exists, (e.g. check for PyMemoryView_FromBuffer) work?
yes, that’s on my TODO list. But unfortunately for a long time already. So are you iterating over the buffer or only passing it back to C++? There’s also the CObject API if it only gets passed around.
I’ve checked in methods (in v5-34-00-patches and trunk) into TPython to do the MethodProxy checking, as for that at least I see no other good way.
