Performance best practices

Hi fellow PyROOT’ers

Now that data is tickling in for most of us, I would be nice to hear how you tackle the growing datasets in PyROOT/Python.

As I guess we all share the preference of Python over C++ in this forum, I also suppose that we have decided to take the sacrifice of performance over easier programming, but optimizing can take many forms, and compiled languages are just one of many options.

I have been looking at SciPy and NumPy for most of my computing needs outside HEP, but because of the heavy reliance on ROOT, its is not trivial to unfold and optimize ones code on the algorithmic level, but I might be wrong, have anyone succeded in applying any of the methods described in this link: scipy.org/PerformancePython

My own attempt at a scalable system started out by simply wanting to use all the cores on my workstation, it later developed into using all the cores on all the machines in our office and later a small cluster. The framework that came out of this effort is available here: gluino.com and only requires ROOT and Python 2.6 or higher. The system is “PROOF-like” in the sense that it uses the fact that our data is independent on the event level, but much easier to set up (you just run a python script, and puf you have another worker available). But even this system is slow per iteration.

So a few questions, have you used PROOF in PyROOT, was it worth the trouble?
Are you using other methods like embedded compiled code, can you feel the difference? - how does ROOT behave in this regard?

And a special question to Wim, I heard a small bird singing that you are interested in PyPy and LLVM, can you see a performance effort somewhere on the horizon?

/Morten

Morten,

as chance would have it, there’s a Sprint at CERN this very week to address that.

Cheers,
Wim

Hi,

of course you may do with your computing resources whatever you wish :slight_smile: but it is err, simply “counterintuitive” to use interpreters for compute intense tasks (says the guy maintaining ROOT’s C++ interpreter). After all, that is exactly what compiled languages were made for! You are now arguing “but if I use three horses it’s really much faster than with one horse, and these horseshoes are pretty awesome, too!”. “Well, enjoy”, says the rest of the world sitting in their Ferraris :slight_smile:

Concerning the possibilities of optimizing python code (also with LLVM) I can only warmly recommend you to watch the presentation by Colin Winter, one of the Unladen Swallow devs, at the LLVM dev meeting last year: http://llvm.org/devmtg/2009-10/Winter_UnladenSwallowLLVM-400kbps.mov.

Cheers, Axel.

Hi Axel,

[quote=“Axel”]
of course you may do with your computing resources whatever you wish :slight_smile: but it is err, simply “counterintuitive” to use interpreters for compute intense tasks (says the guy maintaining ROOT’s C++ interpreter).[/quote]
I agree, but its still a cost/benefit problem, Python allows me to program in near pseudo code, and that makes me creative, in C++ I first have to deconstruct my thoughts into a series of hacks and counter logic, and normally I would “lose” most of my ideas after a few hours of wrestling with a problem. I know that the counter argument is that I should just prototype in Python and re-implement in C++ when I know what I want, and that is true.

At the fear of becoming philosophical (and moving this thread away from its purpose) I would say that the one lesson learned from moving the HEP community to C++ is that most of us are crappy programmers, we are after all first and foremost physicists, and even though C++ is fast and very expressive most of us produce utter crap because our focus is elsewhere. For me and many I have talked to about this topic, Python because of its “semi-official” status at the LHC experiments, is the savior, we can do physics in this language even though its just a short lasting dream, and Fortran 95/2003 would have been a better fit today.

But most of this is just ranting, and the very short answer is that for some prototype work, one computer is not fast enough, and better language alternatives are too time consuming.

I have seen it a few times, but I think PyPy have made some real progress since back then.