Improve speed of csv conversion

I have to convert a 10GB CSV file into TTree on a 4core machine with 4GB memory.
This is the relevant code :

TTree *ptreeData = new TTree("treeData","ntuple containing the simulation result data");
Long64_t nlines = ptreeData->ReadFile(csvInput,branch_desc);

I noticed that when I ran the program, my processor/memory/disk usage is low. According to my calculus, it will spend 37hours for conversion.
[ul][li]Should I change buffer usage ? How ? [/li]
[li]Should I use Proof Lite ? How ?[/li]
[li]Is there a way to view a percentage of conversion on runtime ?

Any suggestion will be greatly appreciated.


Please. :cry:


Sorry for the late reply.
37 hours to fill a tree out of 10GB seems really a lot.
Could you please post all your macro?

G. Ganis

Ps: there is no PROOF packetizer to process lines in text files, i.e. it cannot be done straightly with PROOF; though a workaround may, probably, be found.

Hi Ganis,
Thanks for your answer but the problem is solved. I don’t know how nor why but it works now. Occasionnally, is it possible to run TMultiLayerPerceptorn::Train() on Proof Lite ?


Not that I know. One should check with the author if it can be modularized so that the main loops can be parallelized. But from a quick look at the code it looks like that requires a deep restructuring.

G. Ganis