Speed up/optimize csv to root conversion process

[Danny_Lee]

for (Int_t SubRun=1;SubRun<(No+1);SubRun++){
      TString froot = Form("Raw/Run%04d/Data_root/Run%04d_%04d.root",DataFile,DataFile,SubRun);
      TString fcsv = Form("Raw/Run%04d/Data_csv/Run%04d_%04d.csv",DataFile,DataFile,SubRun);
      
      TFile *f = new TFile(froot,"RECREATE");
      TTree *tree = new TTree("ntuple","data from csv file");

      tree->ReadFile(fcsv,"Time/F:CH11/F:CH12/F:CH21/F:CH22/F:CH31/F:CH32/F:CH41/F:CH42/F",',');
      f->Write();
   } 

Hi guys, The above is a snippet of my code where I do the conversion of csv files into root. I can do what I want here which is convert it from csv to root.

However, one issue I noticed was the speed of how long this process takes. On average each csv file has about 7millions entries which takes about 1-2minute to do so. With this where I may be having up 100 files, this could take up a few hours.

I know the process is already kind of fast and maybe too much to ask for. But I would definitely would love to speed this up. If anyone has some suggestion on how I might improve this would love to hear your opinion.

Danny

[Wile_E_Coyote]

Try to run multiple ROOT processes simultaneously (e.g. as many as you have CPUs or even twice that many), one process per each csv file.

[Danny_Lee]

I’m supposed that I can use the PROOF feature. The machine I’m testing the code on has 2 cores,4threads. I’m planning to use my code to do this csv root conversion in another machine which is much better than the computer im using right now.

Am I doing this right by adding

TProof::Open("workers=2");

before the start of the loop where i convert the csv to root?