UPDATE TFile from PROOF output instead of RECREATE

[Sergio_Gonzalez]

Dear ROOT users,

I have a TSelector that can be processed either via TProof or just with TChain::Process. My goal is to merge some files where each contain a list of TTrees in a way that the output file contains the merged TTrees.

If I run with TChain::Process the output looks as expected. However, if I use TProof the output file just contains the last TTree as if it had been overwritting each time the file.

In SlaveBegin I have these lines:

      m_UsePROOF = m_mergeJob->GetUsePROOF();
      if (m_UsePROOF){
         m_prooffile = new TProofOutputFile(filename, "ML");
         m_prooffile->SetOutputFileName(TString(subdir+"/"+filename).Data());
         m_outfile = m_prooffile->OpenFile("UPDATE");
      }
      else {
         m_outfile = new TFile(TString(subdir+"/"+filename).Data(),"UPDATE");
      }

Where depending if it’s using PROOF or TChain it defines m_outfile differently, but in both cases with the option UPDATE, which I would expect to give the result I want.

Is there something else I need to do with PROOF so that the output just keeps being updated with each new merged tree?

Here are some lines I have in SlaveTerminate in case there is an issue there:

      m_outfile->Purge();
      m_outfile->Write();
      m_outfile->Close();

      if (m_UsePROOF){
         TDirectory *savedir = gDirectory;
         savedir->cd();
         m_prooffile->Print();
         fOutput->Add(m_prooffile);
      }

Thanks for the help and best regards,
Sergio

[eguiraud]

Hi,
if I understand your usecase correctly, with PROOF you would have multiple processes writing the same key to the same TFile at the same time. That’s probably not safe.

I think the typical pattern in these cases is to have each process write to a different file and merge the files (e.g. with hadd) after PROOF is done.

Cheers,
Enrico

[Sergio_Gonzalez]

Hi Enrico,
Thanks for your reply. Indeed you are right, it is probably not reasonable to do the update as I was proposing.

I thought that each process was working individually on separate files and then at the end they were merged and I wanted to know if the merged file could be updated. But if they are writing on the same TFile at the same time I guess is not possible.

Hadding the output would be a solution but that would mean that I need more available space during the merging… I think that I’ll just produce one file per tree and just keep them separately.

Thanks!
Sergio