Saving TTree inside TList object instead of TDirectory

ROOT
1

I am currently working on merging 54 root files, each containing a simple structure: they have a directory that contains a TList that, in turn, holds a few histograms and a TTree object. Each file is 50-250MB big (with a 1.5 compression factor), and trying to hadd them I have an excessive memory usage.

I tried writing my own code to do the merging, reading each file and setting the output myself. The problem seems to be that, if I ask the TTree not to be stored in the base directory, but try to only place it inside the list by doing ::SetDirectory(0) and then add to list, it is a memory-resident tree which, when merging the 54 files, uses up all my memory (it’s around 8GB of uncompressed data) before being saved at the end of the macro… Maybe that’s why hadd also uses too much memory? I don’t know…

I tried creating a buffer file just for the tree and adding it to the list that then gets saved to the merged file, but then the tree is corrupt (the data got buffered - and saved - only in the buffer…). Finally, I tried keeping it in the base directory and adding it to the TList, but if the TTree object is acquired through the TList by Tree* fTree = (TTree*) list->FindObject(“fTree”) then it seems its entries are all zero for some reason… Weirdly, when opening with the Browser it works and the data is all there…

Is there some way to accomplish saving the TTree solely inside a TList with proper disk caching instead of keeping the whole thing in active memory? Something analogous to SetDirectory( )?

The reason for me to keep everything inside a TList is that this way I keep backwards compatibility with other code, too… If I get rid of the TList I have to rewrite a number of other things, too… But in the end I may just have to do that I guess.

Thanks you very much for any help!!

Snippet of code that creates the merged file:

[code] TFile* fileFusion = TFile::Open(MergedName, “RECREATE”);
if (!fileFusion || !fileFusion->IsOpen())
cout<<“Couldn’t open file!”<<endl;
fileFusion->cd();
TDirectoryFile *lDirectory = new TDirectoryFile(“PWG2CheckPerformanceLambda_PP_MC”,“PWG2CheckPerformanceLambda_PP_MC”);
lDirectory->cd();

TList* fListHistVZero = new TList();
fListHistVZero->SetOwner();
TTree *fTree = new TTree(“fTree”,“CascadeCandidates”);
fListHistVZero->Add(fTree);[/code]

2

[quote]and trying to hadd them I have an excessive memory usage. [/quote]You could try to merge them in smaller batch (for example using the -n option of hadd … however you will need to trunk and/or top of the v5.32 patch branch to fix an issue with -n and TList).

[quote]I tried writing my own code to do the merging, reading each file and setting the output myself. [/quote]In first approximation, I would say this merging code should follow strictly the way the original file where written (i.e. the code that wrote them somehow manage to have a disk-based TTree stored in the TList).

[quote]Finally, I tried keeping it in the base directory and adding it to the TList, but if the TTree object is acquired through the TList by Tree* fTree = (TTree*) list->FindObject(“fTree”) then it seems its entries are all zero for some reason.[/quote]Most likely it is because the list was stored before the TTree was completely filled.

Cheers,
Philippe.

3

Hello Philippe,

Thanks a lot for your input! I did not get much success with hadd -n either, so I ended up deciding to change the other pieces of code that I didn’t want to change and just store the tree in the base directory.

I suspect that, when it was within the TList object, it was always within memory - but it was not a problem until merging, where the amount of memory needed exceeded available memory. So now I think things should work nicely… And even if I write my own merging code, saving the TTree directly to a directory will surely activate caching… and problem solved. :smiley:

Thanks again!