Hadd: 100Gb TTree Limit/compression levels

ROOT
1

I want to merge 30 ~15Gb files containing an TNtuple using hadd, but get the following error:

Fatal in <TFileMerger::RecursiveRemove>: Output file of the TFile Merger (targeting xxx.root) has been deleted (likely due to a TTree larger than 100Gb)
Indeed, the target file is 100Gb when the error occurs. The files are generated from simulation data that writes an entry every break point.

I have 512Gb of RAM on my server, and I don’t want to slow down hadd by using more compression if I can help it. So my question is: Is there an easy way to get around the apparent 100Gb limit?

On the other hand, the ntuples are written with the default level one compression. Perhaps I could write the files with higher compression and use the same compression level in hadd (I see that using different compression levels slows hadd-ing quite a bit)?

[[[Edit: Just realized: I am using Geant4’s G4RootAnalysisManager (v9.6) in my simulation code. This class does not expose the compression level functionality, so it seems that I am stuck with writing the files using the default compression level.]]]

Thoughts and answers would be much appreciated.

2

Hi,

One thing you can do is to not merge the file but instead use them as a group by using a TChain (you could even write yet another file that contains just the TChain description/list of files). You can play with the compression factor in hadd by adding the new compression level after the -f (-f0 for example). Another alternative is to write a macro replicating the hadd behavior (see main/src/hadd.cxx and the file TFileMerger) and set TTree::SetMaxTreeSize to a large value.

Cheers,
Philippe.