How to add root files when the target root file is larger than 100 GB

ROOT
1

Hello experts,
I am trying to add many root files having the same tree name. I am simply using the “hadd” option to add all of them. When the target root file is less than 100 GB, I see no problem. But when the target root file is more than 100 GB, I see that the root is crashing. For example, this is one such crash message:

Fill: Switching to new file: ./user.jmaurer.ABp102_364250.Sherpa_222_NNPDF30NNLO_llll.e5894_s3126_r10724_p3990_output_merged_1.root
Fatal in <TFileMerger::RecursiveRemove>: Output file of the TFile Merger (targeting ./user.jmaurer.ABp102_364250.Sherpa_222_NNPDF30NNLO_llll.e5894_s3126_r10724_p3990_output_merged.root) has been deleted (likely due to a TTree larger than 100Gb)
aborting
#0  0x00007f1872e4941c in waitpid () from /lib64/libc.so.6
#1  0x00007f1872dc6f12 in do_system () from /lib64/libc.so.6
#2  0x00007f1873f5f64d in TUnixSystem::StackTrace() () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libCore.so
#3  0x00007f1873e377b9 in DefaultErrorHandler(int, bool, char const*, char const*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libCore.so
#4  0x00007f1873e3723a in ErrorHandler () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libCore.so
#5  0x00007f1873e4df91 in TObject::Fatal(char const*, char const*, ...) const () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libCore.so
#6  0x00007f1873ebc1fb in THashList::RecursiveRemove(TObject*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libCore.so
#7  0x00007f1873d87f33 in TROOT::RecursiveRemove(TObject*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libCore.so
#8  0x00007f1873e4c2fb in TNamed::~TNamed() () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libCore.so
#9  0x00007f1874663859 in TFile::~TFile() [clone .localalias.196] () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libRIO.so
#10 0x00007f187662260e in TTree::ChangeFile(TFile*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libTree.so
#11 0x00007f1876629540 in TTree::CopyEntries(TTree*, long long, char const*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libTree.so
#12 0x00007f1876628a40 in TTree::Merge(TCollection*, TFileMergeInfo*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libTree.so
#13 0x00007f187464fd5b in TFileMerger::MergeRecursive(TDirectory*, TList*, int) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libRIO.so
#14 0x00007f187464e970 in TFileMerger::PartialMerge(int) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_95/ROOT/6.16.00/x86_64-slc6-gcc62-opt/lib/libRIO.so
#15 0x0000000000407a63 in main::{lambda(TFileMerger&, int, int)#2}::operator()(TFileMerger&, int, int) const [clone .constprop.171] ()
#16 0x0000000000405338 in main ()

What is the best practice to add root files when the target root file is so large in size? Is there a way to increase the max limit in hadd?

Cheers,
Arka

_ROOT Version: root 6.16.00
_Platform: Centos7
_Compiler: x86_64-slc6-gcc62-opt

2

Hello Arka,
TTree::SetMaxTreeSize should help you with that. Take a look at Root 6.04.14 hadd 100Gb and rootlogon and Hadd: 100Gb TTree Limit/compression levels for extra context and let me know if it helps.
Cheers,
Vincenzo

3

Hi Vincenzo,
I tried the solution mentioned by you in the first link,
but I am getting this error [1] while using the command:

LD_PRELOAD=startup_C.so hadd  output.root input_*

May I know what I am doing wrong here?
Cheers,
Arka

[1]

ERROR: ld.so: object 'startup_C.so' from LD_PRELOAD cannot be preloaded: ignored.
4

For the record, this is reported as Loading....

but I am getting this error [1] while using the command:

You need to create the file startup.C and compile it and the result (startup_C.so need to be in a directory listed in your LD_LIBRARY_PATH)

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.