Unable to MergeTrees, CloneTree, or hadd cloned TTrees

I am attempting to “merge”/combine two trees. Both trees have been cloned from other trees. I have attempted hadd:

hadd all.root F1.root F13.root
hadd Target file: all.root
hadd Source file 1: F1.root
hadd Source file 2: F13.root
hadd Target path: all.root:/

 *** Break *** segmentation violation

===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x000000350c2ac65e in waitpid () from /lib64/libc.so.6
#1  0x000000350c23e609 in do_system () from /lib64/libc.so.6
#2  0x00007f6d869e5f88 in TUnixSystem::StackTrace() () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#3  0x00007f6d869e4e03 in TUnixSystem::DispatchSignals(ESignals) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#4  <signal handler called>
#5  0x00007f6d83e15d3b in TTree::CopyAddresses(TTree*, bool) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#6  0x00007f6d83e213cf in TTree::CloneTree(long long, char const*) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#7  0x00007f6d83e1b311 in TTree::Merge(TCollection*, TFileMergeInfo*) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#8  0x00007f6d855ed59a in TFileMerger::MergeRecursive(TDirectory*, TList*, int) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libRIO.so
#9  0x00007f6d855ebe5b in TFileMerger::PartialMerge(int) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libRIO.so
#10 0x0000000000402cac in main ()
===========================================================


The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007f6d83e15d3b in TTree::CopyAddresses(TTree*, bool) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#6  0x00007f6d83e213cf in TTree::CloneTree(long long, char const*) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#7  0x00007f6d83e1b311 in TTree::Merge(TCollection*, TFileMergeInfo*) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#8  0x00007f6d855ed59a in TFileMerger::MergeRecursive(TDirectory*, TList*, int) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libRIO.so
#9  0x00007f6d855ebe5b in TFileMerger::PartialMerge(int) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libRIO.so
#10 0x0000000000402cac in main ()
===========================================================


Fatal in <TFileMerger::RecursiveRemove>: Output file of the TFile Merger (targetting all.root) has been deleted (likely due to a TTree larger than 100Gb)
aborting
#0  0x000000350c2ac65e in waitpid () from /lib64/libc.so.6
#1  0x000000350c23e609 in do_system () from /lib64/libc.so.6
#2  0x00007f6d869e5f88 in TUnixSystem::StackTrace() () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#3  0x00007f6d86a078cf in DefaultErrorHandler(int, bool, char const*, char const*) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#4  0x00007f6d86a07141 in ErrorHandler () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#5  0x00007f6d86a17b12 in TObject::Fatal(char const*, char const*, ...) const () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#6  0x00007f6d869901bb in THashList::RecursiveRemove(TObject*) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#7  0x00007f6d86a18958 in TObject::~TObject() () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#8  0x00007f6d855df339 in TFile::~TFile() () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libRIO.so
#9  0x00007f6d869930a5 in TList::Delete(char const*) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#10 0x00007f6d86a2e6df in TROOT::~TROOT() () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#11 0x000000350c235ebd in __cxa_finalize () from /lib64/libc.so.6
#12 0x00007f6d86668fe6 in __do_global_dtors_aux () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#13 0x0000000000000000 in ?? ()
Aborted (core dumped)[/code]

(And by the way, these files do not sum to > 100G.)

I have attempted TChain::CloneTree:
[code]root [0] TChain tch("CCNuPionInc")
root [1] tch.AddFile("F1.root/CCNuPionInc")
(Int_t)1
root [2] tch.AddFile("F13.root/CCNuPionInc")
(Int_t)1
root [3] TFile* f = new TFile("out.root","RECREATE")
root [4] tch.CloneTree(-1,"fast")

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x000000350c2ac65e in waitpid () from /lib64/libc.so.6
#1  0x000000350c23e609 in do_system () from /lib64/libc.so.6
#2  0x00007f5102cbdf88 in TUnixSystem::StackTrace() () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#3  0x00007f5102cbce03 in TUnixSystem::DispatchSignals(ESignals) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#4  <signal handler called>
#5  0x00007f50f5e95d3b in TTree::CopyAddresses(TTree*, bool) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#6  0x00007f50f5ea13cf in TTree::CloneTree(long long, char const*) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#7  0x00007f50f5df957e in G__G__Tree_126_0_41(G__value*, char const*, G__param*, int) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#8  0x00007f5101d76209 in Cint::G__ExceptionWrapper(int (*)(G__value*, char const*, G__param*, int), G__value*, char*, G__param*, int) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#9  0x00007f5101c74f41 in G__execute_call () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#10 0x00007f5101c75da2 in G__call_cppfunc () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#11 0x00007f5101c58aa8 in G__interpret_func () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#12 0x00007f5101c46f1c in G__getfunction () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#13 0x00007f5101d2d74a in G__getstructmem(int, G__FastAllocString&, char*, int, char*, int*, G__var_array*, int) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#14 0x00007f5101d22887 in G__getvariable () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#15 0x00007f5101c2429d in G__getitem () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#16 0x00007f5101c29598 in G__getexpr () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#17 0x00007f5101ca2ad7 in G__exec_statement () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#18 0x00007f5101c1a0a1 in G__exec_tempfile_core () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#19 0x00007f5101c1a39e in G__exec_tempfile_fp () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#20 0x00007f5101cae929 in G__process_cmd () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCint.so
#21 0x00007f5102c8586f in TCint::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#22 0x00007f5102cc75aa in TApplication::ProcessLine(char const*, bool, int*) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#23 0x00007f510156148b in TRint::HandleTermInput() () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libRint.so
#24 0x00007f5102cbb69e in TUnixSystem::CheckDescriptors() () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#25 0x00007f5102cbbb63 in TUnixSystem::DispatchOneEvent(bool) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#26 0x00007f5102d1b566 in TSystem::InnerLoop() () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#27 0x00007f5102d1d5bb in TSystem::Run() () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#28 0x00007f5102cc584f in TApplication::Run(bool) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libCore.so
#29 0x00007f5101562a04 in TRint::Run(bool) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libRint.so
#30 0x00000000004010bc in main ()
===========================================================


The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007f50f5e95d3b in TTree::CopyAddresses(TTree*, bool) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#6  0x00007f50f5ea13cf in TTree::CloneTree(long long, char const*) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so[/code]

And I have made a variety of TTree:MergeTrees attempts:
[code]
TList* list = new TList;
TTree* T1=LoopChain(mparam1);
TTree* T13=LoopChain(mparam13);
list->Add(T1);
list->Add(T13);
list->Print();
cout << list << endl;
TTree* truthtree = TTree::MergeTrees(list);
truthtree->SetName("truth");
[/code]

The result of [code]list->Print()[/code] prints out the two TTrees, as I expect them to. They are not empty, they have all the branches, etc.

The result of the [code]cout << list << endl;[/code] produces a nonempty memory address.

It doesn't crash at [code]TTree::MergeTrees(list)[/code]. Instead (with dbg, because the backtrace by itself doesn't list the problem line) it crashes at [code]truthtree->SetName("truth")[/code]

[code]===========================================================
#5  0x00007f6641fa5d3b in TTree::CopyAddresses(TTree*, bool) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#6  0x00007f6641fb13cf in TTree::CloneTree(long long, char const*) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#7  0x00007f6641faa1b4 in TTree::MergeTrees(TList*, char const*) () from /grid/fermiapp/minerva/software_releases/v10r8p9/lcg/external/LCGCMT/LCGCMT_61/LCG_Settings/../../../ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#8  0x000000000041b24c in main (argc=1, argv=0x7ffc66d81df8) at Script.cxx:389
===========================================================[/code]

Note that in every case, we crash with these lines:

[code]===========================================================
#5  0x00007f294382dd3b in TTree::CopyAddresses(TTree*, bool) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
#6  0x00007f29438393cf in TTree::CloneTree(long long, char const*) () from /grid/fermiapp/minerva/software_releases/lcgcmake/lcg_61/external/ROOT/5.34.05/x86_64-slc6-gcc44-opt/lib/libTree.so
===========================================================[/code]

I don't have trouble merging trees with any of these methods with trees that were not cloned.

To produce the problematic cloned tree, I use, basically:
[code]TChain ch("CCNuPionInc");
ch.Add("somefile.root");
ch.SetBranchStatus("somebranches",1);
ch.SetBranchStatus("otherbranches",0);
TTree* temptruthtree = mch.CloneTree(0);
for(int i=0; i<500; ++i){
	ch.GetEntry(i);
	temptruthtree->Fill();
}

I’ll work on coming up with a simplified piece of code and input files to reproduce the problem. But in the meantime, does anything here look fishy?

Hi,

Indeed, I don’t see anything obviously ‘wrong’ in the description you give and a reproducer is needed to make further progress. Running the failing example through valgrind might give additional clues.

Cheers,
Philippe.