I am merging a large number of ROOT files using a variation of copytree3.C with ROOT 6.08.06. I have around 300 ROOT files to be merged. The script reads the files, stores them in a vector and passes the vector to chain.Add(vector).
The process finishes and outputs a single ROOT file but throws some errors and warnings along the way:
Info in TBranchElement::InitializeOffsets: TTree created with an older schema, some data might not be copied in ‘slow-cloning’ mode; fast-cloning should have the correct result. ‘Tau3’ is missing when constructing the branch ‘Jet’.
Info in TBranchElement::InitializeOffsets: TTree created with an older schema, some data might not be copied in ‘slow-cloning’ mode; fast-cloning should have the correct result. ‘NSubJets’ is missing when constructing the branch ‘Jet’.
Info in TBranchElement::InitializeOffsets: TTree created with an older schema, some data might not be copied in ‘slow-cloning’ mode; fast-cloning should have the correct result. ‘MassDrop’ is missing when constructing the branch ‘Jet’.
Info in TBranchElement::InitializeOffsets: TTree created with an older schema, some data might not be copied in ‘slow-cloning’ mode; fast-cloning should have the correct result. ‘TrimmedMass’ is missing when constructing the branch ‘Jet’.
Warning in TStreamerInfo::BuildCheck:
The StreamerInfo for version 1 of class MissingET read from the file BB-4p-300-700-v1510_14TEV_140PileUp_129341946.root
has a different checksum than the previously loaded StreamerInfo.
Reading objects of type MissingET from the file BB-4p-300-700-v1510_14TEV_140PileUp_129341946.root
(and potentially other files) might not work correctly.
Most likely the version number of the class was not properly
updated [See ClassDef(MissingET,1)].
Warning in TStreamerInfo::CompareContent: The following data member of
the on-file layout version 1 of class ‘MissingET’ differs from
the in-memory layout version 1:
float Phi; //
vs
float Eta; //
Warning in TStreamerInfo::CompareContent: The following data member of
the in-memory layout version 1 of class ‘MissingET’ is missing from
the on-file layout version 1:
float Phi; //
Error in TExMap::Remove: key 106374640 not found at 37
Error in TExMap::Remove: key 48775328 not found at 58
Error in TExMap::Remove: key 175250800 not found at 7
Error in TExMap::Remove: key 175250800 not found at 7
Error in TExMap::Remove: key 175250800 not found at 7
Error in TExMap::Remove: key 175250800 not found at 7
Error in TExMap::Remove: key 109814480 not found at 74
Error in TExMap::Remove: key 48660256 not found at 25
This is just a sample of them. I am not sure if this error is something to be worried about. Also, is it related to the fact that those ROOT files were created using an older schema as the warning states?
Your input his highly appreciated.
The message indicates that the files you are using have been written with different schema for the class MissingET and that when the change was made, the author ‘forgot’ to increase the version number. Because of this you might not be able to read those files in the same ROOT session and may not be able to merge them. (If I interpreter correctly the message some files have MissingET::Eta and some do not, ROOT can not guess what should be the value to use when it is missing).
Similarly the files seems to have various schema for the class Jet leading to similar problem.
Note: did you try using the ‘hadd’ utility rather than adapting copytree3.C? In particular it would default to using the fast cloning mode (i.e. be faster) which would also likely solve the problem for Jet.
Thanks for your answer.
The macro doesn’t only merge the files but applies a cut on MET before doing so in order to reduce the size of the resulting file. If I use hadd on the 308 files without the cut, it crashes since it exceeds 100 GB:
hadd Target path: result.root:/
Fill: Switching to new file: result_1.root
Fatal in <TFileMerger::RecursiveRemove>: Output file of the TFile Merger (targeting result.root) has been deleted (likely due to a TTree larger than 100Gb)
aborting
#0 0x00007ff7de33b98c in __libc_waitpid (pid=8949, stat_loc=stat_loc
entry=0x7fff39d76a10, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
#1 0x00007ff7de2c0592 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
#2 0x00007ff7df3133f3 in TUnixSystem::StackTrace() () from /home/amin/Documents/root-6.08.06/lib/libCore.so
#3 0x00007ff7df255491 in DefaultErrorHandler(int, bool, char const*, char const*) () from /home/amin/Documents/root-6.08.06/lib/libCore.so
#4 0x00007ff7df254eda in ErrorHandler () from /home/amin/Documents/root-6.08.06/lib/libCore.so
#5 0x00007ff7df265c24 in TObject::Fatal(char const*, char const*, ...) const () from /home/amin/Documents/root-6.08.06/lib/libCore.so
#6 0x00007ff7df2c3c8b in THashList::RecursiveRemove(TObject*) () from /home/amin/Documents/root-6.08.06/lib/libCore.so
#7 0x00007ff7df265fc8 in TObject::~TObject() () from /home/amin/Documents/root-6.08.06/lib/libCore.so
#8 0x00007ff7df750369 in TFile::~TFile() () from /home/amin/Documents/root-6.08.06/lib/libRIO.so
#9 0x00007ff7ca264852 in TTree::ChangeFile(TFile*) () from /home/amin/Documents/root-6.08.06/lib/libTree.so
#10 0x00007ff7ca26a1e9 in TTree::CopyEntries(TTree*, long long, char const*) () from /home/amin/Documents/root-6.08.06/lib/libTree.so
#11 0x00007ff7ca25ffe3 in TTree::Merge(TCollection*, TFileMergeInfo*) () from /home/amin/Documents/root-6.08.06/lib/libTree.so
#12 0x00007ff7ca280f6c in ROOT::merge_TTree(void*, TCollection*, TFileMergeInfo*) () from /home/amin/Documents/root-6.08.06/lib/libTree.so
#13 0x00007ff7df75368c in TFileMerger::MergeRecursive(TDirectory*, TList*, int) () from /home/amin/Documents/root-6.08.06/lib/libRIO.so
#14 0x00007ff7df7548f1 in TFileMerger::PartialMerge(int) () from /home/amin/Documents/root-6.08.06/lib/libRIO.so
#15 0x0000000000402ba8 in main ()
Aborted (core dumped)
Is there a way to use hadd in a macro after the cuts have been applied?
hadd is a small shell around the class TFileMerger (see main/src/hadd.cxx) but this would not let you add a selection.
Either way, since you need to apply a selection and want to skim, you will need to read the content of all the branches (i.e. will not be able to use fast cloning). rooteventselector is likely to encounter the same problem as you did.
You have two alternative.
One is to segregate the files by categories, putting together the files that have the same schema, and merging only the files of the same category. You will then end-up with a few files that you can ‘attach’ toghether as a TChain (rather than a physical merge).
One is to load a library that handle the schema evolution (i.e. properly define the classes and how to read the old schemas and/or set default for the missing members).
Note that both those solutions might still have problem with the MissingET problems where two schema where set to have the same version number (bug at writing time). You might not be able to read those files in the same ROOT sessions (i.e. you may have to run your analysis in two separate process and then join/hadd the resulting histograms).
Thanks for the suggestions.
Going with your second one, I found that those files were made using Delphes 3.0.9. By loading a library that handles schema evolution or recognizes old schema, do you mean using the old version of Delphes?
I don’t think they do.
Those files are the SNOWMASS backgrounds and reproducing them again using the new Delphes version is not an easy thing.
Anyway, thanks for the help.
Those files are the SNOWMASS backgrounds and reproducing them again using the new Delphes version is not an easy thing.
Not what I had in mind. The question was whether the newer version of the Delphes software contained code (and/or ROOT customization) to allow reading older files.
Thanks for the clarification. I will direct the question to the Delphes team as I am unaware if such a tool exists in their new version.
Thanks again for pointing this and for all the help.