Error while merging large root files

ROOT
1

Dear ROOT experts,

I am experiencin an issue with merging nearly 100 root files of about 70 Gega combined. I use the following script for merging [1]. But when my Output reaches 50 Gega it crashes with this error[2]

Any ideas ?
Thanks in advance

[1]

import ROOT
import os, sys

print 'Merging %s' % sys.argv[1]

print "Max tree size",ROOT.TTree.GetMaxTreeSize()
ROOT.TTree.SetMaxTreeSize(200000000000) # 200 Gb
print "Updated tree size",ROOT.TTree.GetMaxTreeSize()

rm = ROOT.TFileMerger(False)
rm.SetFastMethod(True)


path = 'mypath'
file_output = '%s.root' % sys.argv[1]
file_list = []
for path, dirs, files in os.walk(path):
  for filename in files:
    if ('%s_part' % sys.argv[1]) in filename: file_list.append(path+filename)

print "Input file list:",file_list
print "Output file:",file_output

for F in file_list:

    print "Adding ->",F
    rm.AddFile(F)

rm.OutputFile(file_output)
rm.Merge()
```========================================================================
[2]
SysError in <TFile::WriteBuffer>: error writing to file mc16e.root (-1) (File too large)
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
TTree::CopyEntries:0: RuntimeWarning: The output TTree (CollectionTree) must be associated with a writable file (mc16e.root).
Error in <TFileMerger::Merge>: error during merge of your ROOT files
2

Are you writing to CERNBox? There’s a 50 GB file size limit there.

3

Yes, I am using my eos work directory. I still have 500 Giga
Why 50 Giga ?

Best,
Yassine

4

Is there any way to solve that ? I have only eos space

5

Use a TChain, instead of a huge single file with a TTree.

6

Hi again,

the problem still persists even if i use TChain. here is the script that i used :

import ROOT
import os, sys

##print 'Merging %s' % sys.argv[1]

print "Max tree size",ROOT.TTree.GetMaxTreeSize()
ROOT.TTree.SetMaxTreeSize(200000000000) # 200 Gb
print "Updated tree size",ROOT.TTree.GetMaxTreeSize()
file_list = []
ch = ROOT.TChain("CollectionTree")

for file in file_list:
  ch.Add(file)
  print(file)

ch.Merge("all.root")
7

Not to merge it (which would of course run into the same file size limit).

You should use a TChain to read the tree out of multiple files instead of merging into one.

If you’re using RDataFrame, you can just give it a vector (or glob) of filenames to open. Similarly, if you’re using uproot, you can use uproot.iterate with a list (or glob) of filenames.

8

I’m not sure if that does work for me. In order to be able to use my code which computes some scale factors. My file has to be merged.

My code takes as input only one file contains data and the other contains MC. I was able to merge data because it wasn’t so large like MC.

9

The problem was solved by increasing the the file max size to 100G.

Best,

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.