Fast merge & automatic file overflow

ROOT
1

Hi,

I suspect that when I am merging fast (TChain::Merge(outputFileName,“fast”)), the automatic file overflow (defined through a call to TTree:SetMaxTreeSize(…)) is not performed.

Somebody can confirm ?

Thanks.
David.

2

This is correct.

Cheers,
Philippe.

3

Is it a design limitation which cannot be removed ?

4

They are some issues. Since the copy is done basket by basket (instead of entry by entries), I think if this was implemented, the file overflow would basically have to be done on input files boundaries (because for example one of the branch may have only one basket or more exactly it is very unlikely to fine a basket boundaries where all the branches are on the same entry number).

Also it is a bit counter-intuitive to merge files into several output files :slight_smile:

Cheers,
Philippe

5

Also it is a bit counter-intuitive to merge files
into several output files Smile

:slight_smile:

but logical, when there are many many files to be merged, resulting from big skimming jobs.

you will also note that there is a note in the online documentation of TChain::Merge(), which says the automatic file overflow apply. Would be good to add that it is not true in the case in “fast” option.

6

[quote]Would be good to add that it is not true in the case in “fast” option.[/quote]Indeed.

Cheers,
Philippe

7

We are proceeding with two steps :

  1. the skimming jobs, which are eventually run in parallell, each one with a subset of the input files
  2. the merge job, which is merging the results of 1)
8

Whatsoever, one could want to merge together many small files, but yet define a maximum file size. That sounds not so exotic.

9

I see :slight_smile:

Philippe

10

Hi David,

The fast cloning/merging mode is now (in CVS) implementing the automatic file overflow (check at each input file boundary).

Cheers,
Philippe

11

Thanks. I will probably wait for next release before I test.