Is it possible to store huge events (>several GB) in TTre

ROOT
1

Dear root devels and experts,

for quite some while I am storing MC air shower data in ROOT format.
Momentarily I am following the standard TTree philosophy, and this is
also what I would prefer to do in the future. In principle what I do is:

fTree->Branch(“mc”, &event); ,

where event is one MC event that contains e.g. a list of particles (of
course a TClonesArray …), that can easily grow to >10^9 particles - or
more (depending on your settings).

The problem I am encountering is that a single of these events can
easily have the size of several GBs. Under many circumstance and on
typical computer systems this rapidly leads to memory depletion,
swapping and crashes, since always one full event has to be build up in
memory, and only when it is completed it is stored as a big blob with

fTree->Fill();

to the disk. I was thinking of many different tree designs, but the
principle problem I could never solve.

The only way I see now, is to give up the concept of storing the data in
trees, and create TFolder objects for each MC event, that I can fill
with data in a much more controlled way. Unfortunately I loose all the
nice features of trees.

Before I now go on and go for the TFolder solution, it is my last hope
to write to the roottalk list and ask for any experience out there on
storing REALLY HUGE events (many GB each) with a TTree.

Thank you very much for any hints and help

Cheers
Ralf Ulrich

2

ROOT files are not limited in size (you can create what the OS allows you to do).
By default, Tree size is limited to 1.9 GByte. You can extend this limit, eg

//Authorize Trees up to 2 Terabytes (if the system can do it) TTree::SetMaxTreeSize(1000*Long64_t(2000000000));

If you have enough memory to store 100 million objects in a TClonesArray (this will probably require 10 GBytes of memory), you can create a Tree in split mode where each branch will have a buffer up to (say) 1 GByte.

However, this requirement could be symptomatic of a ill-defined problem.
Why not store each entry in the TClonesArray as a separate entry in a Tree?
This way you will get a very conventional tree with no memory penalty.

Rene

3

Thank you for your answer. I interpret it as an (expected):
No, this is not possible.

As you mentioned, I will redesign the problem to solve the issue, which will unfortunately mean to give up the top level TTree to store entire events.