TChain and BuildIndex exhausts memory

ROOT
1

Hi I am chaining 3 ~900MB files together in a chain and then building an index across them. Then I loop over the chain extracting certain quantities via the index. The prbblem I am having is that the script seems to crawl along only using 4% of the cpu but 85% of the machines memory (I am guessing the machine is also swapping heavily because the job is constantly in the D state). Could someone tell me if this is a result of the index building on the TChain and is there a better way to approach this problem so the job will run faster. Thanks.

I am using root 5.02.00, thanks for any help.

Here is a simple example of what I am doing:

[code]
Int_t index_chain()
{
TChain * e = new TChain(“eventTree”);
TChain * t = new TChain(“trackTree”);
e->Add(".root");
t->Add(".root");
Int_t tr_event, tr_run, ev_event, ev_run;
t->SetBranchAddress(“ievent”,&tr_event);
t->SetBranchAddress(“irun”,&tr_run);
e->SetBranchAddress(“ievent”,&ev_event);
e->SetBranchAddress(“irun”,&ev_run);

e->BuildIndex(“irun”,“ievent”);
Int_t tracks = t->GetEntries();
for( Int_t it=0 ; it <tracks ; it++)
{
t->GetEntry(it);
e->GetEntryWithIndex(tr_run,tr_event);
}
}[/code]

2

I suspect a memory leak in your classes.
What is the number of entries (int tracks)?
Did you check the size of your program without the index?

Rene

3

Rene,

I have switched to a newer machine and the problem seems to be less
evident. Perhaps it is some actual hardware problem on the previous
machine. Now the memory usage never tops ~20%. I will repost as soon
as the code finishes if the problem is really gone.

To answer your question tracks ~= 2.2Million.

Thanks (as always) for your quick reply.

I also ran the little script I posted (which of course is a very
distilled example) and it seems to run extremely fast. So it was
either the machine or in the more complex code I have a memory leak
somewhere. Like I said above I will repost as soon as this run is
done.

Alex

4

Rene,

I was wrong, the machine is still going to run out of memory once it gets near the end of the run. The only thing I am doing in the loop is filling TMatrixF objects, could this be the problem?

Actually do you mean there may be a problem in the TTrees when they are created? How can I check if this is the problem?

5

Could you change the following line in $ROOTSYS/etc/system.rootrc
Root.ObjectStat: 0
to
Root.ObjectStat: 1
and in your loop add the following line
if (it%10000 ==0) gObjectTable->Print();

If you leak objects (deriving from TObject like TMatrixF) you will see the number of instances increasing.
Are you deleting your matrix objects in the loop?

Rene

6

Rene,

I followed your instructions. It seems that the total number of objects stays static during the loop so I guess I am not leaking memory that way. Is there a way for me to test if there is a problem in the TTrees themselves?

7

Could you run with valgrind and send me the two outputs (ps and txt files)?
valgrind --tool=massif root.exe

Could you also send the last output of gObjectTable->Print(); ?

Rene