BuildIndex aborts for a large TTree

sembrosk · October 10, 2006, 3:40pm

Hi all,

I am trying to reading a large file of events(>4 G bytes) into a TTree and then run BuildIndex on that TTree.

I’m using root 5.08.00

The input file consists of about 61,000,000 events. I create the TTree with:

KSRootPeData* pfRootPe = new KSRootPeData(); TFile* pfTempTreeFile = (TFile*) new TFile("SortTempTreeFile.root", "RECREATE","Temoprary Sort ROOT file"); pfTempTreeFile->cd(); TTree* pfPeTree=new TTree("PeTree","Pe Tree"); pfPeTree->Branch("Pe","KSRootPeData",&pfRootPe,1000,0);
Where KSRootPeData consists of 11 int and double variables for a total length of 68 bytes.

In a while loop I read events from the input file and Fill the TTree:

while(1) { pfRootPe->CopyInPe(pfBinaryPe); //Test for Eof , is so break else Fill TTree . . pfPeTree->Fill(); }

After reading in and filling about ~27,700,000 events I get:

Fill: Switching to new file: SortTempTreeFile_1.root

When the events have all been loaded (into 2 files? I’m assuming root knows what its doing here…) I call BuildIndex on the TTree:

int fNumIndexes=pfPeTree->BuildIndex(nxStr,nyStr);
This aborts (in gdb) with:

Program received signal SIGABRT, Aborted.
[Switching to Thread 16384 (LWP 16227)]
0x41565741 in kill () from /lib/libc.so.6
(gdb) where
#0 0x41565741 in kill () from /lib/libc.so.6
#1 0x414f3771 in pthread_kill () from /lib/libpthread.so.0
#2 0x414f3a7b in raise () from /lib/libpthread.so.0
#3 0x415654d4 in raise () from /lib/libc.so.6
#4 0x41566a08 in abort () from /lib/libc.so.6
#5 0x400bcf57 in __cxa_call_unexpected () from /usr/lib/libstdc++.so.5
#6 0x400bcf94 in std::terminate () from /usr/lib/libstdc++.so.5
#7 0x400bd106 in __cxa_throw () from /usr/lib/libstdc++.so.5
#8 0x400bd35f in operator new () from /usr/lib/libstdc++.so.5
#9 0x400bd42f in operator new[] () from /usr/lib/libstdc++.so.5
#10 0x426acc4a in TTreeIndex::TTreeIndex ()
from /home/sembrosk/local/src/root/lib/libTreePlayer.so
#11 0x426ae468 in TTreePlayer::BuildIndex ()
from /home/sembrosk/local/src/root/lib/libTreePlayer.so
#12 0x41104b54 in TTree::BuildIndex ()
from /home/sembrosk/local/src/root/lib/libTree.so
#13 0x08052b66 in main (argc=2, argv=0xbfffe65c) at src/ksPeSort.cpp:222

During the BuildIndex the memory usage slowly climbs and reaches ~32% by the time the seg fault occurs.

Can anyone tell me how I should be doing this? This code works fine for files <27,700,000 (before root decides to make the second file).

Thanks,
Glenn

brun · October 10, 2006, 5:20pm

Could you post the result of

after the Fill loop ?
Could you post your code when building the index?

The result of BuildIndex must fit in memory, so in your case it can be a quite large object, but you seem to have another problem before reaching the memory limitation.

Rene

sembrosk · October 10, 2006, 6:08pm

Here is the ouput that includes the pfPeTree->Print();

Note a few things:
1: the pfPeTree->Print(); seems to only look at the first file as does the pfPeTree->GetEntries();
2: I haven’t cleaned up the additional SortTempTreeFile_*1 made from previous attemps. but root seems to handle this ok.

Warning in : file SortTempTreeFile_1.root already exist, trying with 2 underscores
Warning in : file SortTempTreeFile__1.root already exist, trying with 3 underscores
Warning in : file SortTempTreeFile___1.root already exist, trying with 4 underscores
Warning in : file SortTempTreeFile____1.root already exist, trying with 5 underscores
Fill: Switching to new file: SortTempTreeFile_____1.root
ksPeSort: Found input file EOF

*Tree eTree : Pe Tree *
*Entries : 27795080 : Total = 2517771738 bytes File Size = 1566452776 *

   :          : Tree compression factor =   1.59                       *

*Br 0 e : *
*Entries :27795080 : Total Size= 2553491568 bytes File Size = 1553352063 *
*Baskets : 2316256 : Basket Size= 1100 bytes Compression= 1.59 *
…
ksPeSort: Input file read successfully------Tue Oct 10 13:52:54 2006

ksPesort: Number of PEs read in: 61423124
ksPeSort: Number of PEs in tree: 27795080
ksPeSort: NX minimum= -11930
ksPeSort: NY minimum= -10869
ksPeSort: Sorting by NY fastest and increasing
Aborted

The code follows:
KSPeData* pfBinaryPe = new KSPeData();
KSRootPeData* pfRootPe = new KSRootPeData();

TFile* pfTempTreeFile = (TFile*) new TFile(“SortTempTreeFile.root”,
“RECREATE”,“Temoprary Sort ROOT file”);
pfTempTreeFile->cd();
TTree* pfPeTree=new TTree(“PeTree”,“Pe Tree”);

pfPeTree->Branch(“Pe”,“KSRootPeData”,&pfRootPe,1000,0);

int nxmin=0; //If minimum is > 0 we don’t care
int nymin=0;
int icount=0;
while(1)
{
bool goodread=pfPeFile->ReadPe(pfBinaryPe);
if(goodread)
{
if(pfBinaryPe->fNxfNx;
}
if(pfBinaryPe->fNyfNy;
}
pfRootPe->CopyInPe(pfBinaryPe);
pfPeTree->Fill();
icount++;
}
else //Check for eof here.
{
if(!pfPeFile->foundEOF())
{
std::cout<fNx + %i",nxoffset);
char nyStr[80];
sprintf(nyStr,“Pe->fNy + %i”,nyoffset);

int fNumIndexes=pfPeTree->BuildIndex(nxStr,nyStr); //Ny varies fastest

Thanks,
Glenn

brun · October 11, 2006, 9:16am

You did not send the most useful information (full output of tree.Print())
Because you cannot post the large file, could you provide a simple
simulator of your case creating a Tree with the same branches and filling it
with values close to your original program?

Rene

sembrosk · October 11, 2006, 12:15pm

Rene,
I will do as you ask and provide a program to simulate the problem. You should know, however, that I did provide all the full ouput of the myTree.Print(). Thats it. Thats all that came out.

Glenn

brun · October 11, 2006, 12:19pm

You should, at least, replace the statement:

by

Let me know once you have the simulator.

Rene

sembrosk · October 11, 2006, 2:00pm

Rene,
I attach a program that is supposed to simulate my problem. While this simulation does Abort after calling Fill about 55,000,000 times it doesn’t quite behave as my program did. It does NOT give me the:

Fill: Switching to new file: SortTempTreeFile__1.root

comment.

I’ve tried this with the splitlevel set at 99 and at 0. When at 99 the TFile is much smaller when it fails (~900Mbytes).

Glenn
LargeTTreeTest.cpp (2.75 KB)

pcanal · October 11, 2006, 9:31pm

Hi,

I can not reproduce the problem with ROOT 5.13/04.
Could you please try it out with the latest release?

Thanks,
Philippe

sembrosk · October 13, 2006, 4:44pm

I have simplified the test program (LargeTTreeTest2.cpp ) and attach it and the build script for it. (TestBuild.txt)

Under root 5.13.04 and 5.12.00e at event ~44,000,000 of the calls to Fill() I get:

Fill: Switching to new file: SortTempTreeFile_1.root

The program then continues on, presumaly filling a TTree in this new file, but when it calls the BuildIndex it appears to only do it for this last file. (As if their TTrees were supposed to be chained but wern’t? Does BuildIndex work for chained TTrees?)

Glenn

ps. If in LagreTTreeTest2.cpp you set

fPeFill=false;

then the program successfully (as far as I can tell) fills the first TTree with 61,000,000 entries (compression probably has some effect here). However, it then Aborts when it tries to do a BuildIndex on this TTree.
This happens for root versions:

5.08.00
5.12.00e
5.13.04

If, after running this program with fPeFill=false, I run root and look at the file produced, SortTempTreeFile.root,

TFile f(“SortTempTreeFile.root”)

I get:
Warning in : file SortTempTreeFile.root probably not closed, trying to recover
Info in : SortTempTreeFile.root, recovered key TTree:PeTree at address 338420238
Info in : SortTempTreeFile.root, recovered key TTree:PeTree at address 342916614
Warning in : successfully recovered 2 keys

which is understandable.

If I then look at the file with the TBrowser it shows 2 TTress: PeTree and PeTree;41.

If I run BuildIndex I get:

root [2] PeTree->BuildIndex(“fNx+12000”,“fNy+12000”)
Error: Symbol G__exception is not defined in current scope (tmpfile):1:
Error: type G__exception not defined FILE:(tmpfile) LINE:1
(Int_t)0
*** Interpreter error recovered ***
TestBuild.txt (439 Bytes)
LargeTTreeTest2.cpp (3.7 KB)

pcanal · October 13, 2006, 5:11pm

Note that in your code, you never write the TFile itself. (this leads to the Recover message). In addition[quote]I get:
Fill: Switching to new file: SortTempTreeFile_1.root [/quote]when this appears it means (see TTree::ChangeFile) that your pfTempTreeFile is now invalidated (the switch of file induce a deletion (in memory) of the TFile object). So the correct end to function must be:pfPeTree->GetFile()->Write(); // this replaces pfPeTree->Write();

Indeed and explicitly! After the ChangeFile your TTree object is still a TTree and not a TChain. It only points to the data in the last file. In order to build the index on all trees you might need to either open the file one by one or possibly create a TChain yourself.

[quote]Error: Symbol G__exception is not defined in current scope (tmpfile):1:
Error: type G__exception not defined FILE:(tmpfile) LINE:1 [/quote]

This looks more like an install problem … (did you do gmake install and/or gmake cintdlls).

Also you may want to simply disable the ChangeFile and create a single file (See TTree::SetMaxTreeSize)

Cheers,
Philippe.

sembrosk · October 13, 2006, 6:43pm

Philippe.
Wow! Thanks this solved one of my problems. If I replace:

TTree* pfPeTree=new TTree(“PeTree”,“Pe Tree”);

with:

TTree* pfPeTree=new TTree(“PeTree”,“Pe Tree”);
pfPeTree->SetMaxTreeSize(20000000000LL);

in LargeTTreeTest2.cpp then it creates just a single file of 61,000,000 entries in one TTree (even with bool fPeFill=true; ). This solves half my problem.

(Note TBrowser shows PeTree PeTree;41 PeTree;42)

I still get the Aborted message when I do the BuildIndex.

If I replace:
pfPeTree->Write();
pfPeTree->Print();

with:

pfPeTree->Write();
pfPeTree->Print();
pfTempTreeFile->Write();

in LargeTTreeTest2.cpp then if after it Aborts when the BuildIndex command, I start up root and do:

root [0] TFile f(“SortTempTreeFile.root”)
root [1] TTree* h= (TTree*)f->Get(“PeTree”);
root [2] h->BuildIndex(“fNx”,“fNy”)
Error: G__getvariable: expression (tmpfile):1:
Error: Can not access pointer to function 0x0 from interpreter(1) FILE:(tmpfile) LINE:1
Error: $()1109394656 Syntax error? (tmpfile):1:
Syntax Error: * (tmpfile):1:
Error: Illegal pointer operation (tovalue) (tmpfile):1:
Error: memory allocation for ((*)1109394656) size=4 pinc=1 FILE:(tmpfile) LINE:1
(Int_t)0
*** Interpreter error recovered ***
root [3]

This is the result for roots 5.0.8, 5.12.00e and 5.13.4. All of these were downloaded yesterday and made with:
./configure
gmake
gmake cintdlls
gmake install

Glenn

ps: With the use of the SetMaxTreeSize it now appears I have to delete the very large (>3 g byte) SortTempTreeFile.root before re-running LargeTTreeTest2. This is strange since I thought the “RECREATE” in the TFile constructor call should have caused that to happen. Doesn’t look like it does for this big file.

pcanal · October 13, 2006, 10:04pm

I noted the following in your build.txt:-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCEPlease do not use this, as it change the meaning of fpos_t (and related function) without change their link time symbol. So if you use them and link again a library that is build differently this might end up being bery bad!

Cheers,
Philippe

sembrosk · October 13, 2006, 10:35pm

I removed the 2 defines. No change in behavior. I seems the program creates a file with a TTree with 61,000,000 entries. This looks fine. I can open the file in root, plot things, everything works. The only thing that doesn’t work in the program or in root is the BuildIndex. Should I be submitting a Bug report on this?

Glenn

pcanal · October 16, 2006, 3:43pm

I still cannot reproduce your problem. I have to believe that you are running out of memory (see the result of the shell command limit or ulimit -a).

Cheers,
Philippe.