Copy tree with memory problems

ROOT
1

Dear root expert,
I have a very simple test job, copy the tree from an old file to
a new file event by event.
The old tree have more then 30000 events and more than
1000 branches, the original size is 800M, the jobs crashed at the end
when it write and save the new file, it look like is due to the memory
problems. The error message is :

terminate called after throwing an instance of 'std::bad_alloc’
what(): St9bad_alloc

My machine is 4 core with 8G memory.

When I run for 10000 events, the jobs is successful,  but when I load

the tree, and print it, I found that the speed of print the new tree is much
slower than the speed of loading old tree, the compressions of tree and
branch are different between new and old tree, also in each branch, the new
tree show " All baskets in memory". How can I fix the problems.
I attach the print messages and my code

Old tree :

*Tree :CollectionTree: CollectionTree *
*Entries : 36588 : Total = 2508577943 bytes File Size = 832988990 *

  •    :          : Tree compression factor =   3.01                       *

*Br 0 :RunNumber : RunNumber/I *
*Entries : 36588 : Total Size= 147382 bytes File Size = 1120 *
*Baskets : 4 : Basket Size= 32000 bytes Compression= 114.28 *
…
*Br 1 :EventNumber : EventNumber/I *
*Entries : 36588 : Total Size= 147402 bytes File Size = 46517 *
*Baskets : 4 : Basket Size= 32000 bytes Compression= 2.75 *
…
*Br 2 :StreamESD_ref : StreamESD_ref/C *
*Entries : 36588 : Total Size= 5767282 bytes File Size = 434011 *

new Tree :

*Tree :CollectionTree: CollectionTree *
*Entries : 10000 : Total = 519198944 bytes File Size = 191956168 *

  •    :          : Tree compression factor =   1.00                       *

*Br 0 :RunNumber : RunNumber/I *
*Entries : 10000 : Total Size= 40850 bytes All baskets in memory *
*Baskets : 1 : Basket Size= 32000 bytes Compression= 1.00 *
…
*Br 1 :EventNumber : EventNumber/I *
*Entries : 10000 : Total Size= 40866 bytes All baskets in memory *
*Baskets : 1 : Basket Size= 32000 bytes Compression= 1.00 *
…
*Br 2 :StreamESD_ref : StreamESD_ref/C *
*Entries : 10000 : Total Size= 1581118 bytes All baskets in memory *
*Baskets : 50 : Basket Size= 32000 bytes Compression= 1.00 *
…

My code :
void test2() {
TBenchmark* Mybench = new TBenchmark();
Mybench->Start(“Analysis time”);
TChain *fchain = new TChain(“CollectionTree”);
fchain->Add(“user09.JoaquinPovedaTorres.0108173.topmix_Egamma.AANT.AANT._00001.root.1”);
TTree *ntuple = NULL;
int index =0;
char buffer[40];
cout<<" entry = “<GetEntries()<<endl;
ntuple = fchain->CloneTree(0);
ntuple->SetDirectory(0);
fchain->CopyAddresses(ntuple);
for(int i=0; iGetEntries()&&i<10000; i++){
if(i%2000==0)cout<<” i = "<<i<<endl;
fchain->GetEntry(i);
ntuple->Fill();
ntuple->AutoSave();
}
TFile *f = new TFile(“test.root”,“recreate”);
ntuple->Write();
ntuple->Delete();
f->Write();
f->Close();
Mybench->Stop(“Analysis time”);
Mybench->Show(“Analysis time”);
}
thanks in advance.
with best regards
—Haiping

2

UseTFile *f = new TFile("test.root","recreate"); cout<<" entry = "<<fchain->GetEntries()<<endl; ntuple = fchain->CloneTree(0); fchain->CopyAddresses(ntuple); for(int i=0; i<fchain->GetEntries()&&i<10000; i++){ if(i%2000==0)cout<<" i = "<<i<<endl; fchain->GetEntry(i); ntuple->Fill(); ntuple->AutoSave(); } By using SetDirectory(0); you had explicitly requested for the TTree to be memory resident.

Cheers,
Philippe.

3

when doing:

ntuple->SetDirectory(0); you keep the new tree in memory. Why are you doing that?

Rene

4

Hi Rene and Philippe,
Thank you very much for quick responds. For the simple jobs,
It will work well when I remove the line SetDirctory(0).
But in practice, My programs is a bit complex than it, I define
a class, and the class include the TFile and TTree. When I
apply the class, First I define the TFile and TTree, and then
copy the events from other Files to this file one by one. If I did not
set SetDirctory(0). It will show :

Error in TBranchElement::Fill: Failed filling branch:trk_rec_px, nbytes=-1
Error in TTree::Fill: Failed filling branch:CollectionTree1.trk_rec_px, nbytes=-1
This error is symptomatic of a Tree created as a memory-resident Tree
Instead of doing:
TTree T = new TTree(…)
TFile f = new TFile(…)
you should do:
TFile f = new TFile(…)
TTree T = new TTree(…)
,***********************************************************************
in some events, also, it is random for the events.
but actually, I defend the TFile bebefor the TTree. Any suggestion?

Thanks in advance.
with best regards
- --Haiping

[quote=“hpeng”]Dear root expert,
I have a very simple test job, copy the tree from an old file to
a new file event by event.
The old tree have more then 30000 events and more than
1000 branches, the original size is 800M, the jobs crashed at the end
when it write and save the new file, it look like is due to the memory
problems. The error message is :

terminate called after throwing an instance of 'std::bad_alloc’
what(): St9bad_alloc

My machine is 4 core with 8G memory.

When I run for 10000 events, the jobs is successful,  but when I load

the tree, and print it, I found that the speed of print the new tree is much
slower than the speed of loading old tree, the compressions of tree and
branch are different between new and old tree, also in each branch, the new
tree show " All baskets in memory". How can I fix the problems.
I attach the print messages and my code

Old tree :

*Tree :CollectionTree: CollectionTree *
*Entries : 36588 : Total = 2508577943 bytes File Size = 832988990 *

  •    :          : Tree compression factor =   3.01                       *

*Br 0 :RunNumber : RunNumber/I *
*Entries : 36588 : Total Size= 147382 bytes File Size = 1120 *
*Baskets : 4 : Basket Size= 32000 bytes Compression= 114.28 *
…
*Br 1 :EventNumber : EventNumber/I *
*Entries : 36588 : Total Size= 147402 bytes File Size = 46517 *
*Baskets : 4 : Basket Size= 32000 bytes Compression= 2.75 *
…
*Br 2 :StreamESD_ref : StreamESD_ref/C *
*Entries : 36588 : Total Size= 5767282 bytes File Size = 434011 *

new Tree :

*Tree :CollectionTree: CollectionTree *
*Entries : 10000 : Total = 519198944 bytes File Size = 191956168 *

  •    :          : Tree compression factor =   1.00                       *

*Br 0 :RunNumber : RunNumber/I *
*Entries : 10000 : Total Size= 40850 bytes All baskets in memory *
*Baskets : 1 : Basket Size= 32000 bytes Compression= 1.00 *
…
*Br 1 :EventNumber : EventNumber/I *
*Entries : 10000 : Total Size= 40866 bytes All baskets in memory *
*Baskets : 1 : Basket Size= 32000 bytes Compression= 1.00 *
…
*Br 2 :StreamESD_ref : StreamESD_ref/C *
*Entries : 10000 : Total Size= 1581118 bytes All baskets in memory *
*Baskets : 50 : Basket Size= 32000 bytes Compression= 1.00 *
…

My code :
void test2() {
TBenchmark* Mybench = new TBenchmark();
Mybench->Start(“Analysis time”);
TChain *fchain = new TChain(“CollectionTree”);
fchain->Add(“user09.JoaquinPovedaTorres.0108173.topmix_Egamma.AANT.AANT._00001.root.1”);
TTree *ntuple = NULL;
int index =0;
char buffer[40];
cout<<" entry = “<GetEntries()<<endl;
ntuple = fchain->CloneTree(0);
ntuple->SetDirectory(0);
fchain->CopyAddresses(ntuple);
for(int i=0; iGetEntries()&&i<10000; i++){
if(i%2000==0)cout<<” i = "<<i<<endl;
fchain->GetEntry(i);
ntuple->Fill();
ntuple->AutoSave();
}
TFile *f = new TFile(“test.root”,“recreate”);
ntuple->Write();
ntuple->Delete();
f->Write();
f->Close();
Mybench->Stop(“Analysis time”);
Mybench->Show(“Analysis time”);
}
thanks in advance.
with best regards
—Haiping[/quote]

5

Hello Rene and Philippe,

I am sorry I  confused both of you. It look like the jobs are

working well. The problem happened this morning is due to
the disk problem on my machine, I found the disk which I want to
write the new file have problems this morning. Now the problem
fixed and program are working well.

Thank you very much
With best regards
—Haiping

[quote=“hpeng”]Hi Rene and Philippe,
Thank you very much for quick responds. For the simple jobs,
It will work well when I remove the line SetDirctory(0).
But in practice, My programs is a bit complex than it, I define
a class, and the class include the TFile and TTree. When I
apply the class, First I define the TFile and TTree, and then
copy the events from other Files to this file one by one. If I did not
set SetDirctory(0). It will show :

Error in TBranchElement::Fill: Failed filling branch:trk_rec_px, nbytes=-1
Error in TTree::Fill: Failed filling branch:CollectionTree1.trk_rec_px, nbytes=-1
This error is symptomatic of a Tree created as a memory-resident Tree
Instead of doing:
TTree T = new TTree(…)
TFile f = new TFile(…)
you should do:
TFile f = new TFile(…)
TTree T = new TTree(…)
,***********************************************************************
in some events, also, it is random for the events.
but actually, I defend the TFile bebefor the TTree. Any suggestion?

Thanks in advance.
with best regards
- --Haiping

[quote=“hpeng”]Dear root expert,
I have a very simple test job, copy the tree from an old file to
a new file event by event.
The old tree have more then 30000 events and more than
1000 branches, the original size is 800M, the jobs crashed at the end
when it write and save the new file, it look like is due to the memory
problems. The error message is :

terminate called after throwing an instance of 'std::bad_alloc’
what(): St9bad_alloc

My machine is 4 core with 8G memory.

When I run for 10000 events, the jobs is successful,  but when I load

the tree, and print it, I found that the speed of print the new tree is much
slower than the speed of loading old tree, the compressions of tree and
branch are different between new and old tree, also in each branch, the new
tree show " All baskets in memory". How can I fix the problems.
I attach the print messages and my code

Old tree :

*Tree :CollectionTree: CollectionTree *
*Entries : 36588 : Total = 2508577943 bytes File Size = 832988990 *

  •    :          : Tree compression factor =   3.01                       *

*Br 0 :RunNumber : RunNumber/I *
*Entries : 36588 : Total Size= 147382 bytes File Size = 1120 *
*Baskets : 4 : Basket Size= 32000 bytes Compression= 114.28 *
…
*Br 1 :EventNumber : EventNumber/I *
*Entries : 36588 : Total Size= 147402 bytes File Size = 46517 *
*Baskets : 4 : Basket Size= 32000 bytes Compression= 2.75 *
…
*Br 2 :StreamESD_ref : StreamESD_ref/C *
*Entries : 36588 : Total Size= 5767282 bytes File Size = 434011 *

new Tree :

*Tree :CollectionTree: CollectionTree *
*Entries : 10000 : Total = 519198944 bytes File Size = 191956168 *

  •    :          : Tree compression factor =   1.00                       *

*Br 0 :RunNumber : RunNumber/I *
*Entries : 10000 : Total Size= 40850 bytes All baskets in memory *
*Baskets : 1 : Basket Size= 32000 bytes Compression= 1.00 *
…
*Br 1 :EventNumber : EventNumber/I *
*Entries : 10000 : Total Size= 40866 bytes All baskets in memory *
*Baskets : 1 : Basket Size= 32000 bytes Compression= 1.00 *
…
*Br 2 :StreamESD_ref : StreamESD_ref/C *
*Entries : 10000 : Total Size= 1581118 bytes All baskets in memory *
*Baskets : 50 : Basket Size= 32000 bytes Compression= 1.00 *
…

My code :
void test2() {
TBenchmark* Mybench = new TBenchmark();
Mybench->Start(“Analysis time”);
TChain *fchain = new TChain(“CollectionTree”);
fchain->Add(“user09.JoaquinPovedaTorres.0108173.topmix_Egamma.AANT.AANT._00001.root.1”);
TTree *ntuple = NULL;
int index =0;
char buffer[40];
cout<<" entry = “<GetEntries()<<endl;
ntuple = fchain->CloneTree(0);
ntuple->SetDirectory(0);
fchain->CopyAddresses(ntuple);
for(int i=0; iGetEntries()&&i<10000; i++){
if(i%2000==0)cout<<” i = "<<i<<endl;
fchain->GetEntry(i);
ntuple->Fill();
ntuple->AutoSave();
}
TFile *f = new TFile(“test.root”,“recreate”);
ntuple->Write();
ntuple->Delete();
f->Write();
f->Close();
Mybench->Stop(“Analysis time”);
Mybench->Show(“Analysis time”);
}
thanks in advance.
with best regards
—Haiping[/quote][/quote]

6

[quote=“pcanal”]UseTFile *f = new TFile("test.root","recreate"); cout<<" entry = "<<fchain->GetEntries()<<endl; ntuple = fchain->CloneTree(0); fchain->CopyAddresses(ntuple); for(int i=0; i<fchain->GetEntries()&&i<10000; i++){ if(i%2000==0)cout<<" i = "<<i<<endl; fchain->GetEntry(i); ntuple->Fill(); ntuple->AutoSave(); } By using SetDirectory(0); you had explicitly requested for the TTree to be memory resident.

Cheers,
Philippe.[/quote]

I just wanted to say I found this post very helpful. I needed to delete the last few entries of each branch in a tree, which I did by copying the original tree to a new one and tuning this line: