"bad allocation" error when splitting one big TTree into many smaller ones

ROOT
1

Hi,
I’ve searched the forum for a similar problem, but couldn’t find a solution, if there is any.
My problem is that I try to split one relatively big TTree into a large numbers of smaller trees. The main tree contains data from a pixelated detector and I want to have a separate tree saved in a separate file for every pixel data. The part of the code where I am doing this is as following:
…
TFile *ff = new TFile(fname,“read”);
TTree oldtree = (TTree)ff->Get(tree_name);
TFile **ff_pix;
TTree newtree;
ff_pix = new TFile[nPixInLoop];
newtree = new TTree[nPixInLoop];
for (Int_t m = 0; m < nPixInLoop; m++)
{
ff_pix[m] = TFile::Open(fname2,“recreate”);
newtree[m] = oldtree->CloneTree(0);
}
cout << “Starting to spit the data” << endl;
for (Long64_t i = 0; i < nentries; i++)
{
oldtree->GetEntry(i);
Int_t pixel = Int_t(channel+0.5);
newtree[pixel]->Fill();
}
…
When running the code, it fails after some time on filling the new tree (newtree[pixel]->Fill()) with the following error:

Exception: bad allocation
Error: Symbol G__exception is not defined in current scope …
Error: type G__exception not defined FILE:C: …
*** Interpreter error recovered ***

The physical size of the trees is not big - the main tree is 200-300 Mb, and each pixel tree could be 5-10 Mb in the end. I have a new PC with lots of memory, so I wouldn’t expect a memory problem anyway. The problem usually happens either when I am trying to split data for many pixels simultaneously (I have a total of 500 pixels) or if one of the pixels has much more data than the others. Ideally, I’d like to perform this split for all 500 pixels in just one loop.

Thanks for the help!

2

Hi,

what is the actual memory usage?
You can use something quick to find out like

void memory()
{
   static ProcInfo_t info;
   gSystem->GetProcInfo(&info);
   const float toMB = 1.f/1024.f;
   printf(" res  memory = %g Mbytes\n", info.fMemResident*toMB);
   printf(" vir  memory = %g Mbytes\n", info.fMemVirtual*toMB);
}

Danilo

3

Hi,
The memory usages starts from
res memory = 88.1602 Mbytes
vir memory = 142.813 Mbytes

and the code crashes at
res memory = 248.172 Mbytes
vir memory = 1859.83 Mbytes

I’ve got 8 Gb of memory and 64-bit Windows 7, so I thought that should be ok.

Thanks.

4

Hi,

thanks for checking: indeed this cannot be the issue.
Could you post a minimal reproducer for us to run?

Danilo

5

It seems to me that your code crashes when “res + vir memory” reaches 2 GB RAM, so maybe your Windoze has some memory limit for applications set.
BTW. I hope you run a 64-bit version of ROOT executables, otherwise the 2 GB RAM limit is probably “imposed” by the executable itself.

6

Hi,
you are actually right! it is a Win-32 executable, so the origin of this 2 Gb limit is clear then.
I checked the ROOT download page and unfortunately I don’t see any pre-compiled 64-bit executables for Windows. Is there then any way not to keep all tree contents in the memory during the splitting operation, to decrease the memory usage? That seems the only way to overcome this problem (I don’t really fancy compiling the distribution package in Windows by myself).

Thanks a lot for the help!