Problem with TTree size in memory

Hi,

I’m experiencing the following problem.
After about 400k events, our program complains that the TTree in memory is too large and rises a “bad alloc” exception.

On the stderr I get:

[quote]SysError in TFile::Flush: error flushing file .//Run003488.root (Stale NFS file handle)
Error in TBranchElement::Fill: Failed filling branch:pid.shape.y3[7], nbytes=-1
Error in TBranchElement::Fill: Failed filling branch:pid.pid.shape.y3[7], nbytes=-1
Error in TBranchElement::Fill: Failed filling branch:events.pid, nbytes=-1
Error in TTree::Fill: Failed filling branch:bxtree.events, nbytes=-1
Error in TBranchElement::Fill: Failed filling branch:laben.decoded_hits.raw_time, nbytes=-1
Error in TBranchElement::Fill: Failed filling branch:laben.decoded_hits.laben.decoded_hits.raw_time,
nbytes=-1
…[/quote]
and so on for all my sub-branches, for many events.

On the stdout I get:

[quote]This error is symptomatic of a Tree created as a memory-resident Tree
Instead of doing:
TTree *T = new TTree(…)
TFile *f = new TFile(…)
you should do:
TFile *f = new TFile(…)
TTree *T = new TTree(…)

[/quote]

a zillion times. The suggestion is odd as I already open first the TFile and then the TTree.

Finally my exception catcher writes:

The (unclosed) file is about 1GB, while MaxTreeSize is set to 128GB and buffer size to 32.000. I’m running on a very performing computing cluster.

I think I should tell ROOT to periodically release the memory after writing to file but how? Autosave? periodical TTree::Write()?

Thanks in advance for the help
regards
Davide D.

P.S.: Unfortunately the program is far too complex and unportable to be sent to you.

Hi Davide,

From your log file, it looks like you simply hit an NFS problem.
For un unknown reason, your process cannot access the file anymore.

NFS is know to be a very bad system. Working with better systems like xrootd is the solution to avoid your program crashing because of a temporary network interruption or similar.

Rene

Hello, I am facing the same problem as below -

Error in TBranchElement::Fill: Failed filling branch:weight_bTagSF_77_eigenvars_C_down, nbytes=-1
Error in TTree::Fill: Failed filling branch:nominal.weight_bTagSF_77_eigenvars_C_down, nbytes=-1, entry=3752
This error is symptomatic of a Tree created as a memory-resident Tree
Instead of doing:
TTree *T = new TTree(…)
TFile *f = new TFile(…)
you should do:
TFile *f = new TFile(…)
TTree *T = new TTree(…)

It shows this repititively. I am trying to create a Tchain as shown below -

TChain ch = new TChain (“nominal”);
while (std::getline(file,str))
{
file_contents = str;
const char
filename = file_contents.c_str();
std::cout << "name: " << filename << “\n”;
TFile* f = TFile::Open(filename);
TTree* tree = (TTree*)f->Get(“nominal”);
treename = file_contents + “/nominal”;
const char* name = treename.c_str();
ch->AddFile(name);
}
I have attached the file for more details.
Thank you
Cross_section_macro.txt (5.72 KB)

The calls:TFile* f = TFile::Open(filename); TTree* tree = (TTree*)f->Get("nominal"); are unnecessary (your code will work the same with or without them … with them you are wasting resources).

The problem is likely that you have

TFile *newfile = new TFile("/gs/project/atlas-mcgill/ssaha/AnalysisTop-2.4.17_NonPrompt/merged_ttbar_R_new.root");where you likely meant

TFile *newfile = new TFile("/gs/project/atlas-mcgill/ssaha/AnalysisTop-2.4.17_NonPrompt/merged_ttbar_R_new.root","RECREATE"); // or at least "UPDATE"without the option, it opens the file for reading and the TTree are kept in memory.

Cheers,
Philippe.

Hi Philippe,
Thank you for the reply. I figured out I do not need the new file, I just need to create two new trees and print the histograms. I have commented out the line below -
TFile *newfile = new TFile("/gs/project/atlas-mcgill/ssaha/AnalysisTop-2.4.17_NonPrompt/merged_ttbar_R_new.root");

I have two new trees ttHTree and ttbarTree, which is what I need. Will the code work fine if I just have two new trees and the rest of the code unchanged to get the histos ?

It is working if I comment out the lines you mentioned.

Thanks a lot !
Best,
Shreya

Hi Shreya,

[quote] just need to create two new trees and print the histograms. I have commented out the line below [/quote]How are you using those 2 trees and histograms? (i.e. what is the purpose, how long do you need them? Depending on how you are using the trees, if you only need them in memory, it might (or might not) be even more efficient to simply use the original trees directly.

Cheers,
Philippe.