I am trying to read a Tree from several ROOT files. After reading several of them, I end up running out of RAM memory and my PC colapses. I made sure to delete, clear and reset every Tree, TFile or Branch adress used in the for cycle, but it does not matter what I do, I always end up with the problem. If instead of reading 50 files I read 10 or 20, the program finishes sucessfully, but after more than ~25, memory is filled.
at the end of each iteration, but the problem still remains. I include a copy of the source code that gives me this issue. Problematic for cycle starts at line 103. I removed small pieces of the code and when removing the Tree_horizontal_strips_whole->Copy part from the cycle seems to avoid the problem, but then of course I cannot obtain the results I need. test.C (12.9 KB)
I really have tried everything I can think of and am desesperate. I believe it may be an internal problem of ROOT itself.
What is your intent here? Is really your intent to copy and keep in memory a subset of the entries of the original tree? Did you measure that you have enough memory to hold all those copies? (the code seems to make partial copy of the horizontal stip trees for each (!!!) entry of the vertical trees … unless the condition are very very selectively … the combinatorics seems like they will grow out of proportion very quickly).
The usual techniques is to not copy the TTree (which are assumed to be very large and not fit in memory) but to scan through them with condition, possibly limiting the entries looked at by using a TEntryList.
To delegate the management of the loops needed (and enable the processing in parallel), you may want to investigate whether you can rewrite your code using RDataFrame (and possibly it 'systematics` feature).
Vertical tree has many particles. I have to look for coincideces within all particles in Horizontal tree. But those coincidences must happen with particles with the same BunchNumber property, so yes, for every particle in Vertical I make a partial copy of Horizontal (whole) with only those particles that have the same BunchNumber that the particle I picked from Vertical Tree. I do not have a problem remaking this Tree over and over, I have memory enough to hold it (because condition is very selective as you point out). In fact I can do it for several ROOT files with no problem, but problem comes when I analyze more than 20-30 on a row and these partial horizontal trees start to stack in memory even though I delete them at the end of each iteration.
I wrote a simple reproducible that mimics your code logic of memory management and I cannot reproduce memory leak/increase in RAM usage overtime on the ROOT side from CopyTree.
Here is the code, you can copy it and test yourself, with a simple root test.C
// test.C
int parseLine(char* line){
// This assumes that a digit will be found and the line ends in " Kb".
int i = strlen(line);
const char* p = line;
while (*p <'0' || *p > '9') p++;
line[i-3] = '\0';
i = atoi(p);
return i;
}
int getMemory(){ //Note: this value is in KB!
FILE* file = fopen("/proc/self/status", "r");
int result = -1;
char line[128];
while (fgets(line, 128, file) != NULL){
if (strncmp(line, "VmRSS:", 6) == 0){
result = parseLine(line);
break;
}
}
fclose(file);
return result;
}
void create_file(){
// for a full reproducible
auto n_entries = 1000000;
auto df = ROOT::RDataFrame(n_entries).Define("x", "rdfentry_").Define("y", "x*x");
df.Snapshot("events", "test_file.root");
std::cout<<"Data size in a TTree: "<<n_entries*2*8/1024.<<" kB"<<std::endl;
}
void test(){
create_file();
auto ref_memory = getMemory();
std::cout<<"Program uses: "<<ref_memory<<" kB"<<std::endl;
for(int i=0; i<100; i++){
// Consider using unique_ptr from modern C++ and avoid manual memory management with new and delete if possible.
// std::unique_ptr<TFile> file( TFile::Open("test_file.root") );
// auto tree = file->Get<TTree>("events");
TFile *file = new TFile("test_file.root", "READ");
TTree *tree = (TTree*) file->Get("events");
TTree *copy; // = nullptr;
auto entry = tree->GetEntry(1); // just in case...
gROOT->cd();
std::cout<<"+ loading initial tree: "<<getMemory()-ref_memory<<" kB"<<std::endl;
copy = tree->CopyTree("x > 42");
auto copy_entry = copy->GetEntry(1); // just in case...
std::cout<<"+ copy "<<i<<": "<<getMemory()-ref_memory<<" kB"<<std::endl;
tree->ResetBranchAddresses(); // you don't really need this
delete tree;
delete file;
copy->ResetBranchAddresses(); // you don't really need this
delete copy;
// And the rest of "delete" attempts you also don't need...
std::cout<<"+ cleanup "<<i<<": "<<getMemory()-ref_memory<<" kB (should not grow)"<<std::endl;
}
}
I have noted that you have quite a lot of histograms…
To be precise 16*16*8 + 32*8 = 2304 in total of TH1D type of which 320 you continuously fill keeping their data between the files.
That is quite a lot
Are you sure this is not a bottleneck for your RAM?
To check this, you can try using TH1F instead of TH1D, it should save half of the RAM for you and you can check if you are able to reach past 20-25 files.
Otherwise, you can try to remove a couple of hundreds of unused histograms temporarily until you need them
If you still see a significant increase in memory after trying this out, then it might be something else. Then maybe you can try to make a reproducible following my example?
Just to point out, I would completely agree with @pcanal that using CopyTree() is very unconventional nowadays. If you have the ability and time, I would really advise to rethink the structure of the ROOT files and try to use RDataFrame for the analysis as much as possible.
E.g. vertical/horizontal/BunchNumber sound to me like an event and hit properties that I should easily be able to Filter within a single TTree w/o creating additional TTree for every bunch.
yes, I am considering remaking my entire programme, I will look into what you just recommended me.
However, as for the histograms that you mention… It is true I have a lot of them, but once they are created, the memory they consume should remain constant, should it not? I mean, at start of my program, all values will be 0, and at the end they will be 0, 1, 2, 8320432, or whatever, but how is it that the required memory would grow bigger as they are filled with more and more data?
Still, if your histograms take, e.g. 95% of available RAM, and data in 0-19 files small enough to fit in the remaining 5%, even small increase in the amount of data in runs 20-25 might cause out of memory problems.
Also this might depend if other software, e.g internet browser, or other users, if you are on a shared cluster use the RAM…
I had never used TEntryList, but just read its documentation and I already changed it. It is working perfectly fine, with no increasing RAM requirements, so I guess it is solved.
The change I had to make was actually very small (just using the Entry List instead of a partial sub Tree), so it really was the second Tree what was causing the issue. I still do not know why it was not being properly deleted, but thanks everybody!
Yes, but inside that loop the Tree is always rewritten, so the content of the previous iteration would disappear, would it not? Besides, at the end of the loop the tree is deleted entirely to begin again with another ROOTfile, so in any case I should see an increase in RAM during the loop and then a sudden drop of it, but not the case.
Tree_horizontal_strips is a simple TTree*, so the TTree it was pointing to before is simply forgotten/lost and not deleted. (if it was a std::unique_ptr<TTree> you would be right)
Besides, at the end of the loop the tree is deleted entirely to begin again with another ROOTfile,
No. All the lost TTree from iteration k==1 to k==Tree_vertical_strips->GetEntries()-1 are never ever deleted, they are literally the leak you were looking for.
