Memory leak

ROOT
1

Hi,

I’m running root to perform task on severl dat-files with each 100 MB. The dat-files are written by a TNT2 card and are interpreted using a class (“TNT2pulseshape”) which is a public TObject and is loaded at the beginning of the macro.

My problem is now: whenever I submit to many files to the macro (or the standalone app which I made out of this macro) I run into memory problem of my machine: the memory quickly files up and soon the machine is almost unresponsible.
The data of the machine (which should be enough for this task):

  • Mandriva 10.1
  • dual Intel® Xeon™ CPU 2.80GHz
  • 1 GB RAM
  • swap 2 GB

The procesing steps of the files are:

They are handed over as an argument to the app (in the case of the compiled binary) where they are (in a for loop as long as their are arguments left) )handed over to the TNT2-class:

strcpy(infile,argv[argc_up]);
TNT2pulseshape ps(infile,"TNT2");

Than I make in three fits for each event which is in the file (there are around 6k events each file each channel, I do only process one specific channel at the moment) with a while loop until there is no event left, and get back into the for loop to process the next argument.

I have the feeling that I miss it to “close” the file somehow somewhere, but I do not know how to do that or where to do that. If it is the task of the tnt2pulseshape class I can have a closer look at this - I didn’t wrote it by myself but have access to the sourcecode and can post it here if it makes sense, I think.

liquidat

2

Please send the shortest possible running setup/script reproducing your problem.

Rene

3

Hi Rene,

sorry for not adding this by my own, learned something more. Here is the code piece. But to run this program you still need a set of these 100MB dat-files and the library to access them. I can upload this somewhere if you want to…

A short explanation about what is happening here in the modified program version:

I start the program with a set of files as an argument and go through each event of each file (roughly 6000 events per file). For each event I make three pol0 fits, here with fixed areas (0 to 2.5, 2.5 to 5, 5 to 20). That’s it.
After this program finished the second file the machine is already swapping heavily.
tof_channel4_analyzer_memory_problem.cpp (1.48 KB)

4

You did not send the necessary information, in particular your class TNT2pulshape, so I can only guess.
If you have many events, your code will effectively leak a lot!
I suggest the following changes

[code]#include
#include
#include
#include
#include
#include
#include
//ROOT-includes
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
//CLHEP-includes
#include “TNT2pulseshape.h”

//TROOT hej(“hej”,“hej”); <==you do not need this line

int main(int argc, char** argv) {
Double_t fit_param_middle;
Double_t fit_param_middle_err;
Double_t fit_param_left;
Double_t fit_param_left_err;
Double_t fit_param_right;
Double_t fit_param_right_err;

Double_t amplitude_difference;
Int_t event_counter = 0.0;
char infile[80];

int argcROOT = 1;
char* argvROOT = argv[0];
TApplication theApp(“App”, &argcROOT, &argvROOT);
TF1 *pol0_fit_middle = new TF1(“pol0_fit_middle”,“pol0”,2.5,5);
TF1 *pol0_fit_left = new TF1(“pol0_fit_left”,“pol0”,0,2.5);
TF1 *pol0_fit_right = new TF1(“pol0_fit_right”,“pol0”,5,20);

for(int argc_up=1;argc_up<argc;argc_up++) {
strcpy(infile,argv[argc_up]);
TNT2pulseshape *ps = new TNT2pulseshape(infile,“TNT2”);

cout << "Processing file " << infile << "\n";

while(ps.getNextEvent()) {
  ps->getPS(3)->Fit("pol0_fit_middle","QRM");
  ps->getPS(3)->Fit("pol0_fit_left","QRM");
  ps->getPS(3)->Fit("pol0_fit_right","QRM");
}
delete ps;

}
return 0;
}
[/code]

Rene

5

Thanks for the detailed and very fast answer, Rene.

About the needed information: I misunderstood you and thought you just want to have a look at the bare script I’m using and just at the root-functions themselves which are used.

I talked to my project coordinator if I can publish files and the used library (the tnt2pulseshape), but the answer needs some time. It shouldn’t be a problem though, it’s just a question of time. Would it be ok if I provide a link to the files? I shouldn’t try attaching 100 MB files here, I think…

liquidat

6

I will never look in 100 Mbytes of source files.
Send the shortest possible running script/setup (maximum a few hundred lines of code) that reproduces your problem.

Rene

7

No, definitely not! The 100 MB are just the data files I process! I got the ok now and shape the scripts down to the minimum which is necessary and post it afterwards. Thanks so far.

liquidat

8

Ok, I think I trimmed down everything as far as possible to have the shortest working setup:

First the script itself: script
As the previous version it takes several data files as arguments. A for loop loads each file, and a while loop processes each event in each file (roughly 6k events per file).
Every while loop there is first a constant fraction done, followed by three fits with ranges determined by the constant fraction (this was different in the first script which I posted since I tried to make it shorter as I should have done).
A counter counts all processed events and gives the number out every hundred events to show that the program is still working, and to give a basic idea how fast it is working.
The script looks for the libtnt2pulseshape.so file which should be compiled/included in the system (with ldconfig and a file in /etc/ld.so.conf.d/. for example).

The tnt2pulseshape-class: Class
This class is used by the script. The important part here is: as mentioned I didn’t write this class, it was used by someone else who already worked with the tnt-card and with root (but with different tasks). I asked the original author if has an idea where the problem could be and he mentioned that probably the copies of the functions in the fit are not deleted. However, if I fit with the option “N”, nothing changes, I still get the memory leak.
There is a compiled version inside which is compiled on a 586 Mandriva system, but it is probably better to recompile (Makefile included, just enter “make”).

The data files: data-file
The data file 100 MB big, this is a tar.bz-version which has roughly 30 MB. If you want to see the effect of the memory leak, just start the programm and give this file 5 or 6 times as an argument. Normally I use diferent files obviously, but it works also with the same file as an argument several times. When the program starts you will see the “Event-Counter”-output and pretty soon you will see the memory leak effect in top for example.

I know that this is very much, and if it’s to much I would understand. Nevertheless thank you very much for helping so far and the offer to help - and especially thank you very much for your patience :slight_smile:

liquidat

9

Hi,

you could’ve at least tried to apply Rene’s suggestions :wink: You create 3 TF1 objects per event, which you aparently only need within each iteration. That’s indeed what people call a mem leak. You might want to delete them at the end of each event. I didn’t test anything, nor did I try to understand anything you do - so beware.

Axel.

10

Hi Axel, thanks for the answer.

And yes, you were right - I first didn’t integrate Rene’s suggestion because I had to create the ranges of the fits live inside the loop, and thought that therefore I have to declare the ranges together with the fit function. And now it dropped from my eyes, I realized that I was mad or crazy (or both), quickly integrated a SetRange(), and now the memory usage of the program is somewhere close to 2.7% and stays there, according to top.

And I’m officially dumb now - I’m awfully sorry for the trouble I caused since it is my own fault and I was just to blind to see it.
I don’t know how I can make good this lost time for you, but I will try to keep an eye on the forum to give at least a little bit back.

Thanks again,

liquidat