ROOT's typical memory use

ROOT
1

Hi all.

I have noticed some things about memory allocation while using ROOT and I’d like to know 1) if what I am seeing is typical behavior and 2) if there’s a way to limit how much memory ROOT uses.

I have a relatively simple Event class with member varialbes of the standard types and no arrays/TClonesArrays. I then make a simple program to open a TFile, a TTree, fill members of an instance of my Event class with some values, TTree::Fill(), and then write the file (standard event loop…). At first, it allocates about 20 MB and hums along fine. But after about one million events (1000000), the memory allocation quickly jumps up to 200 MB and then up to 600, 700 MB and more. I can reproduce this behavior on my system by using the modified versions of Event.cxx/.h and MainEvent.cxx (found in ROOTSYS/test) and running “./Event 2000000 1 99 1”. (In this modified code, its just filling the Event class without any Tracks and just setting one member variable.). On my system (root v5.26 and mac os 10.6), the program initially allocates ~17 MB, but after about 800000 events, that memory use goes up to hundreds of MB.

Is this normal behavior? I suspect that this memory allocation is causes by ROOT trying to compress this data and write it to disk, but the for-loop is quickly creating a large number of events and ROOT can’t keep up. Is this the case? And does ROOT know that my system has ~1 GB of free space in memory and that it can safely allocate that memory as it needs to quickly perform the compression?

Also, is there a way of limiting how much memory ROOT can use?

I started looking at this issue because I was trying to track down a memory leak. At least, I thought it was a memory leak, but I’m not so sure now.

Thanks for your help.

Adam
Event.cxx (14.4 KB)
Event.h (7.72 KB)
MainEvent.cxx (10.6 KB)

2

I cannot reproduce this problem. When running on macos10.5 with your files and Event 10000000 1 99 1 the memory goes up to about 70 MBytes and not more. The program produces a file Event.root of 290 MBytes with a compression factor of 34.2 and with 56 branches.

Rene

3

Hi.

So, I updated to 5.26.00b and this problem seems to have gone away for me now.

Thanks for your help - sorry I wasn’t using the latest-latest version.

Adam

4

Hi Rene.

So, indeed with v5.25.00b, I no longer have a large memory leak issue when I fill 1,000,000+ “simple” events in a tree.

But now, I have this problem again when a class that I store in a TClonesArray has a c++ string member variable.

Using the example in ROOTSYS/test, I added a member variable to the Track class called ‘string fStringName’ and I just set it to some constant value in the constructor. I then added a fStringName.clear() call within the Track::Clear method. When I run ./Event 10000000 1 99 1, the memory allocation goes up significantly.

I can reproduce this problem on two different systems: mac os 10.6.3 / root v5.26.00b and on a linux machine (kernel 2.6.24-16, x86_64) / root v5.26.00b.

In addition, I see the same behavior when I change from a c++ string to a TString (and call TString::Resize(0) within the Track::Clear method).

I have attached Event.cxx and Event.h, which you should be able to drop into your ROOTSYS/test directory, compile and then run ./Event 10000000 1 99 1.

Thanks for your help.
Adam
Event.h (7.78 KB)
Event.cxx (14.4 KB)

5

Hi,

In the Clear method, you will need to call std::string::~string or TString::~TString to properly relinquish the memory (because you later re-initialize it with a call to the constructor of Track which call the constructor of std::string or TString which unconditionally allocate new resources.

Cheers,
Philippe.

6

Hi Philippe, thanks for your response.

This works with the ROOTSYS example, but only because the Event class doesn’t delete the TClonesArray fTrack (fgTrack) when its destroyed. I can’t have a static TClonesArray because in my code I will need to have multiple instances of my Event class within a single program. So, in my code, I have to delete the Event class’s instance of TClonesArray. When the TClonesArray is deleted, it calls the Track destructor, which then breaks because I’ve already deleted my string or TString inside the Track::Clear().

So, I tried two things. First, I decided to stop using TClonesArray::Clear, but instead use TClonesArray::Delete(). I removed the TString::~TString call inside my object’s Clear method. In this way, I could write out my events without a memory leak. Yes, calling TClonesArray::Delete is slower, but I didn’t notice much (if any) difference in processing time, and its no so much of an issue for me.

The second option that seemed to work was using a TString* and making sure to appropriately call new and delete when necessary in my object’s Clear() and destructor methods.

In both cases, I was able to write out my ROOT file without any memory leaks or breaks. Either way seems to “work”.

However, now I have a new problem. I’ve attached my output root file (new.root) for you to look at. When I call TTree::Scan(“fDetectorName.fData”) it breaks when I have an Event where there are no objects in the TClonesArray where fDetectorName is located.

You can reproduce this by
TFile f(“new.root”)
t->Scan(“fDetectorName.fData”);

You’ll see that on entry 32, it breaks because there are no objects in my branch fSingleBoloSubRecs (which was branched from a TClonesArray holding instances of my class EdwHLASingleBoloSubRecord.

The other option is to use a variable length array of Char_t* fDetectorName. But, the problem with this seems to be that I can’t use t->Scan("","fDetectorName==“FID401"”)

I really appreciate your help on this. Thanks,
Adam
new.root (1.04 MB)

7

Hi,

I can indeed reproduce the problem. I have not yet found a good solution/work around.

Philippe.

8

Hi,

The problem is solved by r33330 of the trunk. You can work around the problem by using the command:t->Scan("fDetectorName.fData","Length$(fDetectorName[0])>0","");

Cheers,
Philippe.

9

[quote]The other option is to use a variable length array of Char_t* fDetectorName. But, the problem with this seems to be that I can’t use t->Scan("","fDetectorName==“FID401"”) [/quote]As far as I can tell it does work.

For example with your new.root files:[code]root [1] t->Scan(“fDetectorName.fData”,“fDetectorName.fData==“ID2"”,”")

  • Row * Instance * fDetector *
  •    1 *        1 *       ID2 *

  •    1 *        2 *       ID2 *

  •    1 *        3 *       ID2 *

  •    2 *        1 *       ID2 *

  •    2 *        2 *       ID2 *

  •    8 *        1 *       ID2 *

  •    9 *        1 *       ID2 *[/code]where fData is a char* member of TString.

Cheers,
Philippe.

10

[quote]So, in my code, I have to delete the Event class’s instance of TClonesArray. When the TClonesArray is deleted, it calls the Track destructor, which then breaks because I’ve already deleted my string or TString inside the Track::Clear(). [/quote]Unfortunately your are correct and indeed using Delete or better yet a TString* are two possible solution.

Cheers,
Philippe.