Memory issue

ROOT
1

Dear ROOT experts,
I have found the following weird memory issue.
The problem is that I observe a very different memory usage when accessing trees that should be very similar.

I use the following code:

{
  static ProcInfo_t info;
  const float toMB = 1.f/1024.f;
  
  TFile *f = new TFile("file1.root");
  TTree *t = (TTree*)f->Get("tree");
  gSystem->GetProcInfo(&info);
  printf(" res  memory = %g Mbytes\n", info.fMemResident*toMB);
  printf(" vir  memory = %g Mbytes\n", info.fMemVirtual*toMB);
  
  t->GetEntry(123456);
  gSystem->GetProcInfo(&info);
  printf(" res  memory = %g Mbytes\n", info.fMemResident*toMB);
  printf(" vir  memory = %g Mbytes\n", info.fMemVirtual*toMB);
}

when I run it on file1.root, the output is:

[quote] res memory = 110.797 Mbytes
vir memory = 248.898 Mbytes
res memory = 206.938 Mbytes
vir memory = 344.91 Mbytes
[/quote]
while if i run it on file2.root, i get:

[quote] res memory = 110.492 Mbytes
vir memory = 248.5 Mbytes
res memory = 243.832 Mbytes
vir memory = 1431.67 Mbytes
[/quote]

So >1GB difference between the two.

Both files are of similar size (2.9 and 2.8GB respectively), containing the same tree structure, and produced from the similar input files but just a different subset of them.

The Print() method for the tree in file1.root shows:

[quote]*Entries : 5388513 : Total = 11303126030 bytes File Size = 3021195181 *

  •    :          : Tree compression factor =   3.74                       *

[/quote]
while for file1.root:

[quote]*Entries : 5333526 : Total = 11178390906 bytes File Size = 2980833544 *

  •    :          : Tree compression factor =   3.75                       *

[/quote]
What makes this significant difference in memory usage?

The question is relevant cause when I use these files as inputs to my batch jobs, I might hit the memory threshold of the batch system when the memory usage is unexpectedly large.

Another maybe useful and weird piece of information, is that if I do an hadd of file1+file2, I don’t see any unexpected increase in memory usage, even if I get an entry corresponding to file2. But if I hadd in the opposite order file2+file1, I see the unexpectedly high memory usage.

Any help would be appreciated.
Bruno

2

Hi Bruno,

the huge difference is in VSize, despite a non negligible increase in RSS, 40MB. This could be due to the loading of libraries but the last piece of information you attach seems to contradict this ansatz (hadd of f1+f2 and hadd of f2+f1).
What version of ROOT are you using? What is stored in these files? (ROOT objects, custom objects whose dictionaries are in libraries…)
In addition note that GetProcInfo reads files on linux which are not updated “live”. It would be nice to add a sleep, say 0.5 seconds to be sure, before reading the memory values.

Cheers,
Danilo

3

Hi Danilo,
Thanks for the prompt reply!
(and sorry for my delay cause I thought i was subscribed but i wasn’t).

I’m using ROOT 6.02/05. The file contains only a tree with branches that are just ints, floats, and arrays of ints/floats. Nothing custom.
Regarding the “live” update of the memory. Actually, initially I saw this problem when doing a loop over events, and I did the memory tracking using ‘pmap’ on the terminal every few seconds. What I saw was a big jump of memory usage at the beginning of the loop, and afterwards a steady but very slow increase. Again for f1 the memory usage was small (with no significant increase), while for f2 it was unexpectedly large.

Any ideas?

4

Hi Bruno,

I think I need a reproducer.
Can you share the root file and a macro (tree setup, loop with measurement inside) which illustrates this behaviour?

Cheers,
Danilo

5

Hi Danilo,

The files are big. Cernbox is the best solution I found to share them:
cernbox.cern.ch/index.php/s/FCJadk4Gc7xgisi

Please let me know if you prefer any other way.

In the link you can also find a simple macro that reads the file and loops over the tree, printing out the memory status every 500k events.
As a summary, I get the following for f12.root (hadd of f1+f2):

And for f21.root:

Thanks for looking into this,
Bruno

6

Hi Bruno,

I clearly see the effect.
I am investigating.

Cheers,
Danilo

7

Hi Bruno,

could you share the single files separately? Putting them in the same CERNBox dir would be sufficient.
This can help with two more independent measurements.

Cheers,
Danilo

8

Hi Danilo,
I’m not sure I understand, these are already the single original files.
f1.root is the one with the expected behaviour while f2.root is the one with the unexpected memory usage.
I could have included f12.root and f21.root (the hadded ones in different orders), but those add redundant info in very large files. Are those the ones you’d like me to share?

Thanks again for investigating,
Bruno

9

Hi Bruno,

got it: thanks. Those would be useless indeed.
I can see a difference already with those two.

D

10

Hi Bruno,

the behaviour you are seeing is due to the fact that ROOT builds a cache in memory to optimise reading speed.
The size of this cache depends on a number of parameters, among which you have the “autoflush” value (more about this here: root.cern.ch/doc/master/classTT … 9d28188da0).
The size of the cache is proportional to it and in the case of f2, the autoflush value is much bigger than the one in f1 (5255104 and 55860 respectively). Hence the bigger memory usage.

Cheers,
Danilo

11

Hi Danilo,
Sorry, I forgot to mention in the original message that I had played with the AutoFlush and AutoSave options.

If for instance I do:

t->SetAutoFlush(100);

before looping over the events, I still get the same memory usage for both files, with the same large difference among them. Also, SetAutoSave didn’t seem to help.

Am I missing something?

Thanks,
Bruno