Possible memory leak on PROOF

ROOT
1

Hi,

I’m using PROOF with the last trunk version.

When processing data, memory consumption of proofserv.exe processes keeps increasing.

I’m pretty sure that it’s not a problem of my analysis code because the memory consumption lower either with ProofLite or without PROOF at all.

I’ve tried to use gObjectTable->Print(); while enabling

Root.MemStat: 1
Root.MemStat.size: -1
Root.MemStat.cnt: -1
Root.ObjectStat: 1

in .xrootrc file of cluster, but the total size of the objects reported is about 1M while memory consumption of proofserv.exe is about 1G.

 3996 proof     18   0 1061m 570m  20m R 44.9  7.1   1:07.39 proofserv.exe                                                                                   
 3998 proof     18   0 1090m 556m  20m R 69.4  7.0   1:09.84 proofserv.exe                                                                                   
 3994 proof     18   0  768m 415m  20m R 77.8  5.2   1:08.46 proofserv.exe                                                                                   
 4000 proof     18   0  772m 412m  20m R 67.5  5.2   1:10.23 proofserv.exe                                                                                   
 4002 proof     18   0  845m 408m  20m D 62.5  5.1   1:03.00 proofserv.exe                                                                                   
 3990 proof     18   0  741m 394m  20m R 72.1  4.9   1:07.76 proofserv.exe                                                                                   
 3992 proof     18   0  805m 393m  20m R 45.9  4.9   1:06.25 proofserv.exe

For comparison, the following is the same setup,machine and everything with ProofLite:

 4620 proof     25   0  752m 422m  23m D 82.1  5.3   1:57.19 proofserv.exe                                                                                   
 4614 proof     19   0  720m 419m  23m R 50.5  5.3   1:46.93 proofserv.exe                                                                                   
 4612 proof     19   0  754m 417m  23m D 70.2  5.2   1:33.13 proofserv.exe                                                                                   
 4610 proof     18   0  758m 412m  23m R 74.2  5.2   1:40.50 proofserv.exe                                                                                   
 4608 proof     24   0  724m 404m  23m R 83.1  5.1   1:47.13 proofserv.exe                                                                                   
 4616 proof     18   0  713m 403m  23m D 48.2  5.1   1:44.46 proofserv.exe                                                                                   
 4618 proof     18   0  723m 401m  23m R 61.5  5.0   1:55.94 proofserv.exe

These proofserv.exe process also are alive for ten or so minutes after analysis is finished. Their memory consumption stays the same.

–
Best wishes,
Eugeny Boger

2

Hi,
Can you run the workers inside valgrind?
See root.cern.ch/drupal/content/runn … y-valgrind.

Gerri

3

Hi,

Nothing suspicious, the results look like “definitely lost: 9,194 bytes in 15 blocks.”

I used “valgrind=workers” option.

4

Hi,

Did you use the ‘–leak-check=full’ option?

TProof::AddEnvVar("PROOF_WRAPPERCMD", "valgrind_opts:--leak-check=full")

Gerri

5

Hi,

I’ve just tried with leak-check=full, still no success. The results are pretty same (definitely lost: 4,271 bytes in 5 blocks.).

By the way, the valgrind logs weren’t returned by PROOF so I had to find it in last-worker-session/valgrind.log. files. I believe it’s due to proofserver.exe was still alive during actual session termination.

–

6

Hi,

It is rather weird that you have such a big leak and nothing shows up in valgrind.
Did you quit the session and retrieved the logs with TProof::LogViewer after restarting ROOT?
(See ‘Caveats’ section in the doc page: the comment was already there but now I have created a section with it).
Information about memory usage is written when quitting the process.

Gerri

7

Hi,

I have a access to remote PROOF machine, so I just looked through .valgrind.log files in session directory.