Possible memory leak on PROOF


I’m using PROOF with the last trunk version.

When processing data, memory consumption of proofserv.exe processes keeps increasing.

I’m pretty sure that it’s not a problem of my analysis code because the memory consumption lower either with ProofLite or without PROOF at all.

I’ve tried to use gObjectTable->Print(); while enabling

Root.MemStat: 1
Root.MemStat.size: -1
Root.MemStat.cnt: -1
Root.ObjectStat: 1

in .xrootrc file of cluster, but the total size of the objects reported is about 1M while memory consumption of proofserv.exe is about 1G.

 3996 proof     18   0 1061m 570m  20m R 44.9  7.1   1:07.39 proofserv.exe                                                                                   
 3998 proof     18   0 1090m 556m  20m R 69.4  7.0   1:09.84 proofserv.exe                                                                                   
 3994 proof     18   0  768m 415m  20m R 77.8  5.2   1:08.46 proofserv.exe                                                                                   
 4000 proof     18   0  772m 412m  20m R 67.5  5.2   1:10.23 proofserv.exe                                                                                   
 4002 proof     18   0  845m 408m  20m D 62.5  5.1   1:03.00 proofserv.exe                                                                                   
 3990 proof     18   0  741m 394m  20m R 72.1  4.9   1:07.76 proofserv.exe                                                                                   
 3992 proof     18   0  805m 393m  20m R 45.9  4.9   1:06.25 proofserv.exe                                                                                   

For comparison, the following is the same setup,machine and everything with ProofLite:

 4620 proof     25   0  752m 422m  23m D 82.1  5.3   1:57.19 proofserv.exe                                                                                   
 4614 proof     19   0  720m 419m  23m R 50.5  5.3   1:46.93 proofserv.exe                                                                                   
 4612 proof     19   0  754m 417m  23m D 70.2  5.2   1:33.13 proofserv.exe                                                                                   
 4610 proof     18   0  758m 412m  23m R 74.2  5.2   1:40.50 proofserv.exe                                                                                   
 4608 proof     24   0  724m 404m  23m R 83.1  5.1   1:47.13 proofserv.exe                                                                                   
 4616 proof     18   0  713m 403m  23m D 48.2  5.1   1:44.46 proofserv.exe                                                                                   
 4618 proof     18   0  723m 401m  23m R 61.5  5.0   1:55.94 proofserv.exe                                                                                   

These proofserv.exe process also are alive for ten or so minutes after analysis is finished. Their memory consumption stays the same.

Best wishes,
Eugeny Boger

Can you run the workers inside valgrind?
See root.cern.ch/drupal/content/runn … y-valgrind.



Nothing suspicious, the results look like “definitely lost: 9,194 bytes in 15 blocks.”

I used “valgrind=workers” option.


Did you use the ‘–leak-check=full’ option?

TProof::AddEnvVar("PROOF_WRAPPERCMD", "valgrind_opts:--leak-check=full")



I’ve just tried with leak-check=full, still no success. The results are pretty same (definitely lost: 4,271 bytes in 5 blocks.).

By the way, the valgrind logs weren’t returned by PROOF so I had to find it in last-worker-session/valgrind.log. files. I believe it’s due to proofserver.exe was still alive during actual session termination.


It is rather weird that you have such a big leak and nothing shows up in valgrind.
Did you quit the session and retrieved the logs with TProof::LogViewer after restarting ROOT?
(See ‘Caveats’ section in the doc page: the comment was already there but now I have created a section with it).
Information about memory usage is written when quitting the process.



I have a access to remote PROOF machine, so I just looked through .valgrind.log files in session directory.