as a colleague tried to open 2300 Root files (not at once, but in blocks of 5 at a time) in one
application, he ran out of memory. I looked somewhat closer to this problem and observed
that the memory footprint of the application goes up by ~15MByte with each TFile::Open(),
even if the previous TFile got closed and deleted. That’s not a big deal if you loop over
10 files, but for some thousand files that’s too much.
My question: Is there a known memory leak for TFile::Open()/Close()/~, or is there
definitely no problem and I have to search the 15 MByte elsewhere?
I tracked down my (our) memory leak somewhat more. Running in valgrind with different
numbers of files I found some one-per-file leaks. This is the largest one:
==32512== 14,086,692 bytes in 4,088 blocks are still reachable in loss record 245 of 246
==32512== at 0x401AE0A: operator new (vg_replace_malloc.c:197)
==32512== by 0x420EE2D: TStorage::Alloc(unsigned) (in /afs/slac.stanford.edu/package/cernroot/vol19/510\00c/Linux24SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x425D7D3: TObjArray::Init(int, int) (in /afs/slac.stanford.edu/package/cernroot/vol19/510\00c/Linux24SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x425C935: TObjArray::TObjArray(int, int) (in /afs/slac.stanford.edu/package/cernroot/vol1\9/51000c/Linux24SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x4247089: TClonesArray::TClonesArray(char const*, int, bool) (in /afs/slac.stanford.edu/p\ackage/cernroot/vol19/51000c/Linux24SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x8973A9B: KanTClones::KanTClones(KanClonesVector&, char const*) (in /afs/slac.stanford.ed\u/g/babar/build/s/steinke/tstAnal26/bin/Linux24SL3_i386_gcc323/KanTestUtil)
==32512== by 0x896C4CA: KanClonesVector::Streamer(TBuffer&) (in /afs/slac.stanford.edu/g/babar/build/s/\steinke/tstAnal26/bin/Linux24SL3_i386_gcc323/KanTestUtil)
==32512== by 0x826A81C: KanClonesVector_I::Streamer(TBuffer&) (in /afs/slac.stanford.edu/g/ba\bar/build/s/steinke/tstAnal26/bin/Linux24SL3_i386_gcc323/KanTestUtil)
==32512== by 0x4275765: TClass::Streamer(void*, TBuffer&) (in /afs/slac.stanford.edu/package/cernroot/v\ol19/51000c/Linux24SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x4275958: TClass::Streamer(void*, TBuffer&) (in /afs/slac.stanford.edu/package/cernroot/v\ol19/51000c/Linux24SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x41B59E3: TBuffer::ReadObjectAny(TClass const*) (in /afs/slac.stanford.edu/package/cernro\ot/vol19/51000c/Linux24SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x425D4F3: TObjArray::Streamer(TBuffer&) (in /afs/slac.stanford.edu/package/cernroot/vol19/51000c/Linux24SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x89560C8: KanTreeBase::Streamer(TBuffer&) (in /afs/slac.stanford.edu/g/babar/build/s/stei\nke/tstAnal26/bin/Linux24SL3_i386_gcc323/KanTestUtil)
==32512== by 0x88DCA6D: KanEventTree::Streamer(TBuffer&) (KanEventCint.cc:2461)
==32512== by 0x41D41CE: TKey::ReadObj() (in /afs/slac.stanford.edu/package/cernroot/vol19/51000c/Linux2\4SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x41BEDF0: TDirectory::FindObjectAny(char const*) const (in /afs/slac.stanford.edu/package/cernroot/vol19/51000c/Linux24SL3_i386_gcc323/lib/libCore.so)
==32512== by 0x88CBD19: KanEventReaderImpl::setReadTree(TFile&, char const*, KanCompMap::Index, bool) (\KanEventReaderImpl.cc:129)
Do you have an idea what we have to do to avoid to run into this leak in TStorage::Alloc()?
Thanks,
Matthias
[quote=“matthiasSteinke”]Hi Rene,
it’s ROOT 4.03.02. I’ll try to provide a minimal application that reproduces the effect. Give
me a day or so.
Matthias
[quote=“brun”]Matthias,
Which version of ROOT?
Can you reproduce the problem by opening /closing 5 times the same file?
If yes, could you post this file in some public area?
it looks as if KanTClones::KanTClones(KanClonesVector&, char const*) is your default constructor. And it seems as if you allocate memory, either by calling new TClonesArray in there, or because KanTClones deriuves from TClonesArray. Alas, the default constructor must not allocate memory.
You can introduce a default constructor taking no arguments and remove the default value of the first parameter of KanTClones::KanTClones(KanClonesVector&, char const*); in this new default constructor you simply initialize all pointers to 0., which should be sufficient.
Instead of changing your default constructor, you can also add a new contructor:KanTClones::KanTClones(TRootIOCtor*);where you simply initialize all pointers to 0., which should be sufficient.
your recipe did not work. The valgrind output is still the same. Next I’ll try Axel’s proposal.
Matthias
[quote=“pcanal”]Instead of changing your default constructor, you can also add a new contructor:KanTClones::KanTClones(TRootIOCtor*);where you simply initialize all pointers to 0., which should be sufficient.
KanTClones has no explicit default ctor; there are no default arguments in
KanTClones::KanTClones(KanClonesVector&, char const*). Is your suggestion still
worth a try?
Matthias
[quote=“Axel”]Hi,
it looks as if KanTClones::KanTClones(KanClonesVector&, char const*) is your default constructor. And it seems as if you allocate memory, either by calling new TClonesArray in there, or because KanTClones deriuves from TClonesArray. Alas, the default constructor must not allocate memory.
You can introduce a default constructor taking no arguments and remove the default value of the first parameter of KanTClones::KanTClones(KanClonesVector&, char const*); in this new default constructor you simply initialize all pointers to 0., which should be sufficient.
in your backtrace you have the streamer calling
KanTClones::KanTClones(KanClonesVector&, char const*)
I can’t find that in your code. Why? Is SetActive inline, calling this constructor?
in your backtrace you have the streamer calling
KanTClones::KanTClones(KanClonesVector&, char const*)
I can’t find that in your code. Why? Is SetActive inline, calling this constructor?
Axel.[/quote]
the KanTClones ctor is called in KanClonesVector::Streamer():
yes, that’s the streamer I was asking for Now, there are a few problems.
you shouldn’t inherit from TClonesArray. This class is highly optimized, and it’s very difficult to inherit without breaking this optimization. It’s a lot better to contain a TClonesArray.
you are circomventing TClonesArray’s optimization by deleting and re-creating the array when reading it. The whole point of a TClonesArray is to re-use memory. So you should check whether _array’s KanClonesVector is set to *this, and whether it’s objectClassName() is the one you have set for the vector. If so, don’t touch it. If not:
you should call “delete _array”, not _array->Delete(). The two are different; the latter will leave the TClonesArray object around, deleting all the containees. Which is the cause of your memory leak.
thanks for the hints. I’ll go through everything as fast as it’s possilble while listening
to talks in parallel. I’ll let you know if it works (or not).
Matthias
[quote=“Axel”]Hi,
yes, that’s the streamer I was asking for Now, there are a few problems.
you shouldn’t inherit from TClonesArray. This class is highly optimized, and it’s very difficult to inherit without breaking this optimization. It’s a lot better to contain a TClonesArray.
you are circomventing TClonesArray’s optimization by deleting and re-creating the array when reading it. The whole point of a TClonesArray is to re-use memory. So you should check whether _array’s KanClonesVector is set to *this, and whether it’s objectClassName() is the one you have set for the vector. If so, don’t touch it. If not:
you should call “delete _array”, not _array->Delete(). The two are different; the latter will leave the TClonesArray object around, deleting all the containees. Which is the cause of your memory leak.
Now, there are a few problems.