Mem leak even with suppression file

jbb · July 26, 2024, 7:47am

Bonjour

I’m coming back for a small question about memory leak: as we’re developping and using a platform based on ROOT (thanks for the great work you’re doing by the way) we’re monitoring our own memory leaks in our CI/CD approach using your suppression file. Moving from 6.26/06 to 6.28/04 we’ve still a very large increase of mem leaks.

It seems that even a simple piece code is now producing “mem leaks” which prevent us to get a clear way to track down our own mem leaks :P.

To reproduce this I’m providing a very simple C++ code (CreateTree.C attached to this note) that create a tree, fill it, and write it down in a file.
Compiling this with a simple line (provided in a makefile as well)
g++ -g -o createTree CreateTree.C `root-config --cflags --libs`

I’m getting 54 mem leaks (results provided in createTree_log) when running
valgrind --leak-check=full --show-possibly-lost=no --suppressions=${ROOTSYS}/etc/valgrind-root.supp ./createTree

My question is then : is there something I’m doing wrong ? If not should I create a new suppression file to omit these ? I’ve try with the 6.32/02 the same behaviour can be seen (the log provided has been done with this version).

All remarks / ideas are welcome
Thanks in advance
cheers
jb

CreateTreeTest.tar.gz (5.0 KB)

_ROOT Version: 6.32.02
_Platform: Fedora32
_Compiler: 10.0.1

devajith · July 26, 2024, 7:57am

Hi @jbb,

Thanks for your question. Adding @pcanal in the loop.

Cheers,
Dev

pcanal · July 26, 2024, 8:18am

The post says:

and

Moving from 6.26/06 to 6.28/04

which version of ROOT are you using?

The log that you included indicates that indeed the suppression file is not up to date for your root version and OS.

pcanal · July 26, 2024, 8:32am

Note that the example code is semantically broken, it does (in summary):

tree = new TTree(...);
tree->Fill();
file = TFile::Open(...)
tree->Write(); // At which point all but the last basket are removed from memory since they are now on file.
file->Close();
tree->Draw(); // Will only plot the last few entries.

Unless you really need the data to stay in memory, the usual way of doing this is:

file = TFile::Open(...);
tree = new TTree(...);  // Now the tree is attached to the file
tree->Fill();  // Flush the data to disk regularly, reducing peak memory usage.
tree->Write(); 
// here you can also close, the file and reopen it for reading (and reload the tree).
tree->Draw(); // Will only plot the last few entries.

If you are using the pattern described in the example in your production code, it leads to a huge memory inefficiency (need much more RAM than you actually need) because all the data has to stay in memory until the very end (the tree->Write()).
The usual pattern allows the data to be flush to disk on regular basis (and being removed from memory at that point), keep the memory use of the TTree to a minimal.

jbb · July 26, 2024, 8:33am

The change in behaviour we witness, has been seen moving from 6.26/06 to 6.28/04, and is apparently constant from 6.28/04 onwards.

The log I’ve provided have been generated with 6.32/02.

jbb · July 26, 2024, 8:51am

Sorry this example is a bit messy as I tear down a larger one with functions and stuff that was representative of an older issue (I was a bit lazy to reshuffle it completly).

In the v2 version below, I’m simply doing what you advertise and rerun the valgind test still with 6.32/02. The log is provided as well with 155 “errors” none of which I can connect to a line of my code. My local ROOT version has been compiled in DEBUG mode.

Thanks again for the very quick reply.
cheers
jb

CreateTreeTestv2.tar.gz (824 Bytes)

pcanal · July 26, 2024, 8:59am

@devajith Can you check if the ‘leak’ reported by valgrind like:

==44592== 168 bytes in 1 blocks are definitely lost in loss record 8,982 of 11,476
==44592==    at 0x483A809: malloc (vg_replace_malloc.c:309)
==44592==    by 0x898CD15: clang::Parser::AnnotateTemplateIdToken(clang::OpaquePtr<clang::TemplateName>, clang::TemplateNameKind, clang::CXXScopeSpec&, clang::SourceLocation, clang::UnqualifiedId&, bool, bool) (in /export/home/jb242989/root_v6.32.02/lib/libCling.so.6.32.02)
==44592==    by 0x8921B63: clang::Parser::ParseOptionalCXXScopeSpecifier(clang::CXXScopeSpec&, clang::OpaquePtr<clang::QualType>, bool, bool, bool*, bool, clang::IdentifierInfo**, bool, bool) (in /export/home/jb242989/root_v6.32.02/lib/libCling.so.6.32.02)
==44592==    by 0x899A53C: clang::Parser::TryAnnotateCXXScopeToken(bool) (in /export/home/jb242989/root_v6.32.02/lib/libCling.so.6.32.02)
==44592==    by 0x80E5246: cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const (in /export/home/jb242989/root_v6.32.02/lib/libCling.so.6.32.02)

are one time (not worrisome, need update to suppression file ) or are leak for each call to cling::LookupHelper::findScope (case we would need to fix).

pcanal · July 26, 2024, 9:10am

@jbb given the information I have so far, I can not tell whether your actual problem is a memory leak (the memory consumption would still be high at the very end of the process after the closing fo the file and deletion of the TTree) or memory hoarding (The example CreateTreeTest.tar.gz has such hoarding since the TTree holds on to the data until the explicit call to write).

To have more information, could you valgrind with leak check on your actual use case?

jbb · July 26, 2024, 9:29am

The example in CreateTreeTestv2.tar.gz should be less frightening I guess.

In any case, here I might miss something (sorry if it’s the case), but all tests I’m doing are done with the leak-check option set to full, as mentionned in my first thread (see below).

Thanks for the following
jb

jbb · July 26, 2024, 9:40am

If it can be of any use, here is the log I’m getting (still 6.32/02, g++ 10.0.1) when simply running this code:

``
int main(int argc, char** argv)
{

std::cout << "Generating Tree" << std::endl;
TFile *__lfile = TFile::Open("file.root","RECREATE");

__lfile->Close();
std::cout << "Exiting the code" << std::endl;

}
``
with

valgrind --leak-check=full --show-possibly-lost=no --suppressions=${ROOTSYS}/etc/valgrind-root.supp ./createTree

openingFile.txt (47.1 KB)

cheers
jb

pcanal · July 26, 2024, 10:07am

Indeed. What I am now asking is to run the same with your complete code instead of the tear down version. I.e. From what I see it is possible that the leak I mention at Mem leak even with suppression file - #7 by pcanal is a problem but it is unlikely to be large enough to cause the problem you have.

Note that in your last 2 output, the valgrind output was not included (I do not need it but we will need it for the next iterations).

jbb · July 26, 2024, 11:39am

The lines you mentionned in red are indeed in the log that I’ve re-uploaded in this post (I forgot to redirect all the stream and not just the standard one in my |tee command, sorry about that).

So basically just opening and closing a file is turning these “errors”.

thanks again for the dedication
jb

ferhue · July 26, 2024, 1:51pm

This one looks related to: Valgrind reports leak when constructing TChain · Issue #13130 · root-project/root · GitHub or Memory hoarding triggered by the TPluginManager · Issue #14199 · root-project/root · GitHub

ferhue · July 26, 2024, 2:03pm

I just rerun your test v2 with ROOT (almost master). If I add the following line at the end of the code, then I get less errors.
delete __lfile;

In any case, I confirm that the findScope leak is there (just 27kB, but still there).

4,088 bytes in 36 blocks are definitely lost in loss record 6,147 of 6,815
  in clang::Parser::AnnotateTemplateIdToken(clang::OpaquePtr<clang::TemplateName>, clang::TemplateNameKind, clang::CXXScopeSpec&, clang::SourceLocation, clang::UnqualifiedId&, bool, bool) in /home/user/builds/build-root_src-Desktop-Debug/lib/libCling.so
  1: malloc in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so
  2: clang::Parser::AnnotateTemplateIdToken(clang::OpaquePtr<clang::TemplateName>, clang::TemplateNameKind, clang::CXXScopeSpec&, clang::SourceLocation, clang::UnqualifiedId&, bool, bool) in /home/user/builds/build-root_src-Desktop-Debug/lib/libCling.so
  3: clang::Parser::ParseOptionalCXXScopeSpecifier(clang::CXXScopeSpec&, clang::OpaquePtr<clang::QualType>, bool, bool, bool*, bool, clang::IdentifierInfo**, bool, bool) in /home/user/builds/build-root_src-Desktop-Debug/lib/libCling.so
  4: clang::Parser::TryAnnotateCXXScopeToken(bool) in /home/user/builds/build-root_src-Desktop-Debug/lib/libCling.so
  5: cling::LookupHelper::findScope(llvm::StringRef, cling::LookupHelper::DiagSetting, clang::Type const**, bool) const in /home/user/builds/build-root_src-Desktop-Debug/lib/libCling.so
  6: GetClassSharedLibsForModule(char const*, cling::LookupHelper&, bool) in /opt/root_src/core/metacling/src/TCling.cxx:7007
  7: TCling::GetClassSharedLibs(char const*, bool) in /opt/root_src/core/metacling/src/TCling.cxx:7118
  8: TCling::ShallowAutoLoadImpl(char const*) in /opt/root_src/core/metacling/src/TCling.cxx:6170
  9: TCling::DeepAutoLoadImpl(char const*, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, bool) in /opt/root_src/core/metacling/src/TCling.cxx:6223
  10: TCling::AutoLoad(char const*, bool) in /opt/root_src/core/metacling/src/TCling.cxx:6335
  11: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) in /opt/root_src/core/meta/src/TClass.cxx:3104
  12: TClass::GetClass(char const*, bool, bool) in /opt/root_src/core/meta/src/TClass.cxx:2970

pcanal · July 26, 2024, 2:18pm

Yes, this can be reproduced and need to be checked (for example by @debjit).

However, it was not clear whether this was your original problem or not. For the time being, I recommend that you add your own suppression file that hides this.

I opened the issue Potential memory leak in clang triggered by `findScope`. · Issue #16121 · root-project/root · GitHub to keep track of it.

jbb · July 26, 2024, 2:32pm

Thanks

I’ll suppress them for the time being to see if I’m getting back on my feet to detect possible leaks of my own .

Last question: the fact that you focus on the malloc part, means that the conditionnal jump below can safely be ignored ?

cheers
jb

==75628== Conditional jump or move depends on uninitialised value(s)
==75628== at 0xD52162A: llvm::AttrBuilder::addStackAlignmentAttr(llvm::MaybeAlign) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x98FF5A8: clang::CodeGen::CodeGenModule::ConstructAttributeList(llvm::StringRef, clang::CodeGen::CGFunctionInfo const&, clang::CodeGen::CGCalleeInfo, llvm::AttributeList&, unsigned int&, bool, bool) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x969A787: clang::CodeGen::CodeGenModule::SetLLVMFunctionAttributes(clang::GlobalDecl, clang::CodeGen::CGFunctionInfo const&, llvm::Function*, bool) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x96C7EFD: clang::CodeGen::CodeGenModule::SetFunctionAttributes(clang::GlobalDecl, llvm::Function*, bool, bool) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x96CA59B: clang::CodeGen::CodeGenModule::GetOrCreateLLVMFunction(llvm::StringRef, llvm::Type*, clang::GlobalDecl, bool, bool, bool, llvm::AttributeList, clang::CodeGen::ForDefinition_t) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x96CBC9C: clang::CodeGen::CodeGenModule::GetAddrOfFunction(clang::GlobalDecl, llvm::Type*, bool, bool, clang::CodeGen::ForDefinition_t) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x96CC257: clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition(clang::GlobalDecl, llvm::GlobalValue*) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x96C89EC: clang::CodeGen::CodeGenModule::EmitGlobalDefinition(clang::GlobalDecl, llvm::GlobalValue*) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x96C92BA: clang::CodeGen::CodeGenModule::EmitGlobal(clang::GlobalDecl) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x96D1399: clang::CodeGen::CodeGenModule::EmitTopLevelDecl(clang::Decl*) [clone .part.0] (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x96D273F: clang::CodeGen::CodeGenModule::EmitDeclContext(clang::DeclContext const*) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x958FFD0: clang::CodeGeneratorImpl::HandleTopLevelDecl(clang::DeclGroupRef) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628==
==75628== Conditional jump or move depends on uninitialised value(s)
==75628== at 0xD52162A: llvm::AttrBuilder::addStackAlignmentAttr(llvm::MaybeAlign) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x98FF5A8: clang::CodeGen::CodeGenModule::ConstructAttributeList(llvm::StringRef, clang::CodeGen::CGFunctionInfo const&, clang::CodeGen::CGCalleeInfo, llvm::AttributeList&, unsigned int&, bool, bool) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x99029E8: clang::CodeGen::CodeGenFunction::EmitCall(clang::CodeGen::CGFunctionInfo const&, clang::CodeGen::CGCallee const&, clang::CodeGen::ReturnValueSlot, clang::CodeGen::CallArgList const&, llvm::CallBase**, bool, clang::SourceLocation) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x996CD27: clang::CodeGen::CodeGenFunction::EmitCall(clang::QualType, clang::CodeGen::CGCallee const&, clang::CallExpr const*, clang::CodeGen::ReturnValueSlot, llvm::Value*) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x997C053: clang::CodeGen::CodeGenFunction::EmitCallExpr(clang::CallExpr const*, clang::CodeGen::ReturnValueSlot) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x99BDDEC: (anonymous namespace)::ScalarExprEmitter::VisitCallExpr(clang::CallExpr const*) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x99BD75B: clang::CodeGen::CodeGenFunction::EmitScalarExpr(clang::Expr const*, bool) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x960F2CB: clang::CodeGen::CodeGenFunction::EmitReturnStmt(clang::ReturnStmt const&) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x961225B: clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope(clang::CompoundStmt const&, bool, clang::CodeGen::AggValueSlot) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x9674B79: clang::CodeGen::CodeGenFunction::EmitFunctionBody(clang::Stmt const*) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x9683D06: clang::CodeGen::CodeGenFunction::GenerateCode(clang::GlobalDecl, llvm::Function*, clang::CodeGen::CGFunctionInfo const&) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x96CC352: clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition(clang::GlobalDecl, llvm::GlobalValue*) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628==
==75628== Conditional jump or move depends on uninitialised value(s)
==75628== at 0xCE735BC: llvm::getAllocAlignment(llvm::CallBase const*, llvm::TargetLibraryInfo const*) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0xCA0A1EF: llvm::InstCombinerImpl::annotateAnyAllocSite(llvm::CallBase&, llvm::TargetLibraryInfo const*) [clone .part.0] (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0xCA1277B: llvm::InstCombinerImpl::visitCallBase(llvm::CallBase&) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0xCA15A53: llvm::InstCombinerImpl::visitCallInst(llvm::CallInst&) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0xC9BEF7E: llvm::InstCombinerImpl::run() (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0xC9C2298: combineInstructionsOverFunction(llvm::Function&, llvm::InstructionWorklist&, llvm::AAResults*, llvm::AssumptionCache&, llvm::TargetLibraryInfo&, llvm::TargetTransformInfo&, llvm::DominatorTree&, llvm::OptimizationRemarkEmitter&, llvm::BlockFrequencyInfo*, llvm::ProfileSummaryInfo*, unsigned int, llvm::LoopInfo*) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0xC9C31AC: llvm::InstCombinePass::run(llvm::Function&, llvm::AnalysisManagerllvm::Function&) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x978A08D: llvm::detail::PassModel<llvm::Function, llvm::InstCombinePass, llvm::PreservedAnalyses, llvm::AnalysisManagerllvm::Function>::run(llvm::Function&, llvm::AnalysisManagerllvm::Function&) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x979857D: llvm::detail::PassModel<llvm::Function, llvm::PassManager<llvm::Function, llvm::AnalysisManagerllvm::Function>, llvm::PreservedAnalyses, llvm::AnalysisManagerllvm::Function>::run(llvm::Function&, llvm::AnalysisManagerllvm::Function&) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0xD6893E5: llvm::ModuleToFunctionPassAdaptor::run(llvm::Module&, llvm::AnalysisManagerllvm::Module&) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x9789E2D: llvm::detail::PassModel<llvm::Module, llvm::ModuleToFunctionPassAdaptor, llvm::PreservedAnalyses, llvm::AnalysisManagerllvm::Module>::run(llvm::Module&, llvm::AnalysisManagerllvm::Module&) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628== by 0x9559F36: cling::BackendPasses::runOnModule(llvm::Module&, int) (in /export/home/jb242989/root_v6.32.02_deb/lib/libCling.so.6.32.02)
==75628==

pcanal · July 26, 2024, 3:03pm

It only meant that I was focusing on leaks

I opened onditional jump or move depends on uninitialised value triggered in LLVM's CodeGen · Issue #16123 · root-project/root · GitHub for the conditionnal jump.

system · August 9, 2024, 3:04pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.