ROOT with Geant4: LLVM Problems

Hello,

I am attempting to integrate ROOT IO into a Geant4 program, but I’m running into difficulties with how LLVM is being linked which causes some odd runtime errors.

I get seg faults but not all the time. It seems to occur mainly when the Geant4 visualization system is active during a run in an interactive session, but I can run in batch mode okay.

First, I get the following warning message:

Error in UnknownClass::InitInterpreter(): LLVM SYMBOLS ARE EXPOSED TO CLING! This will cause problems; please hide them or dlopen() them after the call to TROOT::InitInterpreter()!

After this, a Seg Fault occurs with a rather complex traceback. The relevant portion of one example is probably the following:

#27 0x00007f96d6cba023 in llvm::MemoryBuffer::getMemBufferRef() const () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#28 0x00007f96d56836e1 in std::_Function_handler<llvm::object::OwningBinaryllvm::object::ObjectFile (llvm::Module&), llvm::SimpleCompiler>::_M_invoke(std::_Any_data const&, llvm::Module&) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#29 0x00007f96d5682cc7 in std::_List_iteratorllvm::ObjectLinkingLayerBase::LinkedObjectSet llvm::IRCompileLayer<llvm::ObjectLinkingLayercling::IncrementalJIT::NotifyObjectLoadedT >::addModuleSet<std::vector<llvm::Module*, std::allocatorllvm::Module* > >(std::vector<llvm::Module*, std::allocatorllvm::Module* >, std::unique_ptr<llvm::RTDyldMemoryManager, std::default_deletellvm::RTDyldMemoryManager >) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#30 0x00007f96d5683553 in llvm::LazyEmittingLayer<llvm::IRCompileLayer<llvm::ObjectLinkingLayercling::IncrementalJIT::NotifyObjectLoadedT > >::EmissionDeferredSetImpl<std::vector<llvm::Module*, std::allocatorllvm::Module* > >::Emit(llvm::IRCompileLayer<llvm::ObjectLinkingLayercling::IncrementalJIT::NotifyObjectLoadedT >&) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#31 0x00007f96d56813ec in cling::IncrementalJIT::getSymbolAddressWithoutMangling(llvm::StringRef, bool) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#32 0x00007f96d567f807 in cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction const&) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#33 0x00007f96d5629786 in cling::Interpreter::executeTransaction(cling::Transaction&) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#34 0x00007f96d5686791 in cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraitscling::Transaction* >) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#35 0x00007f96d562bfc8 in cling::Interpreter::Interpreter(int, char const* const*, char const*, bool) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#36 0x00007f96d54d7d91 in TCling::TCling (this=0x33cf9b0, name=0x7f96d6d06a87 “C++”, title=0x7f96d6d06a71 “cling C++ Interpreter”) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TCling.cxx:1098
#37 0x00007f96d54d4f03 in CreateInterpreter (interpLibHandle=0x1baa2d0) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TCling.cxx:578
#38 0x00007f96eba0022c in TROOT::InitInterpreter (this=0x7f96ebf06640 ROOT::Internal::GetROOT1()::alloc) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TROOT.cxx:1767
#39 0x00007f96eb9f9e03 in ROOT::Internal::GetROOT2 () at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TROOT.cxx:362
#40 0x00007f96eb9f9e2e in ROOT::GetROOT () at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TROOT.cxx:376
#41 0x00007f96ebb892b5 in TClass::GetClass (name=0x7f96f215f5c4 “SimParticle”, load=true, silent=false) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TClass.cxx:2891
#42 0x00007f96ebb44077 in TClonesArray::SetClass (this=0x33e08c0, classname=0x7f96f215f5c4 “SimParticle”, s=100) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/cont/src/TClonesArray.cxx:721
#43 0x00007f96ebb42461 in TClonesArray::TClonesArray (this=0x33e08c0, classname=0x7f96f215f5c4 “SimParticle”, s=100) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/cont/src/TClonesArray.cxx:167

Strangely enough, it seems to crash in different places if I rerun it. I assume this error is related to the warning about the LLVM/Cling symbols (though it may in fact not be!).

I read on a few articles I found on Google that linking Cling in explicitly to a binary can cause these types of issues, but I am not doing this. The libCling.so should be loaded dynamically by TROOT (I think?).

I configure ROOT as follows in my CMake file:

find_package(ROOT REQUIRED COMPONENTS Core RIO)

Then I link in the standard way:

target_link_libraries(myprog ${ROOT_LIBRARIES})

Since this is primarily a Geant4 app, I’m using its compiler switches:

include(${Geant4_USE_FILE})

Perhaps I should try ROOT’s instead? (Not sure it would make any difference.)

I am using these packages:

Geant4 10.2.p02
ROOT 6.06.06
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
cmake version 3.6.1
CentOS Linux release 7.2.1511 (Core)

I am running on a Linux VM but I doubt this has anything to do with it.

Any suggestions for what I can try to do in order to debug this?

Is it possible that using a newer, external LLVM instead of the one packaged with ROOT could help solve this problem? I read that the newer LLVM uses versioned symbols which might help.

Is there an example of a known working CMakeLists.txt which configures both Geant4 and ROOT correctly so they both work without crashing? Specifically, I need to be able to have an active Geant4 visualization like OpenGL in an interactive session for showing an event display without these crashes.

Thanks.

–Jeremy

Hi Jeremy,

ROOT internally is taking care of properly handling libCling which, as you say, should not be linked by the user.
Geant4 already uses ROOT in the persistency examples: is this maybe a place to start debugging?

Danilo

Hi,

…and a few more things: is anything else using llvm? (A likely candidate are linux graphics drivers, sadly; there was a bug in a version from two years ago which published the symbols.) What you can do is put a breakpoint at TROOT::InitInterpreter(), i.e. before libCling gets loaded, and “ask” gdb where the symbol called LLVMEnablePrettyStackTrace is coming from.

Also, providing us with the stack frames < 27 would indeed help. One of the questions is whether anything but libCling shows up in there.

Cheers, Axel.

Thanks for the reply, Axel.

Here is the full traceback in all its glory:

#0 0x00007f96e7cd4b59 in waitpid () from /lib64/libc.so.6
#1 0x00007f96e7c59092 in do_system () from /lib64/libc.so.6
#2 0x00007f96e7c59441 in system () from /lib64/libc.so.6
#3 0x00007f96ebbeaae2 in TUnixSystem::Exec (this=0x18ae900, shellcmd=0x3e2fa30 “/nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/build/etc/gdb-backtrace.sh 11043 1>&2”) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/unix/src/TUnixSystem.cxx:2096
#4 0x00007f96ebbeb35d in TUnixSystem::StackTrace (this=0x18ae900) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/unix/src/TUnixSystem.cxx:2324
#5 0x00007f96ebac4aa6 in DefaultErrorHandler (level=6000, abort_bool=true, location=0x7f96f498e253 “UnknownClass::GetListOfTypes”, msg=0x7ffe224bcf90 “fInterpreter not initialized”) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TError.cxx:192
#6 0x00007f96ebac4dcb in ErrorHandler(Int_t, const char , const char , typedef __va_list_tag __va_list_tag ) (level=6000, location=0x7f96f498e253 “UnknownClass::GetListOfTypes”, fmt=0x7f96ebbf6227 “fInterpreter not initialized”, ap=0x7ffe224bd168) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TError.cxx:251
#7 0x00007f96ebadab83 in TObject::DoError (this=0x7f96ebf06640 ROOT::Internal::GetROOT1()::alloc, level=6000, location=0x7f96ebbf6340 “GetListOfTypes”, fmt=0x7f96ebbf6227 “fInterpreter not initialized”, va=0x7ffe224bd168) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TObject.cxx:885
#8 0x00007f96ebadaf95 in TObject::Fatal (this=0x7f96ebf06640 ROOT::Internal::GetROOT1()::alloc, location=0x7f96ebbf6340 “GetListOfTypes”, fmt=0x7f96ebbf6227 “fInterpreter not initialized”) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TObject.cxx:950
#9 0x00007f96eb9ff5ed in TROOT::GetListOfTypes (this=0x7f96ebf06640 ROOT::Internal::GetROOT1()::alloc) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TROOT.cxx:1534
#10 0x00007f96d54e017e in TCling::CheckClassInfo (this=0x33cf9b0, name=0x3e32f39 “TTree”, autoload=true, isClassOrNamespaceOnly=false) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TCling.cxx:3228
#11 0x00007f96ebb83f87 in TClass::Init (this=0x3e32f20, name=0x7f96e9ce25c8 “TTree”, cversion=19, typeinfo=0x7f96e9f277c0 , isa=0x1928ba0, dfil=0x7f96e9ce2470 “TTree.h”, ifil=0x7f96e9ceaa68 “/nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/tree/tree/src/TTree.cxx”, dl=94, il=396, givenInfo=0x0, silent=false) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TClass.cxx:1426
#12 0x00007f96ebb83628 in TClass::TClass (this=0x3e32f20, name=0x7f96e9ce25c8 “TTree”, cversion=19, info=…, isa=0x1928ba0, dfil=0x7f96e9ce2470 “TTree.h”, ifil=0x7f96e9ceaa68 “/nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/tree/tree/src/TTree.cxx”, dl=94, il=396, silent=false) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TClass.cxx:1276
#13 0x00007f96ebb91948 in ROOT::CreateClass (cname=0x7f96e9ce25c8 “TTree”, id=19, info=…, isa=0x1928ba0, dfil=0x7f96e9ce2470 “TTree.h”, ifil=0x7f96e9ceaa68 “/nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/tree/tree/src/TTree.cxx”, dl=94, il=396) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TClass.cxx:5490
#14 0x00007f96eba76601 in ROOT::Internal::TDefaultInitBehavior::CreateClass (this=0x7f96ebf062b0 <ROOT::Internal::DefineBehavior(void
, void
)::theDefault>, cname=0x7f96e9ce25c8 “TTree”, id=19, info=…, isa=0x1928ba0, dfil=0x7f96e9ce2470 “TTree.h”, ifil=0x7f96e9ceaa68 “/nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/tree/tree/src/TTree.cxx”, dl=94, il=396) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/build/include/Rtypes.h:218
#15 0x00007f96ebba9c36 in ROOT::TGenericClassInfo::GetClass (this=0x7f96e9f31680 <ROOT::GenerateInitInstanceLocal(TTree const
)::instance>) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TGenericClassInfo.cxx:214
#16 0x00007f96e9c3ae56 in TTree::Class () at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/build/tree/tree/G__Tree.cxx:3139
#17 0x00007f96e9ccca01 in TTree::IsA (this=0x33d89d0) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/build/include/TTree.h:561
#18 0x00007f96ebabf30f in TDirectory::Close (this=0x7f96ebf06640 ROOT::Internal::GetROOT1()::alloc) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TDirectory.cxx:529
#19 0x00007f96eb9fc8a8 in TROOT::CloseFiles (this=0x7f96ebf06640 ROOT::Internal::GetROOT1()::alloc) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TROOT.cxx:876
#20 0x00007f96eb9fcec8 in TROOT::EndOfProcessCleanups (this=0x7f96ebf06640 ROOT::Internal::GetROOT1()::alloc) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TROOT.cxx:950
#21 0x00007f96ebbeab70 in TUnixSystem::Exit (this=0x18ae900, code=139, mode=true) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/unix/src/TUnixSystem.cxx:2131
#22 0x00007f96ebbeebe0 in TUnixSystem::DispatchSignals (this=0x18ae900, sig=kSigSegmentationViolation) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/unix/src/TUnixSystem.cxx:3569
#23 0x00007f96ebbe6ecd in SigHandler (sig=kSigSegmentationViolation) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/unix/src/TUnixSystem.cxx:395
#24 0x00007f96ebbeeab6 in sighandler (sig=11) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/unix/src/TUnixSystem.cxx:3539
#25
#26 0x00007f96e7d7a3c1 in __strlen_sse2_pminub () from /lib64/libc.so.6
#27 0x00007f96d6cba023 in llvm::MemoryBuffer::getMemBufferRef() const () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#28 0x00007f96d56836e1 in std::_Function_handler<llvm::object::OwningBinaryllvm::object::ObjectFile (llvm::Module&), llvm::SimpleCompiler>::_M_invoke(std::_Any_data const&, llvm::Module&) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#29 0x00007f96d5682cc7 in std::_List_iteratorllvm::ObjectLinkingLayerBase::LinkedObjectSet llvm::IRCompileLayer<llvm::ObjectLinkingLayercling::IncrementalJIT::NotifyObjectLoadedT >::addModuleSet<std::vector<llvm::Module*, std::allocatorllvm::Module* > >(std::vector<llvm::Module*, std::allocatorllvm::Module* >, std::unique_ptr<llvm::RTDyldMemoryManager, std::default_deletellvm::RTDyldMemoryManager >) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#30 0x00007f96d5683553 in llvm::LazyEmittingLayer<llvm::IRCompileLayer<llvm::ObjectLinkingLayercling::IncrementalJIT::NotifyObjectLoadedT > >::EmissionDeferredSetImpl<std::vector<llvm::Module*, std::allocatorllvm::Module* > >::Emit(llvm::IRCompileLayer<llvm::ObjectLinkingLayercling::IncrementalJIT::NotifyObjectLoadedT >&) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#31 0x00007f96d56813ec in cling::IncrementalJIT::getSymbolAddressWithoutMangling(llvm::StringRef, bool) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#32 0x00007f96d567f807 in cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction const&) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#33 0x00007f96d5629786 in cling::Interpreter::executeTransaction(cling::Transaction&) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#34 0x00007f96d5686791 in cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraitscling::Transaction* >) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#35 0x00007f96d562bfc8 in cling::Interpreter::Interpreter(int, char const* const*, char const*, bool) () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so
#36 0x00007f96d54d7d91 in TCling::TCling (this=0x33cf9b0, name=0x7f96d6d06a87 “C++”, title=0x7f96d6d06a71 “cling C++ Interpreter”) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TCling.cxx:1098
#37 0x00007f96d54d4f03 in CreateInterpreter (interpLibHandle=0x1baa2d0) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TCling.cxx:578
#38 0x00007f96eba0022c in TROOT::InitInterpreter (this=0x7f96ebf06640 ROOT::Internal::GetROOT1()::alloc) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TROOT.cxx:1767
#39 0x00007f96eb9f9e03 in ROOT::Internal::GetROOT2 () at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TROOT.cxx:362
#40 0x00007f96eb9f9e2e in ROOT::GetROOT () at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/base/src/TROOT.cxx:376
#41 0x00007f96ebb892b5 in TClass::GetClass (name=0x7f96f215f5c4 “SimParticle”, load=true, silent=false) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/meta/src/TClass.cxx:2891
#42 0x00007f96ebb44077 in TClonesArray::SetClass (this=0x33e08c0, classname=0x7f96f215f5c4 “SimParticle”, s=100) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/cont/src/TClonesArray.cxx:721
#43 0x00007f96ebb42461 in TClonesArray::TClonesArray (this=0x33e08c0, classname=0x7f96f215f5c4 “SimParticle”, s=100) at /nfs/slac/g/ldmx/users/jeremym/dev/root-6.06.06/core/cont/src/TClonesArray.cxx:167
#44 0x00007f96f21589a1 in Event::Event (this=0x33d69c0) at /nfs/slac/g/ldmx/users/jeremym/dev/ldmx-sw/Event/src/Event.cc:15
#45 0x00007f96f2159afa in RootEventWriter::open (this=0x33c4540) at /nfs/slac/g/ldmx/users/jeremym/dev/ldmx-sw/Event/src/RootEventWriter.cc:51
#46 0x0000000000443c5b in UserRunAction::BeginOfRunAction (this=0x1dbe420) at /nfs/slac/g/ldmx/users/jeremym/dev/ldmx-sw/SimApplication/src/UserRunAction.cc:13
#47 0x00007f96efef6687 in G4RunManager::RunInitialization (this=0x199ec60) at /nfs/slac/g/ldmx/users/jeremym/dev/geant4.10.02.p02/source/run/src/G4RunManager.cc:347
#48 0x00007f96efef1a7d in G4RunManager::BeamOn (this=0x199ec60, n_event=1, macroFile=0x0, n_select=-1) at /nfs/slac/g/ldmx/users/jeremym/dev/geant4.10.02.p02/source/run/src/G4RunManager.cc:272
#49 0x00007f96eff0a253 in G4RunMessenger::SetNewValue (this=0x1b98e50, command=command
entry=0x1b992a0, newValue=…) at /nfs/slac/g/ldmx/users/jeremym/dev/geant4.10.02.p02/source/run/src/G4RunMessenger.cc:376
#50 0x00007f96ec4879f5 in G4UIcommand::DoIt (this=this
entry=0x1b992a0, parameterList=…) at /nfs/slac/g/ldmx/users/jeremym/dev/geant4.10.02.p02/source/intercoms/src/G4UIcommand.cc:230
#51 0x00007f96ec49f7c8 in G4UImanager::ApplyCommand (this=0x1970d60, aCmd=) at /nfs/slac/g/ldmx/users/jeremym/dev/geant4.10.02.p02/source/intercoms/src/G4UImanager.cc:523
#52 0x00007f96ec4a0268 in G4UImanager::ApplyCommand (this=, aCmd=…) at /nfs/slac/g/ldmx/users/jeremym/dev/geant4.10.02.p02/source/intercoms/src/G4UImanager.cc:441
#53 0x00007f96f362b72a in G4UIterminal::ExecuteCommand (this=0x199e800, aCommand=…) at /nfs/slac/g/ldmx/users/jeremym/dev/geant4.10.02.p02/source/interfaces/basic/src/G4UIterminal.cc:166
#54 0x00007f96f362cd04 in G4UIterminal::SessionStart (this=0x199e800) at /nfs/slac/g/ldmx/users/jeremym/dev/geant4.10.02.p02/source/interfaces/basic/src/G4UIterminal.cc:141
#55 0x0000000000438bf0 in SimApplication::run (this=0x19980e0, argc=1, argv=0x7ffe224bf818) at /nfs/slac/g/ldmx/users/jeremym/dev/ldmx-sw/SimApplication/src/SimApplication.cc:84
#56 0x000000000042d7d0 in main (argc=1, argv=0x7ffe224bf818) at /nfs/slac/g/ldmx/users/jeremym/dev/ldmx-sw/SimApplication/src/ldmx_sim.cc:7

I reran my simulation today doing the the exact same thing and it worked…once! Then when I run the simulation again, it crashes but with a different traceback. This is some kind of strange non-deterministic bug. :open_mouth:

A couple questions:

If we did find that there are symbols being exported by another library that were causing problems here, is there anything that could be done about it? (For instance, I’m not sure that if there’s a Linux graphics library bug that I would be able to do anything about it.)

Would it be worth my while to update my ROOT from 6.06.06 to 6.06.08? I know this is only a few patch releases, but I can try it to see if it makes any difference.

Also, might it be useful for me to build standalone LLVM of which the current version is 3.9? I had read some posts online that the symbol exporting was “versioned” in more recent releases of LLVM, which could supposedly help. I was having some memory issues on my VM with linking some of the LLVM libraries but I can work to fix that if you think it might be helpful to use the most recent LLVM release.

Finally, can you tell me what gdb commands to execute to investigate the library symbols? I could probably set the breakpoint on the interpreter but I’m not sure which commands to execute for finding out which library is exporting “LLVMEnablePrettyStackTrace”.

Thanks again!

Hi Jeremy,

your questions might be very relevant but before diving into those I wonder if it’s not better to figure out what is going on with your setup and better understand the non deterministic behaviour you describe. You are not running in MT mode, are you?

Cheers,
D

I have not enabled MT mode in Geant4.

In fact, my setup was working fine for awhile, and then something caused the LVVM issues which I’m not able to identify. Everything seems to originate with opening a new TFile from what I can tell.

I am thinking the non-deterministic behavior has to do with the memory usage and mapping at runtime rather than anything wrong with my application.

I always see this line in the traceback:

#27 0x00007f96d6cba023 in llvm::MemoryBuffer::getMemBufferRef() const () from /u/ey/jeremym/ldmx-dev/root-6.06.06/build/lib/libCling.so

It seems to suggest that the “polluted” symbol space is interfering with LLVM’s memory management, but I don’t know enough about this library to say anything more specific.

Hi,

perhaps you can start removing components around ROOT to see when the program starts failing?

D

Thanks for the suggestion.

My application is not using very much of ROOT right now, just RIO and Core, and whatever lib dependencies those pull in.

That ends up being the following libs:

libCore.so
libRIO.so
libNet.so
libHist.so
libGraf.so
libGraf3d.so
libGpad.so
libTree.so
libRint.so
libPostscript.so
libMatrix.so
libPhysics.so
libMathCore.so
libThread.so
libMultiProc.so

So stripping things down any further means probably disabling ROOT IO entirely in the application (which wouldn’t help debug this).

Hi,

I am confused. The list includes much more than RIO and Core (which includes RIO,Thread and in a “shielded” way libCling): are you linking those libs even if you do not use them? What are the non-ROOT libraries linked?
What is the minimal program dependent on G4 and ROOT which reproduces the issue?

D

Hi,

This should allow us to figure out whether another llvm library is polluting your environment:

gdb --args your-binary...
r
(your program will now crash, but now gdb knows all the symbols. So now you can do:)
b TROOT::InitInterpreter()
r
(gdb will now hold at TROOT::InitInterpreter())
info sharedlibrary
(I need the output of that)
p LLVMEnablePrettyStackTrace

That should tell us…

Cheers, Axel.

Hi, Axel.

Sorry I didn’t reply sooner…

The library information is the following:

From To Syms Read Shared Object Library
0x00007ffff7ddbae0 0x00007ffff7df627a Yes () /lib64/ld-linux-x86-64.so.2
0x00007ffff7bcfff0 0x00007ffff7bd71e8 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4Tree.so
0x00007ffff79958a0 0x00007ffff79b9700 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4GMocren.so
0x00007ffff7723a20 0x00007ffff776a9ac Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4visHepRep.so
0x00007ffff74e3040 0x00007ffff74f0c00 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4RayTracer.so
0x00007ffff72c2e70 0x00007ffff72cd9a8 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4VRML.so
0x00007ffff7090240 0x00007ffff70ad978 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4OpenGL.so
0x00007ffff6e68f80 0x00007ffff6e749b0 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4gl2ps.so
0x00007ffff6c32540 0x00007ffff6c5d5dc Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4interfaces.so
0x00007ffff69282e0 0x00007ffff69f474c Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4persistency.so
0x00007ffff643e760 0x00007ffff6637608 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4analysis.so
0x00007ffff6108330 0x00007ffff6129bf4 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4error_propagation.so
0x00007ffff5ef3640 0x00007ffff5ef6748 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4readout.so
0x00007ffff5bfa310 0x00007ffff5cb1178 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4physicslists.so
0x00007ffff598dd10 0x00007ffff5996200 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4parmodels.so
0x00007ffff57754d0 0x00007ffff577c764 Yes /u/ey/jeremym/ldmx-dev/ldmx-sw-install/lib/libEvent.so
0x00007ffff555f270 0x00007ffff5562218 Yes (
) /u/ey/jeremym/ldmx-dev/ldmx-sw-install/lib/libDetDescr.so
0x00007ffff534a780 0x00007ffff5352894 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4FR.so
0x00007ffff509a8c0 0x00007ffff5117680 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4vis_management.so
0x00007ffff4de7240 0x00007ffff4e42900 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4modeling.so
0x00007ffff4b29830 0x00007ffff4b852fc Yes () /lib64/libGLU.so.1
0x00007ffff48a5fe0 0x00007ffff49021f2 Yes (
) /lib64/libGL.so.1
0x00007ffff4687c20 0x00007ffff468bc34 Yes () /lib64/libSM.so.6
0x00007ffff446eea0 0x00007ffff447c10c Yes (
) /lib64/libICE.so.6
0x00007ffff4149fd0 0x00007ffff41d0cdc Yes () /lib64/libX11.so.6
0x00007ffff3f1d770 0x00007ffff3f27470 Yes (
) /lib64/libXext.so.6
0x00007ffff3d05750 0x00007ffff3d130dc Yes () /lib64/libXmu.so.6
0x00007ffff388f860 0x00007ffff39ebfc4 Yes /nfs/slac/g/ldmx/software/xerces/xerces-c-3.1.4/lib/libxerces-c-3.1.so
0x00007ffff34f8e40 0x00007ffff353d818 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4run.so
0x00007ffff3282f20 0x00007ffff32c7818 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4event.so
0x00007ffff302d8b0 0x00007ffff305d824 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4tracking.so
0x00007ffff15e13a0 0x00007ffff1fd2ea8 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4processes.so
0x00007ffff1060340 0x00007ffff106c808 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4zlib.so
0x00007ffff0e37e00 0x00007ffff0e50cdc Yes (
) /lib64/libexpat.so.1
0x00007ffff0bb1990 0x00007ffff0c14c40 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4digits_hits.so
0x00007ffff0973a20 0x00007ffff0982a9c Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4track.so
0x00007ffff067c3b0 0x00007ffff070ce3c Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4particles.so
0x00007ffff02469e0 0x00007ffff03f1d2c Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4geometry.so
0x00007fffeff2aa00 0x00007fffeff903b0 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4materials.so
0x00007fffefceebc0 0x00007fffefd0e088 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4graphics_reps.so
0x00007fffefa90dd0 0x00007fffefacbb4c Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4intercoms.so
0x00007fffef843980 0x00007fffef872de8 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4global.so
0x00007fffef57ec30 0x00007fffef5c7b38 Yes /nfs/slac/g/ldmx/software/geant4/geant4.10.02.p02-install/lib64/libG4clhep.so
0x00007fffef0d3960 0x00007fffef252adc Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libCore.so
0x00007fffeebad5b0 0x00007fffeed287a0 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libRIO.so
0x00007fffee8a9130 0x00007fffee8e8358 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libNet.so
0x00007fffee4069d0 0x00007fffee5a559c Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libHist.so
0x00007fffee01ece0 0x00007fffee0aa0d0 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libGraf.so
0x00007fffedd66950 0x00007fffedd97e48 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libGraf3d.so
0x00007fffedacc520 0x00007fffedb0a6f8 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libGpad.so
0x00007fffed7cdc10 0x00007fffed84b6cc Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libTree.so
0x00007fffed54aa50 0x00007fffed559f8c Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libRint.so
0x00007fffed2feb10 0x00007fffed324120 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libPostscript.so
0x00007fffecd6fe80 0x00007fffecd8f458 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libPhysics.so
0x00007fffeca33780 0x00007fffecad09b8 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libMathCore.so
0x00007fffec7c1a20 0x00007fffec7cb238 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libThread.so
0x00007fffec5a56c0 0x00007fffec5a91f0 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libMultiProc.so
0x00007fffec2f3510 0x00007fffec35a59a Yes () /lib64/libstdc++.so.6
0x00007fffebf9b4b0 0x00007fffec0059e8 Yes (
) /lib64/libm.so.6
0x00007fffebd82af0 0x00007fffebd92298 Yes () /lib64/libgcc_s.so.1
0x00007fffeb9dd3e0 0x00007fffebb21670 Yes (
) /lib64/libc.so.6
0x00007fffeb7a78a0 0x00007fffeb7b2514 Yes () /lib64/libpthread.so.0
0x00007fffeb57e8c0 0x00007fffeb58957e Yes (
) /lib64/libglapi.so.0
0x00007fffeb3554a0 0x00007fffeb36826c Yes () /lib64/libselinux.so.1
0x00007fffeb14cc40 0x00007fffeb14d56c Yes (
) /lib64/libXdamage.so.1
0x00007fffeaf475f0 0x00007fffeaf49948 Yes () /lib64/libXfixes.so.3
0x00007fffead44600 0x00007fffead4470c Yes (
) /lib64/libX11-xcb.so.1
0x00007fffeab34020 0x00007fffeab3bdd8 Yes () /lib64/libxcb-glx.so.0
0x00007fffea926920 0x00007fffea927a78 Yes (
) /lib64/libxcb-dri2.so.0
0x00007fffea722d50 0x00007fffea7234e4 Yes () /lib64/libxcb-dri3.so.0
0x00007fffea51fc50 0x00007fffea520428 Yes (
) /lib64/libxcb-present.so.0
0x00007fffea316bd0 0x00007fffea31aa34 Yes () /lib64/libxcb-randr.so.0
0x00007fffea10b830 0x00007fffea10dc48 Yes (
) /lib64/libxcb-xfixes.so.0
0x00007fffe9f02610 0x00007fffe9f05a4c Yes () /lib64/libxcb-render.so.0
0x00007fffe9cfc020 0x00007fffe9cfcbc8 Yes (
) /lib64/libxcb-shape.so.0
0x00007fffe9af6060 0x00007fffe9af7ffc Yes () /lib64/libxcb-sync.so.1
0x00007fffe98dbcc0 0x00007fffe98ea2a8 Yes (
) /lib64/libxcb.so.1
0x00007fffe96cf990 0x00007fffe96cfc6c Yes () /lib64/libxshmfence.so.1
0x00007fffe94ca010 0x00007fffe94cc958 Yes (
) /lib64/libXxf86vm.so.1
0x00007fffe92bf400 0x00007fffe92c5484 Yes () /lib64/libdrm.so.2
0x00007fffe90b8ed0 0x00007fffe90b99d0 Yes (
) /lib64/libdl.so.2
0x00007fffe8eb4510 0x00007fffe8eb5b64 Yes () /lib64/libuuid.so.1
0x00007fffe8c5f1b0 0x00007fffe8c9b15c Yes (
) /lib64/libXt.so.6
0x00007fffe8a37110 0x00007fffe8a43d64 Yes () /lib64/libnsl.so.1
0x00007fffe87d35f0 0x00007fffe88188b0 Yes (
) /lib64/libpcre.so.1
0x00007fffe85aff30 0x00007fffe85c5e90 Yes () /lib64/liblzma.so.5
0x00007fffe8399170 0x00007fffe83a56f0 Yes (
) /lib64/libz.so.1
0x00007fffe8142460 0x00007fffe817d150 Yes () /lib64/libssl.so.10
0x00007fffe7daa200 0x00007fffe7e9c9a8 Yes (
) /lib64/libcrypto.so.10
0x00007fffe7b0bec0 0x00007fffe7b10b1c Yes () /lib64/libcrypt.so.1
0x00007fffe7871ec0 0x00007fffe78de104 Yes (
) /lib64/libfreetype.so.6
0x00007fffe75a19f0 0x00007fffe76229e8 Yes /nfs/slac/g/ldmx/software/root/root-6.06.08-build/lib/libTreePlayer.so
0x00007fffe734dec0 0x00007fffe734ebdc Yes () /lib64/libXau.so.6
0x00007fffe710d010 0x00007fffe713ef64 Yes (
) /lib64/libgssapi_krb5.so.2
0x00007fffe6e40b50 0x00007fffe6ea6a2c Yes () /lib64/libkrb5.so.3
0x00007fffe6c19570 0x00007fffe6c1a144 Yes (
) /lib64/libcom_err.so.2
0x00007fffe69ea770 0x00007fffe6a07648 Yes () /lib64/libk5crypto.so.3
0x00007fffe67e3ba0 0x00007fffe67e42f9 Yes (
) /lib64/libfreebl3.so
0x00007fffe65d76e0 0x00007fffe65de75c Yes () /lib64/libkrb5support.so.0
0x00007fffe63d15b0 0x00007fffe63d21cc Yes (
) /lib64/libkeyutils.so.1
0x00007fffe61b9a40 0x00007fffe61c8854 Yes () /lib64/libresolv.so.2
0x00007fffe5fa5240 0x00007fffe5fac454 Yes (
) /lib64/libnss_files.so.2
0x00007fffe5d99150 0x00007fffe5d9f57c Yes () /lib64/libnss_nis.so.2
0x00007fffe5b92100 0x00007fffe5b95590 Yes (
) /lib64/libnss_dns.so.2
0x00007fffe598d680 0x00007fffe598ed7c Yes () /usr/lib64/gconv/UTF-16.so
0x00007fffe328a9c0 0x00007fffe3f7d47f Yes (
) /lib64/libLLVM-3.6-mesa.so

with this for the symbol

(gdb) p LLVMEnablePrettyStackTrace
$1 = {<text variable, no debug info>} 0x7fffe33f70d0

I do see that libLLVM-3.6-mesa.so is loaded here.

Are you able to tell the problem from this information?

Thanks.

Hi,

Yes - we pick up LLVM symbols from /lib64/libLLVM-3.6-mesa.so. As we discussed at CHEP I have no means of hiding their symbols from our use. We hide our symbols - but that doesn’t help against libLLVM-3.6-mesa.so polluting the dynamic loader’s symbol table.

As I pointed out, they have recognized that as a bug and fixed it. But that doesn’t help us here. I guess the best we can do is ask the CentOS people to backport the fix. Is anyone willing to pursue this?

Cheers, Axel.

Hi,

I suspect a work-around is to build Geant4 without support for OpenGL.

Cheers,
Philippe.