Can not start ROOT 6.20.02 from LCG 97 - SystemDir is missing

solodkov · May 25, 2020, 6:28pm

Dear ROOT experts

I have problems with ROOT on one of virtual machines at CERN running standard CentOS 7, see below
Could you please help me

With best regards
Sanya

(aiatlas178) ~> source /cvmfs/sft.cern.ch/lcg/views/LCG_97/x86_64-centos7-gcc8-dbg/setup.sh
(aiatlas178) ~> which root
/cvmfs/sft.cern.ch/lcg/views/LCG_97/x86_64-centos7-gcc8-dbg/bin/root
(aiatlas178) ~> root
root.exe: /workspace/build/projects/ROOT-v6.20.02/src/ROOT/v6.20.02/interpreter/cling/lib/Interpreter/CIFactory.cpp:605: {anonymous}::collectModuleMaps(clang::CompilerInstance&, llvm::SmallVectorImpl<std::__cxx11::basic_string<char> >&)::<lambda(llvm::StringRef, const string&, const string&, std::__cxx11::string&)>: Assertion `llvm::sys::fs::exists(SystemDir) && "Must exist!"' failed.



(aiatlas178) ~> source /cvmfs/sft.cern.ch/lcg/views/LCG_97/x86_64-centos7-gcc8-opt/setup.sh                                                                                                                                                                                                                              (aiatlas178) ~> which root
/cvmfs/sft.cern.ch/lcg/views/LCG_97/x86_64-centos7-gcc8-opt/bin/root
(aiatlas178) ~> root      

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f72c5bba46c in waitpid () from /lib64/libc.so.6
#1  0x00007f72c5b37f62 in do_system () from /lib64/libc.so.6
#2  0x00007f72c6c67233 in TUnixSystem::StackTrace() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCore.so
#3  0x00007f72c6c69a84 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCore.so
#4  <signal handler called>
#5  0x00007f72c31a6483 in (anonymous namespace)::RedirectingFileSystemParser::uniqueOverlayTree((anonymous namespace)::RedirectingFileSystem*, (anonymous namespace)::Entry*, (anonymous namespace)::Entry*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#6  0x00007f72c31a658d in (anonymous namespace)::RedirectingFileSystemParser::uniqueOverlayTree((anonymous namespace)::RedirectingFileSystem*, (anonymous namespace)::Entry*, (anonymous namespace)::Entry*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#7  0x00007f72c31ac97c in (anonymous namespace)::RedirectingFileSystem::create(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, void (*)(llvm::SMDiagnostic const&, void*), llvm::StringRef, void*, llvm::IntrusiveRefCntPtr<clang::vfs::FileSystem>) [clone .isra.519] () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#8  0x00007f72c31ad076 in clang::vfs::getVFSFromYAML(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, void (*)(llvm::SMDiagnostic const&, void*), llvm::StringRef, void*, llvm::IntrusiveRefCntPtr<clang::vfs::FileSystem>) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#9  0x00007f72c11cd566 in (anonymous namespace)::collectModuleMaps(clang::CompilerInstance&, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#10 0x00007f72c11ceb6d in (anonymous namespace)::setupCxxModules(clang::CompilerInstance&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#11 0x00007f72c11d3b68 in (anonymous namespace)::createCIImpl(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, cling::CompilerOptions const&, char const*, std::unique_ptr<clang::ASTConsumer, std::default_delete<clang::ASTConsumer> >, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&, bool, bool) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#12 0x00007f72c11d4beb in cling::CIFactory::createCI(llvm::StringRef, cling::InvocationOptions const&, char const*, std::unique_ptr<clang::ASTConsumer, std::default_delete<clang::ASTConsumer> >, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#13 0x00007f72c1269927 in cling::IncrementalParser::IncrementalParser(cling::Interpreter*, char const*, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#14 0x00007f72c11f31dc in cling::Interpreter::Interpreter(int, char const* const*, char const*, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&, bool, cling::Interpreter const*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#15 0x00007f72c117ded7 in TCling::TCling(char const*, char const*, char const* const*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#16 0x00007f72c117efd4 in CreateInterpreter () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#17 0x00007f72c6b20a19 in TROOT::InitInterpreter() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCore.so
#18 0x00007f72c6b20eff in ROOT::Internal::GetROOT2() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCore.so
#19 0x00007f72c6b2e725 in TApplication::TApplication(char const*, int*, char**, void*, int) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCore.so
#20 0x00007f72c70c8d38 in TRint::TRint(char const*, int*, char**, void*, int, bool) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libRint.so
#21 0x00000000004009be in main ()
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007f72c31a6483 in (anonymous namespace)::RedirectingFileSystemParser::uniqueOverlayTree((anonymous namespace)::RedirectingFileSystem*, (anonymous namespace)::Entry*, (anonymous namespace)::Entry*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#6  0x00007f72c31a658d in (anonymous namespace)::RedirectingFileSystemParser::uniqueOverlayTree((anonymous namespace)::RedirectingFileSystem*, (anonymous namespace)::Entry*, (anonymous namespace)::Entry*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#7  0x00007f72c31ac97c in (anonymous namespace)::RedirectingFileSystem::create(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, void (*)(llvm::SMDiagnostic const&, void*), llvm::StringRef, void*, llvm::IntrusiveRefCntPtr<clang::vfs::FileSystem>) [clone .isra.519] () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#8  0x00007f72c31ad076 in clang::vfs::getVFSFromYAML(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, void (*)(llvm::SMDiagnostic const&, void*), llvm::StringRef, void*, llvm::IntrusiveRefCntPtr<clang::vfs::FileSystem>) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#9  0x00007f72c11cd566 in (anonymous namespace)::collectModuleMaps(clang::CompilerInstance&, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#10 0x00007f72c11ceb6d in (anonymous namespace)::setupCxxModules(clang::CompilerInstance&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#11 0x00007f72c11d3b68 in (anonymous namespace)::createCIImpl(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, cling::CompilerOptions const&, char const*, std::unique_ptr<clang::ASTConsumer, std::default_delete<clang::ASTConsumer> >, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&, bool, bool) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#12 0x00007f72c11d4beb in cling::CIFactory::createCI(llvm::StringRef, cling::InvocationOptions const&, char const*, std::unique_ptr<clang::ASTConsumer, std::default_delete<clang::ASTConsumer> >, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#13 0x00007f72c1269927 in cling::IncrementalParser::IncrementalParser(cling::Interpreter*, char const*, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#14 0x00007f72c11f31dc in cling::Interpreter::Interpreter(int, char const* const*, char const*, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&, bool, cling::Interpreter const*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#15 0x00007f72c117ded7 in TCling::TCling(char const*, char const*, char const* const*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#16 0x00007f72c117efd4 in CreateInterpreter () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCling.so
#17 0x00007f72c6b20a19 in TROOT::InitInterpreter() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCore.so
#18 0x00007f72c6b20eff in ROOT::Internal::GetROOT2() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCore.so
#19 0x00007f72c6b2e725 in TApplication::TApplication(char const*, int*, char**, void*, int) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libCore.so
#20 0x00007f72c70c8d38 in TRint::TRint(char const*, int*, char**, void*, int, bool) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-10e75/x86_64-centos7-gcc8-opt/lib/libRint.so
#21 0x00000000004009be in main ()
===========================================================

eguiraud · May 26, 2020, 9:42am

Hi @solodkov,
there seems to be some problem in your environment: /workspace/build/projects/ROOT-v6.20.02 in your first error message looks suspicious. Plus, if you just log into lxplus and use that same exact LCG view, things seem fine.

What happens if you source a different ROOT release, e.g. with source /cvmfs/sft.cern.ch/lcg/views/LCG_96/x86_64-centos7-gcc8-dbg/setup.sh? Does the error still mention /workspace/build/projects/ROOT-v6.20.02/ then? If yes, you might have environment variables like LD_LIBRARY_PATH, PATH, ROOT_INCLUDE_PATH or others that point to a different ROOT installation.

I am sorry I cannot be of more help! Do you know what the installation at /workspace/build/projects/ROOT-v6.20.02/ is?

Cheers,
Enrico

vvassilev · May 26, 2020, 10:19am

If you execute echo "" | c++ -fsyntax-only -xc++ -v - and echo "" | c++ -fsyntax-only -xc -v - what would be the output. I suspect the location of libstdc++ cannot be discovered by ROOT.

solodkov · May 26, 2020, 10:26am

environment is identical on lxplus and on this machine
/workspace directory is not mentioned anywhere and it doesn’t exist
I guess it’s just location where ROOT was compiled by LCG team

eguiraud · May 26, 2020, 10:30am

Ah, you are right of course.

solodkov · May 26, 2020, 10:46am

I tried to start older ROOT from LCG_96 and the error was different
ROOT clearly complained about missing headers

/cvmfs/sft.cern.ch/lcg/contrib/gcc/6.2.0native/x86_64-slc6/bin/ …/lib/gcc/x86_64-pc-linux-gnu/6.2.0/…/…/…/…/include/c++/6.2.0/cctype:42:10: fatal error: ‘ctype.h’ file not found

So, I’ve installed glibc-headers rpm and was able to start ROOT, both 6.18 and 6.20

Could you please make error messages in new ROOT more informative

Cheers, Sanya

eguiraud · May 26, 2020, 10:52am

Indeed that’s a reasonable request! Could you please open a ticket at https://sft.its.cern.ch/jira/projects/ROOT/issues with a description of the problem you had and the request for improvement? So this does not get lost.

Cheers,
Enrico

vvassilev · May 26, 2020, 12:26pm

Currently (and wrongfully) ROOT assumes that glibc lives in /usr/include (as on 99% of the machines). This commit in an open PR intends to resolve such issues: [cxxmodules] Teach cling to build modules; Build dependent modules explicitly. by vgvassilev · Pull Request #5521 · root-project/root · GitHub

system · June 9, 2020, 12:26pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.