Error in modulemap.overlay! ...on Compute Nodes

ROOT
1

ROOT Version: 6.24/06
Platform: linuxx8664gcc
Compiler: g++ 10.3.0

I have compiled ROOT on the head node of our computing cluster, which works splendidly, and now I am trying to run it on the computing nodes of the cluster. However, I get the error that the relative path at root level is not discoverable.

error: entry with relative path at the root level is not discoverable
{ 'name': '', 'type': 'directory',
          ^~
Error in modulemap.overlay!

 *** Break *** segmentation violation

I tried using ldd to see what libraries root expects to find and all are available on the compute nodes. This was not a problem with root5.

Here is the entire stack trace.

===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f0cf3a8041c in waitpid () from /lib64/libc.so.6
#1  0x00007f0cf39fdf12 in do_system () from /lib64/libc.so.6
#2  0x00007f0cf4b8551b in TUnixSystem::StackTrace() () from /home/software/root/root-6.24.06/install/lib/libCore.so
#3  0x00007f0cf4b82a55 in TUnixSystem::DispatchSignals(ESignals) () from /home/software/root/root-6.24.06/install/lib/libCore.so
#4  <signal handler called>
#5  0x00007f0cf106f8d6 in llvm::vfs::OverlayFileSystem::pushOverlay(llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#6  0x00007f0cee544586 in (anonymous namespace)::collectModuleMaps(clang::CompilerInstance&, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&) [clone .constprop.0] () from /home/software/root/root-6.24.06/install/lib/libCling.so
#7  0x00007f0cee544ceb in (anonymous namespace)::setupCxxModules(clang::CompilerInstance&) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#8  0x00007f0cee5484e7 in (anonymous namespace)::createCIImpl(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, cling::CompilerOptions const&, char const*, std::unique_ptr<clang::ASTConsumer, std::default_delete<clang::ASTConsumer> >, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&, bool, bool) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#9  0x00007f0cee54a2d7 in cling::CIFactory::createCI(llvm::StringRef, cling::InvocationOptions const&, char const*, std::unique_ptr<clang::ASTConsumer, std::default_delete<clang::ASTConsumer> >, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#10 0x00007f0cee5fe102 in cling::IncrementalParser::IncrementalParser(cling::Interpreter*, char const*, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#11 0x00007f0cee57a83b in cling::Interpreter::Interpreter(int, char const* const*, char const*, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&, bool, cling::Interpreter const*) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#12 0x00007f0cee4daa01 in TCling::TCling(char const*, char const*, char const* const*) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#13 0x00007f0cee4dc941 in CreateInterpreter () from /home/software/root/root-6.24.06/install/lib/libCling.so
#14 0x00007f0cf4a2c8fc in TROOT::InitInterpreter() () from /home/software/root/root-6.24.06/install/lib/libCore.so
#15 0x00007f0cf4a2cdaf in ROOT::Internal::GetROOT2() () from /home/software/root/root-6.24.06/install/lib/libCore.so
#16 0x00007f0cf4a392dd in TApplication::TApplication(char const*, int*, char**, void*, int) () from /home/software/root/root-6.24.06/install/lib/libCore.so
#17 0x00007f0cf4fb70c7 in TRint::TRint(char const*, int*, char**, void*, int, bool) () from /home/software/root/root-6.24.06/install/lib/libRint.so
#18 0x000000000040095e in main ()
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum https://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007f0cf106f8d6 in llvm::vfs::OverlayFileSystem::pushOverlay(llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#6  0x00007f0cee544586 in (anonymous namespace)::collectModuleMaps(clang::CompilerInstance&, llvm::SmallVectorImpl<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&) [clone .constprop.0] () from /home/software/root/root-6.24.06/install/lib/libCling.so
#7  0x00007f0cee544ceb in (anonymous namespace)::setupCxxModules(clang::CompilerInstance&) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#8  0x00007f0cee5484e7 in (anonymous namespace)::createCIImpl(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, cling::CompilerOptions const&, char const*, std::unique_ptr<clang::ASTConsumer, std::default_delete<clang::ASTConsumer> >, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&, bool, bool) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#9  0x00007f0cee54a2d7 in cling::CIFactory::createCI(llvm::StringRef, cling::InvocationOptions const&, char const*, std::unique_ptr<clang::ASTConsumer, std::default_delete<clang::ASTConsumer> >, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#10 0x00007f0cee5fe102 in cling::IncrementalParser::IncrementalParser(cling::Interpreter*, char const*, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#11 0x00007f0cee57a83b in cling::Interpreter::Interpreter(int, char const* const*, char const*, std::vector<std::shared_ptr<clang::ModuleFileExtension>, std::allocator<std::shared_ptr<clang::ModuleFileExtension> > > const&, bool, cling::Interpreter const*) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#12 0x00007f0cee4daa01 in TCling::TCling(char const*, char const*, char const* const*) () from /home/software/root/root-6.24.06/install/lib/libCling.so
#13 0x00007f0cee4dc941 in CreateInterpreter () from /home/software/root/root-6.24.06/install/lib/libCling.so
#14 0x00007f0cf4a2c8fc in TROOT::InitInterpreter() () from /home/software/root/root-6.24.06/install/lib/libCore.so
#15 0x00007f0cf4a2cdaf in ROOT::Internal::GetROOT2() () from /home/software/root/root-6.24.06/install/lib/libCore.so
#16 0x00007f0cf4a392dd in TApplication::TApplication(char const*, int*, char**, void*, int) () from /home/software/root/root-6.24.06/install/lib/libCore.so
#17 0x00007f0cf4fb70c7 in TRint::TRint(char const*, int*, char**, void*, int, bool) () from /home/software/root/root-6.24.06/install/lib/libRint.so
#18 0x000000000040095e in main ()
===========================================================
2

Dear Jimmy,

thanks for posting, sorry to hear ROOT is not working out-of-the-box. Some questions to hopefully get closer to understanding the problem:

  1. Does ROOT work on the head node, where you compiled it? Is the environment on computing nodes somehow different?
  2. How are you “loading” ROOT? Do you run it from the directory it was installed into, or is it moved after the installation step?

Cheers,
Jonas

3

Hi Jonas,
Thanks for replying.

  1. ROOT works perfectly on the head node. The environment on the compute nodes can be slightly different. However, the directory for the compilers and ROOT are available on both compute and head nodes. The slight differences are really minute, for example, gdb is installed on the head node but not on compute nodes. The head and compute nodes don’t share a /usr/bin and /usr/lib, but all the dependencies are also installed on the compute nodes.

  2. I run ROOT from the directory it was installed. Nothing is moved. The disk where ROOT and GCC is installed is also available to the compute nodes.

The problem seems to come from cling (interpreter/cling/lib/Interpreter/CIFactory.cpp and/or interpreter/llvm/src/lib/Support/VirtualFileSystem.cpp). If it could just tell me which directory it looked for but could not find I could pin point the issue. At the moment the error isn’t helping. Do you if there is a way to make it tell me what is missing?

Cheers,
Jimmy

4

Hi,

could the slight differences in principle include the availability / location of development headers? Due to Cling’s C++ interpretation, they are needed for the proper functioning of ROOT.

There isn’t a direct way to debug this, but given that you already compile from source, you could apply the following debug printout to dump the generated modulemap.overlay:

diff --git a/interpreter/cling/lib/Interpreter/CIFactory.cpp b/interpreter/cling/lib/Interpreter/CIFactory.cpp
index c4aced66da..d6f2174fb8 100644
--- a/interpreter/cling/lib/Interpreter/CIFactory.cpp
+++ b/interpreter/cling/lib/Interpreter/CIFactory.cpp
@@ -739,6 +739,7 @@ namespace {
       MOverlay.insert(0, "{\n 'version': 0,\n 'roots': [\n");

       MOverlay += "]\n }\n ]\n }\n";
+      llvm::errs() << MOverlay << "\n";

       const std::string VfsOverlayFileName = "modulemap.overlay.yaml";
       if (HSOpts.Verbose)

Please note that ROOT 6.24/06 is rather old; while the patch is written for master it should also apply for 6.24. However, please note that we might not be able to fix problems in 6.24 should there be an issue on our side.

Jonas

5

Hi,

In principle, yes. However, in my many weeks of trying to fix this issue, I installed all header in a directory available to all nodes just to test if that was the issue. That solution did not work, and this allowed to rule out development headers as the issue. But I am open to test any other solution you have.

In the mean time, I patched CIFactory.cpp and I have some more in formation.

{
 'version': 0,
 'roots': [
{ 'name': '', 'type': 'directory',
'contents': [
   { 'name': 'libc.modulemap', 'type': 'file',
  'external-contents': '/home/software/root/root-6.24.06/install/etc/cling/libc.modulemap'
}
 ]
 },
{ 'name': '/home/software/gcc/gcc10/install/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../include/c++/10.3.0', 'type': 'directory',
'contents': [
   { 'name': 'std.modulemap', 'type': 'file',
  'external-contents': '/home/software/root/root-6.24.06/install/etc/cling/std.modulemap'
}
 ]
 }]
 }
 ]
 }

error: entry with relative path at the root level is not discoverable
{ 'name': '', 'type': 'directory',
          ^~
Error in modulemap.overlay!

This empty directory name seems to be the issue. Does Cling interpret this ' ' as the root directory (/) or the src directory for root? Why would it interpret it correctly for head node and incorrectly for compute nodes? (I might be way off with my interpretation of the overlay).

For the head node this line look like:

{ 'name': '/usr/include', 'type': 'directory',

and indeed ‘/usr/include’ does not exist on the compute nodes. So it looks like I need to find a way to compile root and have it look for all headers in a common (to head and compute nodes) directory and never look in ‘/usr/include’

Jimmy

6

Yes, /usr/include has the development headers for the libc, the most basic library for compiled programs. You will need to get this installed on the compute nodes (probably by the system administrator), otherwise ROOT will not work.

7

Thank you. Your patch helped me find the root of the problem, and it was a simple fix from there.

Sincerely,
Jimmy

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.