Cling execution __dso_handle missing

Hello, i’ve compiled cling in Windows and Linux. The windows machine is a W7 64 bits with VS2010.
The linux is a OpenSuse 11.4 64 bits with gcc 4.5.1. In windows it was compiled with the visual studio itself
and in linux cmake & make without any further parameter.

In both of them cling compiles ok but when running it i have a missing simbol __dso_handle.
The complete output on windows is :

LLVM ERROR: Could not resolve external global address: __dso_handle
Stack dump:
0. Running pass ‘X86 Machine Code Emitter’ on function ‘@__cxx_global_var_init

In Linux the message is shorter but related to __dso_handle also and doesn’t have Stack Dump.

Do i need to add some parameters to the compilation ? Some folders to the path ?
Something else ?

Thanks

Update:

It seems it won’t work in Windows for now (correct me if i’m wrong) :

http://d.hatena.ne.jp/ohtorii/20110716/1310783800 (Japanese)
http://llvm.org/bugs/show_bug.cgi?id=9213

and i didn’t see a work around on llvm, not at least in the point that is marked by these urls.

Thanks again, Juan

Hello, i don’t know if this is the right way to contribute/discuss some things about cling.
If there is another way, please let me know.

I finally get a cling that starts in windows with MSVC2010. It starts and runs simple commands
like for loops or variable declarations.

The things i made to make cling work on windows are:

  1. First, you have to set -fno-use-cxa-atexit as a parameter to the cling command to avoid the __dso_handle thing.

  2. Only with this cling enters in an infinite loop trying to parse Templates. This patch resolves this infinite loop:

Index: tools/clang/lib/Sema/SemaTemplateInstantiateDecl.cpp
===================================================================
--- tools/clang/lib/Sema/SemaTemplateInstantiateDecl.cpp	(revision 142137)
+++ tools/clang/lib/Sema/SemaTemplateInstantiateDecl.cpp	(working copy)
@@ -3282,17 +3282,24 @@
     PendingInstantiations.insert(PendingInstantiations.begin(),
                                  Pending.begin(), Pending.end());
   }
+  
+  //Visit ensure only once each definition
+  //New definitions wont be evaluated here
+  int PendingLocalImplicitInstantiationsSize = PendingLocalImplicitInstantiations.size();
+  int PendingInstantiationsSize              = PendingInstantiations.size();
 
-  while (!PendingLocalImplicitInstantiations.empty() ||
-         (!LocalOnly && !PendingInstantiations.empty())) {
+  while  (PendingLocalImplicitInstantiationsSize > 0  ||
+         PendingInstantiationsSize > 0) {
     PendingImplicitInstantiation Inst;
 
-    if (PendingLocalImplicitInstantiations.empty()) {
+    if (PendingLocalImplicitInstantiationsSize == 0) {
       Inst = PendingInstantiations.front();
       PendingInstantiations.pop_front();
+	  PendingInstantiationsSize--;
     } else {
       Inst = PendingLocalImplicitInstantiations.front();
       PendingLocalImplicitInstantiations.pop_front();
+	  PendingLocalImplicitInstantiationsSize--;
     }
 
     // Instantiate function definitions
@@ -3340,6 +3347,10 @@
     InstantiateStaticDataMemberDefinition(/*FIXME:*/Inst.second, Var, true,
                                           DefinitionRequired);
   }
+  if ((LangOpts.DelayedTemplateParsing && LateTemplateParser) || (!LangOpts.DelayedTemplateParsing )) {
+	  assert(PendingLocalImplicitInstantiations.empty() && PendingInstantiations.empty() && "Pending instantiation not resolved");
+  }
+  
 }
 
 void Sema::PerformDependentDiagnostics(const DeclContext *Pattern,[/code]

it ensures that the function in charge to resolve templates visit only once each template and also resolve only the template that were present at the moment of the start of the function. 

The problem with this is that although the loop is finished, the templates are not resolved.


3) Publish in clang::Parser  the LateTemplateFunctionCallback and instantiate it before PerformPendingInstantiations(). 
This is to resolve the templates in microsoft mode.

[code]Index: tools/cling/lib/Interpreter/IncrementalParser.cpp
===================================================================
--- tools/cling/lib/Interpreter/IncrementalParser.cpp	(revision 41538)
+++ tools/cling/lib/Interpreter/IncrementalParser.cpp	(working copy)
@@ -295,7 +295,10 @@
       m_Consumer->HandleTopLevelDecl(DeclGroupRef(*I));
     }
 
+	
+	m_Parser->getActions().SetLateTemplateParser(Parser::LateTemplateParserCallback, m_Parser.get());
     getCI()->getSema().PerformPendingInstantiations();
+	m_Parser->getActions().SetLateTemplateParser(NULL, NULL);
 
     DClient.EndSourceFile();
Index: tools/clang/include/clang/Parse/Parser.h
===================================================================
--- tools/clang/include/clang/Parse/Parser.h	(revision 142693)
+++ tools/clang/include/clang/Parse/Parser.h	(working copy)
@@ -243,6 +243,8 @@
 
   DeclGroupPtrTy FinishPendingObjCActions();
 
+  static void LateTemplateParserCallback(void *P, const FunctionDecl *FD);
+
 private:
   //===--------------------------------------------------------------------===//
   // Low-Level token peeking and consumption methods.
@@ -1117,7 +1119,6 @@
     LateParsedTemplateMapT;
   LateParsedTemplateMapT LateParsedTemplateMap;
 
-  static void LateTemplateParserCallback(void *P, const FunctionDecl *FD);
   void LateTemplateParser(const FunctionDecl *FD);
 
   Sema::ParsingClassState
  1. The last thing is that the Preprocessor was left in a state by the template parser that doesn’t allow him to parse more code. Here is a patch to clang ParseTemplate that worked for me:
Index: tools/clang/lib/Parse/ParseTemplate.cpp
===================================================================
--- tools/clang/lib/Parse/ParseTemplate.cpp	(revision 142693)
+++ tools/clang/lib/Parse/ParseTemplate.cpp	(working copy)
@@ -1254,6 +1254,8 @@
   DeclGroupPtrTy grp = Actions.ConvertDeclToDeclGroup(LMT.D);
   if (grp)
     Actions.getASTConsumer().HandleTopLevelDecl(grp.get());
+
+  PP.RemoveTopOfLexerStack();
 }
 
 /// \brief Lex a delayed template function for late parsing.

What do you think about the problems/question/patches ?

Juan

Hi,

[quote=“Juan”]Hello, i’ve compiled cling in Windows and Linux. The windows machine is a W7 64 bits with VS2010.
The linux is a OpenSuse 11.4 64 bits with gcc 4.5.1. In windows it was compiled with the visual studio itself
and in linux cmake & make without any further parameter.

In both of them cling compiles ok but when running it i have a missing simbol __dso_handle.
[/quote]
That is sort-of expected for windows because there are few issues that we need to resolve. Couple of months ago I spent some time on it. I ended up with patches in clang, which they wouldn’t accept. I will have a look again. I remember that there was problem with the M$ linker and the signature of the __cxa_atexit is different.
We have lots of things to do so any help on that is greatly appreciated!

For OpenSuse is weird. Do you link against libc++?

[quote=“Juan”]
Update:

It seems it won’t work in Windows for now (correct me if i’m wrong) :

http://d.hatena.ne.jp/ohtorii/20110716/1310783800 (Japanese)
http://llvm.org/bugs/show_bug.cgi?id=9213

and i didn’t see a work around on llvm, not at least in the point that is marked by these urls.

Thanks again, Juan[/quote]
You might want to read that llvm.org/bugs/show_bug.cgi?id=9213
Cheers,
Vassil

Hi Juan,

There is testsuite for cling in cling_src/test/ (I recommend you to run the testsuite for clang, as well)
You can find how to run the testsuite on windows here: http://clang.llvm.org/get_started.html

Ideally this is not what we want, because cling has to have way to control the destruction of the objects that are created. With -fno-use-cxa-atexit all the objects’ static destructors are called on .q (on exit in cling)

That is completely unexpected. I’d rather first resolve the issue with cxa_atexit and then see what happens to the template instantiations, because they could be related.

Cling policy is to be developed on top of LLVM and clang, all patches for cling itself are greatly appreciated, but the patches in clang and llvm must go via clang’s and llvm’s community. If you can provide an example of source code compiled with clang that ends up with infinite recursion when Sema is instantiating templates this definitely is bug in the compiler and normally you send the example and if you can you send patch candidate.

[quote=“Juan”]
What do you think about the problems/question/patches ?
Juan[/quote]
Great job and reports. However I’d prefer to see the __cxa_atexit resolved first and later we could think how to propose the patches to the LLVM community if we still need them.

Cheers,
Vassil

Hey Vassil,

I’ll try the tests.
About the __cxa_atexit
This says that you are not supposed to emit __cxa_atexit on windows:

http://comments.gmane.org/gmane.comp.compilers.clang.devel/9539

I think __cxa_atexit is not supported in the Win32 platform.
The alternative that Clang builds seems to track all global objects
that needs destruction on module exit and is creating a function responsable
for calling each destructor.

Juan

Hi Juan,
We need such function because we need to control the destruction of the objects. Moreover we enforce it with and provide our own (see Interpreter.cpp).
Correct me if I am wrong but the url that you have sent they are discussing more MinGW-like stuff.
You might want to have a look here as well: http://www.gershnik.com/tips/cpp.asp#atexit_dll
As I said before I remember that there the __cxa_atexit-like function provided by msvc took 2 params less and all this should be matter of #ifdef-s

You asked before how to contribute - having this done would be great start!
BTW I am more puzzled what is the issue that you have with OpenSUSE…

Cheers,
Vassil

Ok, first attempt:

The atexit() function in the microsoft library doesn’t allow you to register destructors as there is not
a place for the parameter this.

To recall the memory from global objects when using -fno-use-cxa-atexit, the clang/llvm leaves a variable
in the module with all the functions that must be called.

Here is a patch to the interprete that does that calls the destructor in this case.
It calls m_ExecutionContext->runStaticDestructorsOnce(module) on the interprete deletion.
If the flag -fno-use-cxa-atexit was present, it should call the destructors on the module globals

+++ tools/cling/lib/Interpreter/Interpreter.cpp	(working copy)
@@ -223,11 +223,12 @@
   //---------------------------------------------------------------------------
   Interpreter::~Interpreter()
   {
-    for (size_t I = 0, N = m_AtExitFuncs.size(); I < N; ++I) {
+	runStaticDestructorsOnce();
+	for (size_t I = 0, N = m_AtExitFuncs.size(); I < N; ++I) {
       const CXAAtExitElement& AEE = m_AtExitFuncs[N - I - 1];
       (*AEE.m_Func)(AEE.m_Arg);
     }
-
+	
     //delete m_prev_module; // Don't do this, the engine does it.
   }
    
@@ -691,6 +692,13 @@
     m_ExecutionContext->runStaticInitializersOnce(module);
   }
 
+  void Interpreter::runStaticDestructorsOnce() const {
+    // Forward to ExecutionContext; should not be called by
+    // anyone except for IncrementalParser.
+    llvm::Module* module = m_IncrParser->GetCodeGenerator()->GetModule();
+	m_ExecutionContext->runStaticDestructorsOnce(module);
+  }
+
   int Interpreter::CXAAtExit(void (*func) (void*), void* arg, void* dso) {
      // Register a CXAAtExit function
      clang::Decl* LastTLD = m_IncrParser->getLastTopLevelDecl();

I will further investigate the use of __cxa_atexit but i think this is also ok to include
this deletion as they are complementary cases.

What do you think ?

The opensuse thing is harder to track to me as i’m not used to develop in linux.
I’ll try it later.

Juan

Hi,
More info about what is used __cxa_atexit you can find in clang’s code generator:
clang/lib/CodeGen/CGDeclCXX:122
You will see that it creates the function and the __dso_handle.
In our case I’d suggest to put the definition of __cxa_atexit and __dso_handle in cling/include/cling/Interpreter/RuntimeUniverse.h so that JIT could find it.

Sorry I should have given the big picture. It will be rather short you can find it on my presentation here: indico.cern.ch/conferenceDispla … fId=159088

Suppose you want to allow statements and expressions to be entered on the global scope. For example:

What you do is wrap it into a function and call the function. However the problem is when you define a variable - it will end up in that function as well, which means that it won’t “be seen” by the other statements. What we do in cling is we extract the variables on the global scope so those variables become global variables. All the global variables are initialized in a hidden function before starting the program (before calling main). However we don’t know the globals that the user would enter. How we are suppose to initialize them? We worked around that by reseting the llvm::Module each time we encounter global variable. Thus we run only the delta of the initializers (we cannot afford to initialize one and the same var twice!). Of course what happens to the initializers happens to the destructors too.

In that case we provide our custom __cxa_atexit which collects the deltas for the destructors and the we call it when we want. Thus we are sure that none of the destrcutors are omitted.

That’s why I am saying that we cannot avoid having __cxa_atexit. In windows maybe the signature is different or maybe it doesn’t exist, but we don’t care. We provide our own implementation and we know when to call it.

If there is unclear please don’t hesitate to ask.

Cheers,
Vassil

[quote]You will see that it creates the function and the __dso_handle.
In our case I’d suggest to put the definition of __cxa_atexit and __dso_handle in cling/include/cling/Interpreter/RuntimeUniverse.h so that JIT could find it.
[/quote]

This was my first attempt to do it but i didn’t get __cxa_atexit nor cling_cxa_atexit called.
I will review and retry this.

Thanks !

Hello, sorry for the delay.

I’ve made the #ifdef for __cxa_atextit and _dso_handle so i can mock them within the ms runtime.
This mostly worked.

But then cling is telling me that a unknown symbols is being used: _Znwj.

_Znwj is the mangled name of the new function in the CXXABI_Itanium, so it won work in the CXXABI_Microsoft.

With this patch you can set the Microsoft ABI

[code] Index: tools/cling/lib/Interpreter/CIFactory.cpp

— tools/cling/lib/Interpreter/CIFactory.cpp (revision 41646)
+++ tools/cling/lib/Interpreter/CIFactory.cpp (working copy)
@@ -171,12 +171,15 @@
}

void CIFactory::SetClingTargetLangOpts(LangOptions& Opts,

  •                                     const TargetInfo& Target) {
    
  •                                     TargetInfo& Target) {
    
    if (Target.getTriple().getOS() == llvm::Triple::Win32) {
    Opts.MicrosoftExt = 1;
    Opts.MSCVersion = 1300;
  • Opts.MicrosoftMode = 1;
     // Should fix http://llvm.org/bugs/show_bug.cgi?id=10528
     Opts.DelayedTemplateParsing = 1;
    
  • Target.setCXXABI("microsoft");
    
    }
    }
    } // end namespace[/code]

but this is not working because most of this file has llvm_unreachable in the method implementations:

llvm.org/svn/llvm-project/cfe/br … Mangle.cpp

I saw in the mailing lists a guy that is contributing some code to this side of clang so i will watch his work.

[quote=“Juan”]Hello, sorry for the delay.

I’ve made the #ifdef for __cxa_atextit and _dso_handle so i can mock them within the ms runtime.
This mostly worked.

But then cling is telling me that a unknown symbols is being used: _Znwj.

_Znwj is the mangled name of the new function in the CXXABI_Itanium, so it won work in the CXXABI_Microsoft.

With this patch you can set the Microsoft ABI

but this is not working because most of this file has llvm_unreachable in the method implementations:

llvm.org/svn/llvm-project/cfe/br … Mangle.cpp

I saw in the mailing lists a guy that is contributing some code to this side of clang so i will watch his work.[/quote]

Hi Juan,
Sorry for the delay (we are having workshop at the US).
Yes indeed the mangling for Windows is incomplete, and the ItaniumABI is not fully supported so I guess there is no other way around but either implementing it by ourselves (I don’t think it would be so hard) or waiting until somebody does it for us. I personally prefer the first (maybe with other people that need it helping). I think it is good idea somebody else to raise that question on the clang-dev (the mailing list of clang) and find more people interested in having that implemented. What do you think?

Otherwise if you want to sort-of test your cxa_atexit implementation you could start cling in c mode (there is no mangling for c) and test it with simple examples. C mode is working to certain extent and for that purpose it will be good enough
You can start cling in c mode with:

cling -x c

Cheers,
Vassil

Vassil.

I’ve been following the cfe mailing list and there is one guy there that is recently contributing
some aspects of the microsoft ABI missing implementations. His nickname is r4start.
He implemented mangling for constructors, is implementing RTTI and has other works also.
As far as i can see his patches are not on trunk yet, he has no commit rights and they are
pending for someone else to do them.

About your proposition, my time to be honest here :slight_smile:
I started this by curiosity, i see cling as a really true c++ CAAS and
if it succeed can be broadly used in other projects. And i wanted to see its real scope.
I can manage to allocate time for helping in the project, something like 10hs a week.
I don’t know if i have the skills needed here, as i’m mostly a windows developer
and for the specific task of the microsoft abi, i have a lot to learn before contributing something
useful for sure. And my english is very bad :slight_smile:.
All that said, i’d like to contribute if you see it useful.

Juan

About Opensuse.

Upgraded the installation to 12.1RC2. (gcc 4.6.2)
Two setups here one with clang/trunk and other with clang/branches/release_30, both compiles ok

With trunk:

  • Until some point cling was returning file not found on #include and after that
    the prompt was ready to receive inputs and it worked for simple commands.
  • After this point it id giving missing __dso_handle on start and it stops.

With release_30:

  • it gives missing __dso_handle on start and it stops.

Juan

[quote=“Juan”]About Opensuse.

Upgraded the installation to 12.1RC2. (gcc 4.6.2)
Two setups here one with clang/trunk and other with clang/branches/release_30, both compiles ok

With trunk:

  • Until some point cling was returning file not found on #include and after that
    the prompt was ready to receive inputs and it worked for simple commands.
  • After this point it id giving missing __dso_handle on start and it stops.

With release_30:

  • it gives missing __dso_handle on start and it stops.

Juan[/quote]

Hi Juan,

Currently we are not supporting the trunk (see root.cern.ch/drupal/content/clin … structions) because there were massive changes in clang that we will be able to adopt once we are back at CERN.

Otherwise I’d really like to see your patches in cling and discuss them. I appreciate your help with Windows. Can you send the patches you have to rootdev@root.cern.ch. I want to have Axel’s and Philippe’s comments as well.

The OpenSUSE issue will be resolved soon and I am not worried about it, I am more worried about Windows support.

I don’t think your English is worse than mine! I don’t think you should worry about it :slight_smile:

Cheers,
Vassil

Vassil,
the mail was rejected by root, any other way to send it ?
my mail is japascual at gmail, just in case

Juan