PyROOT with Apache and django

ROOT
1

We have recently developed some web based interface that utilizes the pyROOT package to create interactive histograms via jsROOT with a django application. This development went very smoothly while using the provided webserver with django. Alas, upon transitioning to an apcahe server to test before release we have run into an issue that has stopped us dead in our tracks. It appears the the pyROOT library is getting hung up during loading causing the web pages to never render. We were able to increase the verbosity of the output to see that indeed the last import is related to ROOT. Here are the status messages:

[Thu Feb 22 10:38:14.128164 2018] [wsgi:error] [pid 18073] [client ] # /opt/root/6.12.06/lib/__pycache__/ROOT.cpython-36.pyc matches /opt/root/6.12.06/lib/ROOT.py
[Thu Feb 22 10:38:14.129220 2018] [wsgi:error] [pid 18073] [client ] # code object from '/opt/root/6.12.06/lib/__pycache__/ROOT.cpython-36.pyc'
[Thu Feb 22 10:38:14.129470 2018] [wsgi:error] [pid 18073] [client ] # /usr/lib64/python3.6/__pycache__/__future__.cpython-36.pyc matches /usr/lib64/python3.6/__future__.py
[Thu Feb 22 10:38:14.129588 2018] [wsgi:error] [pid 18073] [client ] # code object from '/usr/lib64/python3.6/__pycache__/__future__.cpython-36.pyc'
[Thu Feb 22 10:38:14.129662 2018] [wsgi:error] [pid 18073] [client ] import '__future__' # <_frozen_importlib_external.SourceFileLoader object at 0x7f3d3312c358>
[Thu Feb 22 10:38:14.129908 2018] [wsgi:error] [pid 18073] [client ] # /opt/root/6.12.06/lib/__pycache__/cppyy.cpython-36.pyc matches /opt/root/6.12.06/lib/cppyy.py
[Thu Feb 22 10:38:14.130011 2018] [wsgi:error] [pid 18073] [client ] # code object from '/opt/root/6.12.06/lib/__pycache__/cppyy.cpython-36.pyc'
[Thu Feb 22 10:38:14.497359 2018] [wsgi:error] [pid 18073] [client ] # extension module 'libPyROOT' loaded from '/opt/root/6.12.06/lib/libPyROOT.so'
[Thu Feb 22 10:38:14.497480 2018] [wsgi:error] [pid 18073] [client ] # extension module 'libPyROOT' executed from '/opt/root/6.12.06/lib/libPyROOT.so'
[Thu Feb 22 10:38:14.497517 2018] [wsgi:error] [pid 18073] [client ] import 'libPyROOT' # <_frozen_importlib_external.ExtensionFileLoader object at 0x7f3d331344e0>

It appears to be when loading the C++ python extension something hangs and the process gets stuck. The site loads without issue when commenting out the import ROOT statement. Any suggestions would be greatly appreciated. Perhaps @wlav might have some insight?

2

After some hacking we find that the following line in cppyy.py was the culprit for hanging on load. If we comment these lines out then there is no issue render the main page, but we then fail to render pages that make use of ROOT components, for example simply using ROOT.TH1F hangs.

#--- Enable Autoloading ignoring possible error for the time being
try:    _backend.gInterpreter.EnableAutoLoading()
except: pass
3

That gInterpreter call is likely a red herring: it basically shows that the problem is with the first call.

4

Agreed, just not sure where to go from here. I suspect it has something to do with the crippled environment that the httpd process are run in, but I’m not sure what I need to correct.

5

Did you copy over (and gave access to) the ‘etc’ directory of the ROOT installation? There are a bunch spurious dependencies, including etc/class.rules and etc/plugins that cause weird behaviour and crashes if not present.

6

The etc folder should be accessible. After some more tracing it seems to hang in PyROOT::TPyROOTApplication::CreatePyROOTApplication or at least at the point where that call is made.

7

How does it know where the etc folder is? There is no access to root-config as the path is not setup correctly for it and setting it in the environment variables for httpd leads to a 500 error.

8

$ROOTSYS

(blahblah to hit 20 chars to allow posting)

9

Already thought of that one. Have it in the httpd conf.

10

So the hangup is in CreatePyROOTApplicatoin, not in the TPyROOTApplication constructor? Looking at the code, the latter is doing all the ROOT thingies, the former just massages argv/argc. (Aside, it does that precisely b/c there is not argv in sys under the apache module.)

11

Just before the following call in ROOT.py is the last time the python script is responsive. I haven’t been able to probe into the C++ aspect yet.

def __finalSetup( self ):
    # ...
    appc.CreatePyROOTApplication()
12

There’s a boatload of stuff already “live” at that point, including all of cling, but this is likely the first call where wrappers are being generated by ROOT/meta. Those do not show errors if anything fails. How about the $ROOTSYS/include directory? The dictionary payload contains #include’s to many ROOT headers.

13

Everything in the install directory is readable for everyone. Maybe I have the httpd configured incorrectly so access there is denied.

14

Before diving into wrapper code and unless you have a simple way of tracing C++, add this line:

_root.LookupCppEntity('kRed').__get__(_root)

just before the CreatePyROOTApplication() call. See whether it passes that.

Point being that in terms of pure initialization, the only difference should be wrapper creation (I’m assuming that the PyROOT dictionary is already loaded, but that one for sure is pulled in by CreateScopeProxy( ‘PyROOT::TPyROOTApplication’ ), which so far seems to be harmless enough).

15

This passes without issue. I even get the value 632.

16

As far as I can tell the C++ call never gets processed. I added a simple write to a file at the top of that call to /tmp/root.out and the file never gets created. Where /tmp has read and write permissions for all users (also inspected the /var/tmp/systemd-private-###-htpd.service-###/tmp dir and the file is missing). The file does get created if the following script is evaluated by a user on the terminal:

import ROOT
h = ROOT.TH1F("h","H",100,0,1)
17

Print the generated code (‘wrapper’) in TClingCallFunc::make_wrapper in core/metacling/src/TClingCallFunc.cxx, to see whether it properly passes that point. (If it does, that would bisect the problem search neatly. If not, then the problem can still be either PyROOT or Cling.)

18

A simple printf will get passed through to python?

19

Other way around (python prints through C). Either way, I’d use fprintf(stderr, …). The assumption here is of course that apache dups the file descriptors to redirect output (which is by far the easiest thing to do).

20

Good call on stderr. Cling does print the wrapper

#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wformat-security"
__attribute__((used)) extern "C" void __cf_0(void* obj, int nargs, void** args, void* ret)
{
   if (nargs == 0) {
      if (ret) {
         new (ret) (bool) (((PyROOT::TPyROOTApplication*)obj)->CreatePyROOTApplication());
         return;
      }
      else {
         ((PyROOT::TPyROOTApplication*)obj)->CreatePyROOTApplication();
         return;
      }
   }
   if (nargs == 1) {
      if (ret) {
         new (ret) (bool) (((PyROOT::TPyROOTApplication*)obj)->CreatePyROOTApplication(*(bool*)args[0]));
         return;
      }
      else {
         ((PyROOT::TPyROOTApplication*)obj)->CreatePyROOTApplication(*(bool*)args[0]);
         return;
      }
   }
}
#pragma clang diagnostic pop