pyROOT hanging on TTreeIndex


ROOT Version: 6.14.04
Platform: SLC6
Compiler: gcc62


Hi, I noticed that one of my pyROOT scripts started hanging recently, so I tried to track the exact reason and it seems it has to do with the loading of the following library:

from ROOT import TTreeIndex

This seems to be a problem on several machines, including ones on lxplus, as well as with interactive python etc. Simply writing the line above in a test.py file, and running:

python -m trace --trace test.py

I see the script hanging on the last line of

--- modulename: ROOT, funcname: _importhook
ROOT.py(456):    if name[0:5] == 'ROOT.':
ROOT.py(461):    return _orig_ihook( name, *args, **kwds )
atexit.py(12): _exithandlers = []
atexit.py(13): def _run_exitfuncs():
atexit.py(37): def register(func, *targs, **kargs):
atexit.py(49): if hasattr(sys, "exitfunc"):
atexit.py(52): sys.exitfunc = _run_exitfuncs
atexit.py(54): if __name__ == "__main__":
ROOT.py(766): def cleanup():
ROOT.py(826): atexit.register( cleanup )
 --- modulename: atexit, funcname: register
atexit.py(46):     _exithandlers.append((func, targs, kargs))
atexit.py(47):     return func
ROOT.py(827): del cleanup, atexit
 --- modulename: ROOT, funcname: __getattr1
ROOT.py(560):       if name == '__path__':
ROOT.py(561):          raise AttributeError( name )
 --- modulename: ROOT, funcname: __getattr1
ROOT.py(560):       if name == '__path__':
ROOT.py(564):       self.__finalSetup()
 --- modulename: ROOT, funcname: __finalSetup
ROOT.py(626):       self.__dict__[ 'gROOT' ] = _root.gROOT
ROOT.py(629):       self.__class__.__getattr__ = self.__class__.__getattr2
ROOT.py(630):       del self.__class__.__getattr2
ROOT.py(631):       self.__class__.__setattr__ = self.__class__.__setattr2
ROOT.py(632):       del self.__class__.__setattr2
ROOT.py(636):       hasargv = hasattr( sys, 'argv' )
ROOT.py(637):       if hasargv and PyConfig.IgnoreCommandLineOptions:
ROOT.py(641):       appc = _root.CreateScopeProxy( 'PyROOT::TPyROOTApplication' )
ROOT.py(642):       if (not _builtin_cppyy and appc.CreatePyROOTApplication()) or _builtin_cppyy:

Any help very welcome!

Hi @Demosthene

I tried to reproduce on lxplus but I was unable. Do you know any particular lxplus machine where you see this hanging? Which ROOT installation are you using, is it on CVMFS?

Cheers,
Enric

Hi @etejedor,
Thanks for looking into this!

Logging on to lxplus033 for instance, with python 2.7.4-x86_64-slc6-gcc48 and root 6.14.04-x86_64-slc6-gcc62-opt (from cvmfs, /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/bin/root), I can reproduce this problem.

Hi @Demosthene

I still can’t reproduce, this is what I do in case you want to try the same:

12:07 $ ssh etejedor@lxplus033
...
[etejedor@lxplus033 ~]$ source /cvmfs/sft.cern.ch/lcg/views/LCG_94/x86_64-slc6-gcc62-opt/setup.sh
[etejedor@lxplus033 ~]$ python
Python 2.7.15 (default, Jul 30 2018, 09:38:14) 
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ROOT import TTreeIndex
>>> 

Hi @etejedor,
Reproducing these exact commands still hangs for me. Looking at my full code I left to run, I see it now moved past that import, but I would imagine at least in O(10) minutes if not >1h - which is a problem, and definitely wasn’t the case previously.
I’m now running time on lxplus bash, for the one-line-import script, I’ll post the results here.

Is there anything I should check? I don’t have anything set up in my ~/.bashrc that could affect this…

Do you have anything on your home directory, in ~/.local? Any python package installed there that could interfere?

In ~/.local/lib/python2.7/site-packages/, I see numpy and ruamel, nothing else.

Just to discard them, I would:

  • Temporarily rename your ~/.bashrc to something else.
  • Temporarily rename your ~/.local to something else.

Then log out from lxplus and log in again, run the commands I posted above and see what happens.

Still no progress :confused:

Ok, I’m throwing more ideas:

  • If instead of doing from ROOT import TTreeIndex you do from ROOT import TH1F, does it hang or not? What about just import ROOT?
  • If you source another LCG view, for instance ROOT master from yesterday, what happens?
    source /cvmfs/sft.cern.ch/lcg/views/dev3/latest/x86_64-slc6-gcc62-opt/setup.sh
  • from ROOT import TH1F and import ROOT work fine
  • switching to the dev3 LCG, all three imports work :slight_smile:

Alright, it might be an issue with the CVMFS installation of ROOT for LCG 94, or a temporary glitch in CVMFS that is not serving you some file you need.

I would try LCG 94 in a couple more nodes in lxplus to check if it is an issue of the CVMFS mount of that node.

Is it an option for you to work with dev3 for now, so that you are not blocked by this?

Thanks @etejedor!
I tried LCG 94 on a couple more nodes as you suggested, same story. I can use dev3 for now, but it gives me ROOT Version: 6.17/01; I’m developing code that needs to be compatible for users with 6.14, so we’ll see… :slight_smile:

Hi @Demosthene

ROOT 6.16 will be out very soon, will you move to that version once it is published? If so, please let us know if you still see the issue with that version.

Anyway, it looks more like a CVMFS issue, since I have tried it in several nodes and I cannot make it hang…

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.