Python executable used by configure

Nicola_Mori · October 11, 2013, 2:21pm

Hi, I’m trying to install ROOT 5.34.10 on a Debian 7 x86_64 system. For some reason I built and installed in my home folder a custom version of python (2.7.5) which I’m using in place of the system default 2.7.3:

$ python --version
Python 2.7.5

When running configure I get this error:

$ ./configure 
Checking for source directory ... /home/bonechil/ProvaInstaller/source/root_v5.34.10
Configuring for linuxx8664gcc
Traceback (most recent call last):
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 548, in <module>
    main()
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 266, in addusersitepackages
    user_site = getusersitepackages()
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 241, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 231, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sysconfig.py", line 516, in get_config_var
    return get_config_vars().get(name)
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sysconfig.py", line 449, in get_config_vars
    import re
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/re.py", line 105, in <module>
    import sre_compile
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_compile.py", line 14, in <module>
    import sre_parse
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_parse.py", line 17, in <module>
    from sre_constants import *
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_constants.py", line 18, in <module>
    from _sre import MAXREPEAT
ImportError: cannot import name MAXREPEAT
Traceback (most recent call last):
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 548, in <module>
    main()
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 266, in addusersitepackages
    user_site = getusersitepackages()
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 241, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 231, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sysconfig.py", line 516, in get_config_var
    return get_config_vars().get(name)
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sysconfig.py", line 449, in get_config_vars
    import re
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/re.py", line 105, in <module>
    import sre_compile
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_compile.py", line 14, in <module>
    import sre_parse
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_parse.py", line 17, in <module>
    from sre_constants import *
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_constants.py", line 18, in <module>
    from _sre import MAXREPEAT
ImportError: cannot import name MAXREPEAT
Traceback (most recent call last):
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 548, in <module>
    main()
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 266, in addusersitepackages
    user_site = getusersitepackages()
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 241, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 231, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sysconfig.py", line 516, in get_config_var
    return get_config_vars().get(name)
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sysconfig.py", line 449, in get_config_vars
    import re
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/re.py", line 105, in <module>
    import sre_compile
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_compile.py", line 14, in <module>
    import sre_parse
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_parse.py", line 17, in <module>
    from sre_constants import *
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_constants.py", line 18, in <module>
    from _sre import MAXREPEAT
ImportError: cannot import name MAXREPEAT
Checking for GNU Make version >= 3.80 ... ok
Checking for C compiler ... gcc
Checking for C++ compiler ... g++
Checking for linker (LD) ... g++
Checking for F77 compiler ... gfortran
Checking for libX11 ... no
configure: libX11 MUST be installed
See http://root.cern.ch/drupal/content/build-prerequisites

Apart from the final libX11 failure, I get errors related to the python interpreter. These messages come out because at some point the script executes the default system python executable, i.e. /usr/bin/python, instead of the custom one I’ve installed, but since PYTHONPATH points to my custom installation it fails to load libraries:

$ /usr/bin/python --version
Python 2.7.3


$ /usr/bin/python
Traceback (most recent call last):
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 548, in <module>
    main()
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 266, in addusersitepackages
    user_site = getusersitepackages()
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 241, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/site.py", line 231, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sysconfig.py", line 516, in get_config_var
    return get_config_vars().get(name)
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sysconfig.py", line 449, in get_config_vars
    import re
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/re.py", line 105, in <module>
    import sre_compile
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_compile.py", line 14, in <module>
    import sre_parse
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_parse.py", line 17, in <module>
    from sre_constants import *
  File "/home/bonechil/ProvaInstaller/install/PYTHON_2.7.5/lib/python2.7/sre_constants.py", line 18, in <module>
    from _sre import MAXREPEAT
ImportError: cannot import name MAXREPEAT


$ unset PYTHONPATH


$ /usr/bin/python
Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

To understand which python executable it uses I put a “python --version” line at the beginning of configure which returns 2.7.5. I don’t clearly understand what happens within the script: it seems that at least at its beginning it uses my custom python version, but at some points it uses the system one. I searched for some hard-coded “/usr/bin/python” inside the Root source folder and I found this:

$ grep -r \/usr\/bin\/python *
build/package/debian/rules:           -e 's,#!/usr/bin/python[0-9.]*,#,' \

which indeed seems relate to Debin systems, but I don’t really know if this is relevant or not.
Can someone help me understand what’s happening? Thanks.

wlav · October 11, 2013, 4:37pm

Hi,

/usr/bin/python is not hard-coded in the configure script. Rather, it picks it up from PATH. Make sure that during configure, your custom python comes first.

Cheers,
Wim

Nicola_Mori · October 12, 2013, 1:37pm

Hi Wim, the path to my custom executable comes first in PATH. As you can see in my first post, the environment is set so that:

$ python --version
Python 2.7.5
$ /usr/bin/python --version
Python 2.7.3

wlav · October 12, 2013, 9:26pm

Hi,

okay …

Just grasping at straws here, as none of the symptoms seem to make any sense, but does the custom python work properly on the shell with the custom PYTHONPATH?

Also, why does PYTHONPATH need to point to the system libraries of the custom python? Normally, python finds its own system libraries relative to the location of the executable.

Cheers,
WIm

Nicola_Mori · October 12, 2013, 10:19pm

Yes, the custom python shell works without any problem.

My fault, my understanding of how python works is not very good. It might be that clearing PYTHONPATH will fix the execution of configure since there will probably be no more executable-libraries mismatch… I’ll try on Monday when I’ll have access to the Debian machine at work.

Nevertheless, at some point configure ignores PATH and simply executes the system version of python. Given the fact that I did the same “custom python” trick with other distributions (Scientific, Arch) with no error, I think that the problem might be Debian-specific. Indeed, the only file inside the Root folder where I’ve been able to grep “/usr/bin/python” seems to be related to Debian:

$ grep -r \/usr\/bin\/python -r
build/package/debian/rules:655:       -e 's,#!/usr/bin/python[0-9.]*,#,' \

I don’t know if this makes sense at all, but anyway I thank you Wim for your interest and support.

wlav · October 12, 2013, 10:43pm

Hi,

I saw that already about debian as you wrote that above. But AFAIK, that file is only used by the packager, not by configure, and anyway, do look at the whole line:# Remove she-bang line from a python scripts -for i in $(DTMP)$(PREFIX)/lib/root/$(SOVERS)/*.py ; do \ sed -e 's,#!/usr/bin/env python[0-9.]*,#,' \ -e 's,#!/usr/bin/python[0-9.]*,#,' \ < $$i > $$i.tmp ; \ mv $$i.tmp $$i ;
I.e. it should help you, not hurt (but post-install; not that it is executed, nor that it does anything during configure, and also no python scripts in ROOT (genreflex or GDML) have /usr/bin in the first place AFAIK).

The configure script ignores the ‘patch’ number of python btw., and since both the system and the custom one are 2.7, could you configure and build with PATH pointing to the system, PYTHONPATH empty, and --with-python-incdir and --with-python-libdir to the the custom install?

I know that that does not solve the underlying problem, but at least it gets you going.

Cheers,
Wim

Nicola_Mori · October 14, 2013, 1:35pm

I temporarily removed the /usr/bin/python symlink and tried to run configure again to spot where the system version of python is used (it should give an error): This is the output (only the error):

./configure: 1839: ./configure: lsb_release: not found
./configure: 1842: ./configure: lsb_release: not found
./configure: 1845: ./configure: lsb_release: not found

lsb_release is a python script which has a hardcoded /usr/bin/python shebang. So even if no hardcoded path to python is present in Root scripts, it ends up using the system version of python in any case at some point, at least on Debian systems. I guess that being it an upstream feature/issue then there’s nothing to do about it in Root. At least this “custom python” use-case is very rare, I guess, so it will not be a widespread problem. Let’s hope that people having this problem in future will find this thread…

Finally, I’ve been able to make everything work by undefining PYTHONPATH, so that each version of the interpreter uses its own libraries. Thanks you very much Wim for your precious support!

wlav · October 14, 2013, 1:46pm

Hi,

thanks for being so persistent in digging this out. Yes, python is showing up in some surprising places. Might be worth checking out https://pypi.python.org/pypi/virtualenv then.

Cheers,
Wim