Bug in Python. Or ROOT... PyROOT?

Andrey_Pozdnyakov · August 19, 2019, 1:04pm

Hello,

I’d like to report here a strange behavior with a python script I observe. It contains a call to ROOT.RDataFrame that’s why we suspect it may be related to pyRoot.
The minimal script to reproduce it is this:

from ROOT import *

aaa01 = 1
aaa02 = 1
aaa03 = 1
aaa04 = 1
aaa05 = 1
aaa06 = 1
aaa07 = 1
aaa08 = 1
aaa09 = 1
aaa10 = 1
aaa11 = 1
aaa12 = 1
aaa13 = 1
aaa14 = 1
aaa15 = 1

if True:
    ROOTDataFrame = ROOT.RDataFrame

bbb1 = 1
bbb2 = 1
bbb3 = 1
bbb1 = 1
bbb2 = 1
bbb3 = 1
bbb4 = 1
bbb5 = 1
bbb6 = 1
bbb7 = 1
bbb8 = 1

f01 = 1
f02 = 1
f03 = 1
f04 = 1
f05 = 1
f06 = 1
f07 = 1
f08 = 1
f09 = 1
f10 = 1
f11 = 1
f12 = 1
f13 = 1
f14 = 1

prr = [1 for a in [1]]
xxx0 = 1
xxx1 = 1
xxx2 = 1
for i1, p1 in enumerate(['a']):
    for i2, p2 in enumerate(['a']):
        s1 = 1
        s2 = 2
        s3 = 1
        s4 = 1
        s5 = 1
        s6 = 1
        s7 = 1
        s8 = 1

        s9 = 1
        s10 = 1
        s11 = 1

        s12 = 1
        s13 = 1
        s14 = 1
        s15 = 1
        s16 = 1
        s17 = 1
        s18 = 1
        s19 = 1
        s20 = 1

        branchSuffix = ""

        a1 = 1
        a2 = 1
        a3 = 1
        a4 = 1
        a5 = 1
        a6 = 1

        print branchSuffix

print "Success!"

This script crushes with this error:

  File "BugTest.py", line 88, in <module>
    print branchSuffix
NameError: name 'branchSuffix' is not defined

It is reproducible on lxplus (see file at /afs/cern.ch/user/a/andrey/public/Bugged_script/BugTest.py)

Yes, this code is silly, but you have to have all the lines to reproduce the crash. Particularly:

If any of the lines commented out – the crash is gone.
Most of the variable names are not important, except branchSuffix. Change it to anything else and the crash is gone.

Please tell me I’m not crazy and you can reproduce it.

ROOT Version: 6.18 (also 6.16)
Platform: lxplus (also DESY)
Compiler: N/A

amadio · August 19, 2019, 1:29pm

ROOT doesn’t support doing this anymore, as far as I know.

eguiraud · August 19, 2019, 1:42pm

Indeed this is weird, but as @amadio suggests, changing from ROOT import * to import ROOT fixes things for me – can you check whether that’s the case for you too? I was using the LCG96 release.

Hope this helps!
Enrico

Andrey_Pozdnyakov · August 19, 2019, 6:39pm

I confirm that changing to import ROOT cures the crash.
Perhaps it would good to make this clear to the user somehow that from ROOT import * is not supported. Because all other ROOT related code worked fine with such import. It just created the above mentioned weird behavior (and to be honest I’m not 100% convinced that it is the reason for it).

eguiraud · August 19, 2019, 7:08pm

@etejedor might be able to help further

amadio · August 20, 2019, 6:20am

When you use from ROOT import *, then all ROOT stuff will be in the global namespace. If you then have a name clash between one of your variables and something from ROOT, which seems to be the case above, you will encounter problems. If you want to keep your current style, you can also use from ROOT import foo, bar, baz, which is much safer since you won’t bring all of ROOT’s namespace into your script.

etejedor · August 20, 2019, 7:36am

from ROOT import * is not supported in Python3 (you should get an ImportError if you try it) and, for the reasons Guilherme mentioned, I would discourage its use in Python2 too, since the aim of PyROOT is to be lazy - only lookup stuff if it is actually used from the application. So I would recommend to be specific about what you want to import.

sbinet · August 20, 2019, 8:41am

IIRC, that’s not completely correct.
in the old days (at least), the “star import” was installing an exception hook in the interpreter that would kick in whenever a NameError was raised: it would be the signal for looking into the ROOT module.

presumably, if that exception hook is still in place, that hook has some unintended interference effect w/ your script structure.

when running your script in “verbose” mode (and adding print "--- foo ---" just before your branchSuffix = "" line), I see:

--- foo ---
import systemd # directory /usr/lib64/python2.7/site-packages/systemd
# /usr/lib64/python2.7/site-packages/systemd/__init__.pyc matches /usr/lib64/python2.7/site-packages/systemd/__init__.py
import systemd # precompiled from /usr/lib64/python2.7/site-packages/systemd/__init__.pyc
# /usr/lib64/python2.7/site-packages/systemd/journal.pyc matches /usr/lib64/python2.7/site-packages/systemd/journal.py
import systemd.journal # precompiled from /usr/lib64/python2.7/site-packages/systemd/journal.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/datetime.so", 2);
import datetime # dynamically loaded from /usr/lib64/python2.7/lib-dynload/datetime.so
# /usr/lib64/python2.7/uuid.pyc matches /usr/lib64/python2.7/uuid.py
import uuid # precompiled from /usr/lib64/python2.7/uuid.pyc
import ctypes # directory /usr/lib64/python2.7/ctypes
# /usr/lib64/python2.7/ctypes/__init__.pyc matches /usr/lib64/python2.7/ctypes/__init__.py
import ctypes # precompiled from /usr/lib64/python2.7/ctypes/__init__.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/_ctypes.so", 2);
import _ctypes # dynamically loaded from /usr/lib64/python2.7/lib-dynload/_ctypes.so
# /usr/lib64/python2.7/struct.pyc matches /usr/lib64/python2.7/struct.py
import struct # precompiled from /usr/lib64/python2.7/struct.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/_struct.so", 2);
import _struct # dynamically loaded from /usr/lib64/python2.7/lib-dynload/_struct.so
# /usr/lib64/python2.7/ctypes/_endian.pyc matches /usr/lib64/python2.7/ctypes/_endian.py
import ctypes._endian # precompiled from /usr/lib64/python2.7/ctypes/_endian.pyc
# /usr/lib64/python2.7/ctypes/util.pyc matches /usr/lib64/python2.7/ctypes/util.py
import ctypes.util # precompiled from /usr/lib64/python2.7/ctypes/util.pyc
# /usr/lib64/python2.7/tempfile.pyc matches /usr/lib64/python2.7/tempfile.py
import tempfile # precompiled from /usr/lib64/python2.7/tempfile.pyc
# /usr/lib64/python2.7/io.pyc matches /usr/lib64/python2.7/io.py
import io # precompiled from /usr/lib64/python2.7/io.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/_io.so", 2);
import _io # dynamically loaded from /usr/lib64/python2.7/lib-dynload/_io.so
# /usr/lib64/python2.7/random.pyc matches /usr/lib64/python2.7/random.py
import random # precompiled from /usr/lib64/python2.7/random.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/math.so", 2);
import math # dynamically loaded from /usr/lib64/python2.7/lib-dynload/math.so
dlopen("/usr/lib64/python2.7/lib-dynload/binascii.so", 2);
import binascii # dynamically loaded from /usr/lib64/python2.7/lib-dynload/binascii.so
# /usr/lib64/python2.7/hashlib.pyc matches /usr/lib64/python2.7/hashlib.py
import hashlib # precompiled from /usr/lib64/python2.7/hashlib.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/_hashlib.so", 2);
import _hashlib # dynamically loaded from /usr/lib64/python2.7/lib-dynload/_hashlib.so
dlopen("/usr/lib64/python2.7/lib-dynload/_randommodule.so", 2);
import _random # dynamically loaded from /usr/lib64/python2.7/lib-dynload/_randommodule.so
dlopen("/usr/lib64/python2.7/lib-dynload/cStringIO.so", 2);
import cStringIO # dynamically loaded from /usr/lib64/python2.7/lib-dynload/cStringIO.so
dlopen("/usr/lib64/python2.7/lib-dynload/fcntlmodule.so", 2);
import fcntl # dynamically loaded from /usr/lib64/python2.7/lib-dynload/fcntlmodule.so
import logging # directory /usr/lib64/python2.7/logging
# /usr/lib64/python2.7/logging/__init__.pyc matches /usr/lib64/python2.7/logging/__init__.py
import logging # precompiled from /usr/lib64/python2.7/logging/__init__.pyc
# /usr/lib64/python2.7/weakref.pyc matches /usr/lib64/python2.7/weakref.py
import weakref # precompiled from /usr/lib64/python2.7/weakref.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/syslog.so", 2);
import syslog # dynamically loaded from /usr/lib64/python2.7/lib-dynload/syslog.so
dlopen("/usr/lib64/python2.7/site-packages/systemd/_journal.so", 2);
import systemd._journal # dynamically loaded from /usr/lib64/python2.7/site-packages/systemd/_journal.so
dlopen("/usr/lib64/python2.7/site-packages/systemd/_reader.so", 2);
import systemd._reader # dynamically loaded from /usr/lib64/python2.7/site-packages/systemd/_reader.so
dlopen("/usr/lib64/python2.7/site-packages/systemd/id128.so", 2);
import systemd.id128 # dynamically loaded from /usr/lib64/python2.7/site-packages/systemd/id128.so
import problem # directory /usr/lib64/python2.7/site-packages/problem
# /usr/lib64/python2.7/site-packages/problem/__init__.pyc matches /usr/lib64/python2.7/site-packages/problem/__init__.py
import problem # precompiled from /usr/lib64/python2.7/site-packages/problem/__init__.pyc
# /usr/lib64/python2.7/site-packages/problem/proxies.pyc matches /usr/lib64/python2.7/site-packages/problem/proxies.py
import problem.proxies # precompiled from /usr/lib64/python2.7/site-packages/problem/proxies.pyc
import report # directory /usr/lib64/python2.7/site-packages/report
# /usr/lib64/python2.7/site-packages/report/__init__.pyc matches /usr/lib64/python2.7/site-packages/report/__init__.py
import report # precompiled from /usr/lib64/python2.7/site-packages/report/__init__.pyc
dlopen("/usr/lib64/python2.7/site-packages/report/_pyreport.so", 2);
import report._pyreport # dynamically loaded from /usr/lib64/python2.7/site-packages/report/_pyreport.so
import report.io # directory /usr/lib64/python2.7/site-packages/report/io
# /usr/lib64/python2.7/site-packages/report/io/__init__.pyc matches /usr/lib64/python2.7/site-packages/report/io/__init__.py
import report.io # precompiled from /usr/lib64/python2.7/site-packages/report/io/__init__.pyc
# /usr/lib64/python2.7/site-packages/report/io/TextIO.pyc matches /usr/lib64/python2.7/site-packages/report/io/TextIO.py
import report.io.TextIO # precompiled from /usr/lib64/python2.7/site-packages/report/io/TextIO.pyc
# /usr/lib64/python2.7/site-packages/report/io/GTKIO.pyc matches /usr/lib64/python2.7/site-packages/report/io/GTKIO.py
import report.io.GTKIO # precompiled from /usr/lib64/python2.7/site-packages/report/io/GTKIO.pyc
# /usr/lib64/python2.7/site-packages/report/io/NewtIO.pyc matches /usr/lib64/python2.7/site-packages/report/io/NewtIO.py
import report.io.NewtIO # precompiled from /usr/lib64/python2.7/site-packages/report/io/NewtIO.pyc
# /usr/lib64/python2.7/site-packages/problem/config.pyc matches /usr/lib64/python2.7/site-packages/problem/config.py
import problem.config # precompiled from /usr/lib64/python2.7/site-packages/problem/config.pyc
# /usr/lib64/python2.7/site-packages/problem/exception.pyc matches /usr/lib64/python2.7/site-packages/problem/exception.py
import problem.exception # precompiled from /usr/lib64/python2.7/site-packages/problem/exception.pyc
# /usr/lib64/python2.7/site-packages/problem/tools.pyc matches /usr/lib64/python2.7/site-packages/problem/tools.py
import problem.tools # precompiled from /usr/lib64/python2.7/site-packages/problem/tools.pyc
# /usr/lib64/python2.7/site-packages/problem/watch.pyc matches /usr/lib64/python2.7/site-packages/problem/watch.py
import problem.watch # precompiled from /usr/lib64/python2.7/site-packages/problem/watch.pyc
dlopen("/usr/lib64/python2.7/site-packages/problem/_pyabrt.so", 2);
import problem._pyabrt # dynamically loaded from /usr/lib64/python2.7/site-packages/problem/_pyabrt.so
import dbus # directory /usr/lib64/python2.7/site-packages/dbus
# /usr/lib64/python2.7/site-packages/dbus/__init__.pyc matches /usr/lib64/python2.7/site-packages/dbus/__init__.py
import dbus # precompiled from /usr/lib64/python2.7/site-packages/dbus/__init__.pyc
# /usr/lib64/python2.7/site-packages/dbus/_compat.pyc matches /usr/lib64/python2.7/site-packages/dbus/_compat.py
import dbus._compat # precompiled from /usr/lib64/python2.7/site-packages/dbus/_compat.pyc
# /usr/lib64/python2.7/site-packages/dbus/_version.pyc matches /usr/lib64/python2.7/site-packages/dbus/_version.py
import dbus._version # precompiled from /usr/lib64/python2.7/site-packages/dbus/_version.pyc
# /usr/lib64/python2.7/site-packages/dbus/exceptions.pyc matches /usr/lib64/python2.7/site-packages/dbus/exceptions.py
import dbus.exceptions # precompiled from /usr/lib64/python2.7/site-packages/dbus/exceptions.pyc
# /usr/lib64/python2.7/site-packages/dbus/types.pyc matches /usr/lib64/python2.7/site-packages/dbus/types.py
import dbus.types # precompiled from /usr/lib64/python2.7/site-packages/dbus/types.pyc
dlopen("/usr/lib64/python2.7/site-packages/_dbus_bindings.so", 2);
import _dbus_bindings # dynamically loaded from /usr/lib64/python2.7/site-packages/_dbus_bindings.so
# /usr/lib64/python2.7/site-packages/dbus/_dbus.pyc matches /usr/lib64/python2.7/site-packages/dbus/_dbus.py
import dbus._dbus # precompiled from /usr/lib64/python2.7/site-packages/dbus/_dbus.pyc
# /usr/lib64/python2.7/site-packages/dbus/bus.pyc matches /usr/lib64/python2.7/site-packages/dbus/bus.py
import dbus.bus # precompiled from /usr/lib64/python2.7/site-packages/dbus/bus.pyc
# /usr/lib64/python2.7/site-packages/dbus/connection.pyc matches /usr/lib64/python2.7/site-packages/dbus/connection.py
import dbus.connection # precompiled from /usr/lib64/python2.7/site-packages/dbus/connection.pyc
# /usr/lib64/python2.7/site-packages/dbus/lowlevel.pyc matches /usr/lib64/python2.7/site-packages/dbus/lowlevel.py
import dbus.lowlevel # precompiled from /usr/lib64/python2.7/site-packages/dbus/lowlevel.pyc
# /usr/lib64/python2.7/site-packages/dbus/proxies.pyc matches /usr/lib64/python2.7/site-packages/dbus/proxies.py
import dbus.proxies # precompiled from /usr/lib64/python2.7/site-packages/dbus/proxies.pyc
# /usr/lib64/python2.7/site-packages/dbus/_expat_introspect_parser.pyc matches /usr/lib64/python2.7/site-packages/dbus/_expat_introspect_parser.py
import dbus._expat_introspect_parser # precompiled from /usr/lib64/python2.7/site-packages/dbus/_expat_introspect_parser.pyc
import xml # directory /usr/lib64/python2.7/xml
# /usr/lib64/python2.7/xml/__init__.pyc matches /usr/lib64/python2.7/xml/__init__.py
import xml # precompiled from /usr/lib64/python2.7/xml/__init__.pyc
import xml.parsers # directory /usr/lib64/python2.7/xml/parsers
# /usr/lib64/python2.7/xml/parsers/__init__.pyc matches /usr/lib64/python2.7/xml/parsers/__init__.py
import xml.parsers # precompiled from /usr/lib64/python2.7/xml/parsers/__init__.pyc
# /usr/lib64/python2.7/xml/parsers/expat.pyc matches /usr/lib64/python2.7/xml/parsers/expat.py
import xml.parsers.expat # precompiled from /usr/lib64/python2.7/xml/parsers/expat.pyc
dlopen("/usr/lib64/python2.7/lib-dynload/pyexpat.so", 2);
import pyexpat # dynamically loaded from /usr/lib64/python2.7/lib-dynload/pyexpat.so
Traceback (most recent call last):
  File "t.py", line 90, in <module>
    print branchSuffix
NameError: name 'branchSuffix' is not defined

but no additional module is being imported if I s/branchSuffix/branchSuffixXXX/g. weird huh?
or if I keep branchSuffix but comment out the a1 =1 ... a6 =1 lines, everything’s fine.

wlav · August 20, 2019, 1:10pm

No, it doesn’t import all of ROOT and no, it doesn’t use the exception hook.

from ROOT import * simply shims the lookup function pointer of the global dictionary on the frame of the caller.

And the version that is in PyROOT has a bug in it. Albeit one that takes some effort to expose, as you have: it’s a reference counting problem in the shim that you won’t see until dictionary memory grows too large to extend and gets copied, cleaning it out.

Long since fixed in cppyy master, where it’s called from cppyy.interactive import * to make its intend more clear. It typically adds an overhead of about ~15%, which I personally find a waste for the bit of typing it saves in a script. But for interactive use, it’s great.

system · September 3, 2019, 1:10pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.