Since Python 2 support has stopped from January 1st 2020, we are updating our dependencies to Python 3 (ROOT being our main dependency).
Everything is fine in the case that we compile ROOT with Python 3.5 and use PyROOT on a system with Python 3.5. Things break when the system uses Python 3.6. When I do import ROOT in Jupyter Notebook (using the Python 3.6 kernel), I get:
*** Break *** segmentation violation
#0 0x00007f1cbb35a0cb in __GI___waitpid (pid=10103, stat_loc=stat_loc
entry=0x7ffcad982e00, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1 0x00007f1cbb2d2fbb in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
#2 0x00007f1c9a4b74c2 in TUnixSystem::StackTrace() () from /home/ahmad/biodynamo4/build/third_party/root/lib/libCore.so
#3 0x00007f1c9a4b9db3 in TUnixSystem::DispatchSignals(ESignals) () from /home/ahmad/biodynamo4/build/third_party/root/lib/libCore.so
#4 <signal handler called>
#5 0x00007f1cb00944fe in PyInit_libPyROOT () from /home/ahmad/biodynamo4/build/third_party/root/lib/libPyROOT.so
#6 0x000000000061efd5 in _PyImport_LoadDynamicModuleWithSpec ()
#7 0x000000000061ec40 in ?? ()
#8 0x0000000000523c63 in PyCFunction_Call ()
#9 0x000000000057bb0b in _PyEval_EvalFrameDefault ()
#10 0x00000000005754b7 in ?? ()
AFAIK Python should be backwards compatible (at least among the 3.X minor versions). Any idea why the above breaks PyROOT?
Best,
Ahmad
ROOT Version: v6-18/04 Platform: Ubuntu 16.04 Compiler: Not Provided
Where did you learn that Python minor should be backwards compatible? I believe that’s not what we see; there are subtle changes in object layout and interfaces.
so, the minor versions are backwards compatible at least according to https://devguide.python.org/devcycle/ but going from 3.5 to 3.6 you are talking about forward compatibility which is not possible to have (bug fixes, features etc… )
anyway, given that even in conservative distros like EL7 ones have 3.6 IMHO it would be better to rebase on 3.6 (there are quite a lot of important changes especially if you use asyncio and a lot of other need to have features, see https://docs.python.org/3.6/whatsnew/3.6.html and in more detail here https://docs.python.org/3.6/whatsnew/changelog.html#changelog )
The Python language does not provide backward compatibility.
One compatibility-breaking example in the Python 3.7 update:
async and await are now reserved keywords
Simple example:
#!/usr/bin/env python3
async = 42
No problem in Python 3.6. Syntax error in Python 3.7 I am honestly surprised that this is a thing. To make things worse, Python even allows compatibility-breaking changes across micro releases (e.g. 3.7.1 → 3.7.2 is not even guaranteed to maintain compatibility…)
but going from 3.5 to 3.6 you are talking about forward compatibility
@adrian_sev, I’m actually not talking about forward compatibility; I want a system with Python 3.6 to run PyROOT that was ‘compiled against’ a Python 3.5 interpreter. If Python was backward compatible, I should have had no problems. Unfortunately, it seems not to be the case.
How will the ROOT team be releasing ROOT in the future w.r.t. the Python 3 interface? Will you settle on a specific version and expect the user to have that exact version installed? Will there be a more graceful output when there is a version mismatch, instead of a segfault?
it would be better to rebase on 3.6
The problem with fixating on a single release is that different platforms offer different versions from their official repositories (e.g. Linux Ubuntu Xenial only goes up to 3.5.2). I want to avoid users having to go through the trouble of installing Python 3.6 from some other repository, symlinking python3 to python3.6, etc… Additionally, you’ll get non-official repositories to pull stunts like https://launchpad.net/~jonathonf/+archive/ubuntu/python-3.6
This PPA has been removed from public access as part of a protest against the abuse of open-source projects by large companies
As for a more graceful output: a version check is possible, but not all patch version mismatches are fatal and AFAIK, there’s no documentation of which is and which isn’t.
But as-is, ROOT is in pretty good shape: because the bindings are generated at run-time, you only need to recompile PyROOT, not a large range of extension modules, and then you’re good to go.
Thanks for your reply @wlav. I hope the cppyy method will solve the ABI compatibility problem. We will have to reconsider way to integrate PyROOT to our software once it becomes non-experimental.
I agree that the version mismatches are not all going to break PyROOT, but it would still be a good idea to give a warning message in any case (like I get a warning message of cling when my C++ std library is of a different version than the one cling was compiled with).
We would like to avoid our end-users having to rebuild PyROOT to match their system’s Python, as the extra incurred compile time would be relatively large.
I think for now we’d have to go with prebuilding ROOT with the Python 3 version that each of our supported platforms offer in their official repositories, and perform a run-time check on a version mismatch. And when ROOT’s official releases roll out with the cppyy solution, we can switch to that.
The warning message from Cling is pointless as the C++ std library is very backwards compatible and it’s not really about the standard library, but about the ROOT precompiled header, which shouldn’t be distributed in binary form in the first place…
In cppyy-cling (which contains Cling and a very stripped down version of ROOT), I’ve removed the warning: simply compile with an ancient C++ compiler and create the PCH on install or first use. That works on all platforms, makes support for Windows possible, and also solves those Mac caveats for conda.
As I understand it, work is on-going to be able to fix the ROOT build system to have several versions of PyROOT in parallel. (Extension libraries can be tagged to the specific Python interpreter, including ‘m’ and ‘d’ versions, albeit not with patch number.) That will not solve your problems, but could make life easier for your end-users.
Hopefully at some point they’ll also get around to building the header/modules locally. It always bugged me that they have a fully functional (and compatible) C++ compiler in the form of Clang and yet go out of their way to deal with what’s on the system. In fact, you could run ROOT w/o a local C++ compiler by using Cling’s Clang instead.
I am honestly surprised that this is a thing. To make things worse, Python even allows compatibility-breaking changes across micro releases (e.g. 3.7.1 → 3.7.2 is not even guaranteed to maintain compatibility…)