PyMVA crashes with conda ROOT

Newbie
,
1

Hi,

I am using ROOT v6_24_6 installed with miniforge3 on macOS Big Sur.
I was attempting to use PyKeras method in TMVA, but PyMVA initialization fails. I paste an iPython snippet that reproduces the crash, with error trace included.

(rootenv) $ ipython
Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 06:27:18) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from ROOT import TMVA

In [2]: TMVA.Tools.Instance()
Out[2]: <cppyy.gbl.TMVA.Tools object at 0x14d608dd0>

In [3]: TMVA.PyMethodBase.PyInitialize()
Error processing line 1 of /Users/rshara01-local/miniforge3/envs/rootenv/lib/python3.8/site-packages/google_auth-1.35.0-py3.9-nspkg.pth:

Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/Users/rshara01-local/miniforge3/envs/rootenv/lib/python3.8/site.py", line 169, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
  File "/Users/rshara01-local/miniforge3/envs/rootenv/lib/python3.8/importlib/util.py", line 14, in <module>
    from contextlib import contextmanager
  File "/Users/rshara01-local/miniforge3/envs/rootenv/lib/python3.8/contextlib.py", line 5, in <module>
    from collections import deque
  File "/Users/rshara01-local/miniforge3/envs/rootenv/lib/python3.8/collections/__init__.py", line 24, in <module>
    import heapq as _heapq
  File "/Users/rshara01-local/miniforge3/envs/rootenv/lib/python3.8/heapq.py", line 581, in <module>
    from _heapq import *
SystemError: initialization of _heapq did not return an extension module

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
...
SystemError: initialization of _heapq did not return an extension module

I saw what appears to be a similar issue (for PyROOT) being discussed here:
https://sft.its.cern.ch/jira/si/jira.issueviews:issue-html/ROOT-9950/ROOT-9950.html

I wonder if it is indeed the same thing, and if there is a workaround?

Thank you in advance,
Ralitsa

ROOT Version: 6.24.6 (from conda-forge)
Platform: MacOS 11.5.2
_Python Version: 3.8.10 (from conda)
Compiler: N/A

2

@etejedor Can you comment?

3

Hi,
The referenced issue was solved, we now build the PyROOT libraries with undefined symbols and we don’t link to the python libs anymore.

Perhaps some problem in PyMethodBase.PyInitialize() @moneta ?

I can’t reproduce with conda ROOT 6.24.02 though.

4

Also tried with 6.24.06 and I can’t reproduce either (this is Ubuntu). Perhaps @moneta can try on a mac? I installed with conda create -c conda-forge --name root_624_6 root

5

I get similar issue on a different macOS version, so it persists for me (though the error message is different this time).

On macOS 10.15.7, I created a fresh venv:
conda create -c conda-forge --name py38root624 python=3.8 root

Then tried again:

In [1]: from ROOT import TMVA
In [2]: TMVA.Tools.Instance()
Out[2]: <cppyy.gbl.TMVA.Tools object at 0x7fcea8b6e2f0>
In [3]: TMVA.PyMethodBase.PyInitialize()
<FATAL>                         : Can't import __main__
***> abort program execution
---------------------------------------------------------------------------
runtime_error                             Traceback (most recent call last)
<ipython-input-3-abfda3a85476> in <module>
----> 1 TMVA.PyMethodBase.PyInitialize()

runtime_error: static void TMVA::PyMethodBase::PyInitialize() =>
    runtime_error: FATAL error

_ROOT Version: 6.24.6 (from conda-forge)
_Platform: MacOS 10.15.7 (Catalina)
Python version: 3.8.12 (from conda-forge)

6

@moneta was also able to reproduce with conda on MacOS but he has a different error (pointer being freed was not allocated).

I tried on a MacOS 10.15 with an installation of ROOT from source and I can’t reproduce, so it might be related to the Conda installation on MacOS. @chrisburr do you have an idea of what can be the issue here?

7

The resolved PyROOT issue I originally referenced suggests that similar issues can be expected for PyMVA as well in the presence of both static& dynamic libs. To quote
Note that I will discuss PyROOT, but PyMVA will need the same treatment.
I wonder if that was already addressed? Apologies if it has been, and this is something unrelated.

8

I see your point now, thanks. I’m still not sure that is the cause of the issue you see, I leave it up to @moneta to decide what to do.

For reference, the PR that changed the way we build the PyROOT libraries to not link against libpython is:

A similar thing could be tried with PyMVA.

9

Hi,
We have a PR merged in master that should fix this Condo issue,

Lorenzo