Replace NaNs in RDF

[riccardomanzoni]

Hi,

I want to replace all NaNs in a RDataFrame column before taking a Snapshot.

I’ve tried with the Numba.Declare decorator, for example

@ROOT.Numba.Declare(['float'], 'float')
def heal_nans(x):
    if x != x:
        return 1.
    else:
        return x

as well as

@ROOT.Numba.Declare(['float'], 'float')
def heal_nans(x):
    return np.nan_to_num(x)

and both crash with segmentation violation the moment I declare them.
But then it turns out that if I disable ROOT.EnableImplicitMT() all is fine.

Is this expected / have I missed any warnings in the docs (I’ve referred to ROOT: tutorials/pyroot/pyroot004_NumbaDeclare.py File Reference) / am I doing something stupid?

Thanks a lot!

---

ROOT Version: 6.22/06
Platform: MacOS BigSur
Compiler: x86_64-apple-darwin13.4.0-clang++

---

[eguiraud]

Hi @riccardomanzoni ,
this is not expected, could you please share a minimal reproducer, including a few events, so we can take a look?

Cheers,
Enrico

[riccardomanzoni]

Hi @eguiraud,

here’s a minimal example that crashes as soon as I try to define the function

(hammer3p8) MacBook-Pro:~ master $ ipython

Python 3.8.6 | packaged by conda-forge | (default, Dec 26 2020, 04:50:20)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.20.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import ROOT

In [2]: ROOT.EnableImplicitMT()

In [3]: @ROOT.Numba.Declare(['float'], 'float')
   ...: def heal_nans(x):
   ...:     if x != x:
   ...:         return 1.
   ...:     else:
   ...:         return x
   ...:
 *** Break *** segmentation violation
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libCore.6.22.06.so] TUnixSystem::DispatchSignals(ESignals) (no debug info)
[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[<unknown binary>] (no debug info)
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libLLVM-10.dylib] llvm::PMTopLevelManager::schedulePass(llvm::Pass*) (no debug info)
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libLLVM-10.dylib] llvm::PMTopLevelManager::schedulePass(llvm::Pass*) (no debug info)
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libLLVM-10.dylib] llvm::PassManagerBuilder::populateModulePassManager(llvm::legacy::PassManagerBase&) (no debug info)
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libffi.7.dylib] ffi_call_unix64 (no debug info)
[<unknown binary>] (no debug info)
 *** Break *** segmentation violation
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libCore.6.22.06.so] TUnixSystem::DispatchSignals(ESignals) (no debug info)
[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[<unknown binary>] (no debug info)
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libLLVM-10.dylib] llvm::PMTopLevelManager::schedulePass(llvm::Pass*) (no debug info)
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libLLVM-10.dylib] llvm::PMTopLevelManager::schedulePass(llvm::Pass*) (no debug info)
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libLLVM-10.dylib] llvm::PassManagerBuilder::populateModulePassManager(llvm::legacy::PassManagerBase&) (no debug info)
[/Users/manzoni/opt/anaconda3/envs/hammer3p8/lib/libffi.7.dylib] ffi_call_unix64 (no debug info)
[<unknown binary>] (no debug info)

and here the same thing without enabling implicit multithreading

(hammer3p8) MacBook-Pro:~ master $ ipython
Python 3.8.6 | packaged by conda-forge | (default, Dec 26 2020, 04:50:20)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.20.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import ROOT

In [2]: @ROOT.Numba.Declare(['float'], 'float')
   ...: def heal_nans(x):
   ...:     if x != x:
   ...:         return 1.
   ...:     else:
   ...:         return x
   ...:

In [3]:

Perhaps it’s useful to clarify that I have installed ROOT via conda-forge in this specific environment.

Thanks!
Riccardo

[eguiraud]

@riccardomanzoni

it’s useful to clarify that I have installed ROOT via conda-forge in this specific environment.

Oh uhm yes there are some known issues with the conda package and numba, because both ROOT and numba make use of llvm under the hood. ROOT::EnableImplicitMT just happens to trigger the loading of the ROOT interpreter which then interferes with numba, I think. @chrisburr does this ring any bell?

In the meanwhile using a C++ expression or a different ROOT installation method might work around the issue.

Cheers,
Enrico