Pyroot:: read tree AsNumpy

[adrian_sev]

Hi! So, i did not find any simpler method of reading a tree in numpy other than rdf.
So, i tried this:

        # Get numpy array
        rdf_tracks = ROOT.RDataFrame(tree_track)
        # we can do global filter here for tracks: sign, p_{T,x,y,z} before conversion to numpy
        npy_tracks = rdf_tracks.AsNumpy()

and i get this

./ao2d_process.py 
TypeError: function takes exactly 5 arguments (1 given)

The above exception was the direct cause of the following exception:

SystemError: <cppyy.CPPOverload object at 0x7f23e25703d0> returned a result with an exception set

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home.hdd/adrian/work/AO2Dproto/./ao2d_process.py", line 157, in <module>
    npy_tracks = rdf_tracks.AsNumpy()
  File "/home/physics-tools/ROOT/v6-26-00-patches/lib/ROOT/_pythonization/_rdataframe.py", line 87, in RDataFrameAsNumpy
    return result.GetValue()
  File "/home/physics-tools/ROOT/v6-26-00-patches/lib/ROOT/_pythonization/_rdataframe.py", line 139, in GetValue
    for i, x in enumerate(cpp_reference):
SystemError: <class 'enumerate'> returned a result with an exception set

is this anything known? would there be any problem if the tree have some friends attached?
Thank you!
Adrian

[adrian_sev]

well, the error went away after removing the friends.

[eguiraud]

Hi,
that error is super weird, it looks like enumerate is a class in that script…?

AsNumpy should have no problem with friends, e.g. I can run this correctly:

import ROOT

ROOT.RDataFrame(10).Define("x", "1").Snapshot("main", "f1.root")
ROOT.RDataFrame(10).Define("y", "2").Snapshot("friend", "f2.root")

main = ROOT.TChain("main")
main.Add("f1.root")

friend = ROOT.TChain("friend")
friend.Add("f2.root")

main.AddFriend(friend)

arr = ROOT.RDataFrame(main).AsNumpy()
print(arr)

# prints
{'x': ndarray([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int32), 'friend.y': ndarray([2, 2, 2, 2, 2, 2, 2, 2, 2, 2], dtype=int32)}

If this becomes a problem again, please post a minimal reproducer.
Cheers,
Enrico

[adrian_sev]

Hi! how can i have enumerate as a class ? the script is this
and the part of adding the friends is here
i believe that is the same thing… (unless i missed a detail somewhere)

also, i wanted to ask about the return object of AsNumpy : i see that the output is a dict with branches as ndarrays … is there a way to create an ndarray of ndarrays?
i need to do python - Is there any numpy group by function? - Stack Overflow (make a list of “groups” that have the same value on the “first” column/branch/ndarray ) and even if i get the indexes by processing the grouping column, i do not see how to process further these groups when they are in distinct keys

[eguiraud]

@adrian_sev

how can i have enumerate as a class ?

I have no idea, but from your error message:

SystemError: <class 'enumerate'> returned a result with an exception set

I can’t run the script you shared (I don’t have the input files nor the alice_DataModel script) so I can’t debug further.

@adrian_sev

is there a way to create an ndarray of ndarrays?

Not directly with AsNumpy, but you can post-process the dictionary of numpy arrays as needed.

Cheers,
Enrico

[adrian_sev]

Hi! Thanks a lot for taking a look! this is a test data file: https://asevcenc.web.cern.ch/asevcenc/ao2dproto/AO2D.root

That alice data model is not used at this moment and i even forgot that i did not commit it
We could continue over email if you want: adrian.sevcenco@cern.ch
Thank you!
Adrian

[eguiraud]

Hi @adrian_sev ,
Can you please post here a minimal, self-contained version of the code above that I can run to reproduce the stacktrace above? I’m not sure I understand how to reproduce your error.

Cheers,
Enrico

[adrian_sev]

Hi! So, you can found here both a reproducer script and a data file.
Thanks a lot for looking into this!
Adrian

[eguiraud]

Alright so here’s a minimal reproducer:

#!/usr/bin/env python3
import ROOT
df = ROOT.RDataFrame('DF_2471812916001/O2trackcov', 'AO2D.root')
df.AsNumpy(["fRhoSnpY"])

The problem is not with friend trees, somehow calling AsNumpy on that specific branch trips up PyROOT. Work in progress.

EDIT:

ok, it’s a bug in PyROOT with vectors of chars:

#!/usr/bin/env python3
import ROOT

vec = ROOT.std.vector['char'](['a','b','c'])
for _ in vec:
    pass

yields

TypeError: function takes exactly 5 arguments (1 given)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/blue/Scratchpad/work/forum-pyroot-error/reproducer.py", line 6, in <module>
    for _ in vec:
SystemError: <cppyy.CPPOverload object at 0x7fade90f4130> returned a result with an exception set

This is now Iteration on `std.vector['char']` is broken · Issue #9632 · root-project/root · GitHub .