TTreeReader vs RDataFrame vs branches to read TTree

Should I always use TTreeReader/RDataFrame to read a tree or branches are good too?

In the documentation for TTreeReader it is called

A simple, robust and fast interface to read values from ROOT columnar datasets such as TTree, TChain or TNtuple.

Most examples that I opened now on the forum by users and ROOT developers use also TTreeReader.

However, when I open documentation for TTree, there is only one brief mention of TTreeReader in the description of MakeSelector. All examples use branches.

RDataFrame offers the third possibility.

I can formulate these questions:

  1. Is any of these methods deprecated or not supported (or will be that in the future)?
  2. Are branches the only way to fill a TTree? (I’m asking because reading with branches would be more symmetric in this case)
  3. Are there any known performance differences? I read in TTreeReader’s code that it uses some Proxy Branches; does it mean that it is just a higher level interface to using branches? Is TTreeReader just for some convenience (as I understand, that and RDataFrame are very scalable for other data sets; but what if we speak only about a tree)?
  4. Are any of these ways buggier / less tested? Why is TTreeReader called robust?
  5. Is there any difference between them from Python’s point of view?

Thanks a lot for all answers.

I found a comment by @eguiraud in 2019

I don’t suggest using TTree directly to read ROOT data, it’s a low-level API, gives very few (type) safety guarantees, performs very few sanity checks and it’s also trickier to get good performance out of it.

For me a low-level API is usually performant; but maybe Enrico could explain.

I tried to use TTreeReaderValue from Python (failed yet), and found this comment by @wlav from 2014

[TTreeReader] Recommended from C++, sure. From Python, no. TTreeCache works in both cases, so no worries there.

Is it still the case?

In the manual there is an example with branches, though it ends with TTreeReader. It doesn’t answer the questions in this topic.

I’m sure @eguiraud will help you with this

It seems I can’t use TTreeReader from Python, because I can’t get the value of a TTreeReaderValue (it uses the operator ‘*’ in C++, and I don’t know how translate that to Python; I couldn’t find the answer).

ROOT.TTreeReaderValue['double']
age = ROOT.TTreeReaderValue('double')(reader, "Age")
staff_list = []
for entry in itertools.islice(tree, 10):
    # this adds just objects, need numbers
    staff_list.append(age)

I also found a suggestion by @etejedor in 2019:

…it also reads the whole branch. If you wanted to benefit from the optimization, you would need to do (from Python) SetBranchAddress + GetEntry. Alternatively, you could also use RDataFrame to efficiently read tree data from Python.

Hi @ynikitenko ,
I’ll try to reply to all questions in order but in case I miss some please point them out again:

No

RDataFrame::Snapshot can also be used to write TTrees out.

Under the hood, all interfaces use TTree and TBranch. Raw-TTree-used-naiively can be slow, but if you know what you are doing using TTree directly gives the best (single-thread) performance. The most performant way to use TTrees is to call TBranch::GetEntry on each branch you want to read for each entry rather than calling TTree::GetEntry, and possibly only call TBranch::GetEntry lazily, if strictly needed for a given event, to avoid deserializing a branch value when it’s not needed. TTree::SetBranchAddress is not type-safe, so you can end up reading garbage if you are not careful. TTreeReader helps with that. TTreeReader also helps with only deserializing what you really need, but in many usecases it turns out it’s not as fast as using TTree “smartly” directly.
RDataFrame has the same advantages and disadvantages as TTreeReader, but it also makes it very simple to parallelize the event loop, which is not that simple when using raw `TTree.

I don’t think there is a lot of difference in test coverage. RDataFrame probably has the best direct coverage, but indirectly RDataFrame tests also test TTreeReader and TTree. I think TTreeReader is called more robust in the docs because of the extra type-safe checks, better error handling and an API that is harder to use incorrectly w.r.t. TTree.

Nothing major comes to mind, but of course your mileage may vary. RDataFrame usage from Python often still requires writing C++ helper functions (for now, a more Pythonic RDF API is on my wish list).

See above, you have to know what you are doing to get good performance out of raw TTree, but it’s what you want if you need to spill out the latest (single-thread) % of ROOT I/O performance.

I don’t know why Wim said that TTreeReader was not recommended in Python (in 2014), but I think you can use it just fine. You should be able to use TTreeReaderValue::Get instead of operator*.

Cheers,
Enrico

Thanks a lot, Enrico!
This perfectly answers the question.

However, it seems I can’t use the method Get (and can’t find the answer on the internet).
Here is my more complete code

reader = ROOT.TTreeReader(staff_tree)
ROOT.TTreeReaderValue['int']
age = ROOT.TTreeReaderValue('int')(reader, "staff.Age")
for entry in itertools.islice(reader, 10):
    print(age.Get())

prints

<cppyy.LowLevelView object at 0x7f3fddd313b0>

10 times.

When I try to cast it to int, it gives

*** Break *** segmentation violation

Cheers,
Yaroslav

I can’t get the value of a TTreeReaderValue (it uses the operator ‘*’ in C++

Enrico already gave you an alternative, but in python you can invoke it as __deref__().

When I try to cast it to int, it gives

The return value of Get is T*, so the value is treated as an integer buffer in Python (LowLevelView cppyy type).

What happens if you do age.Get()[0] or age.__deref__()?

1 Like

The method Get produces

print(age.Get())

> …
<cppyy.LowLevelView object at 0x7f5d51f6e500>

When I use

age.Get()[0]

it prints correct ages as numbers! A strange thing is that they are shifted by one (the iteration starts from the 1st entry, not the 0th).

age.__deref__()

works too, but the results are shifted as well.

I iterate as I wrote earlier,

reader = ROOT.TTreeReader(staff_tree)
ROOT.TTreeReaderValue[‘int’]
age = ROOT.TTreeReaderValue(‘int’)(reader, “staff.Age”)
staff_list = []
for entry in itertools.islice(reader, 10):

(islice could be omitted). It works differently when I iterate for entry in staff_tree.

In order to exclude Python/PyROOT shenanigans: are the numbers what you expect if you do:

reader = ROOT.TTreeReader(staff_tree)
ROOT.TTreeReaderValue['int']
age = ROOT.TTreeReaderValue(‘int’)(reader, "staff.Age")
reader.Next()
print(age.Get()[0])
reader.Next()
print(age.Get()[0])

?

And from C++?

Cheers,
Enrico

when I launch this code, it works well.

Here is the whole code:

# via entry’s attributes
f = TFile(“staff.root”)
staff_tree = f.Get(“T”)
# correct
for staff_m in itertools.islice(staff_tree, 10):
___print(staff_m.Age, ", ", sep=’’, end=’’)
print()

# also correct
reader = ROOT.TTreeReader(staff_tree)
ROOT.TTreeReaderValue[‘int’]
age = ROOT.TTreeReaderValue(‘int’)(reader, “staff.Age”)
reader.Next()
print(age.Get()[0])
reader.Next()
print(age.Get()[0])

You can try it yourself, it should be much faster. The file staff.root is generated by an example macro $ROOTSYS/tutorials/pyroot/staff.py . I say that numbers are ‘correct’, because they are the same as TTree::Scan. It looks like a bug in TTreeReader in Python. Could you please fix that? (create an issue or just make a PR). Thank you. Sorry that couldn’t reply quickly, I was a bit busy now.

Hi @ynikitenko ,
sorry, I am not sure I understand the latest issue.

Does this mean the above code is slower than expected (in which case, what are you comparing it with?) or that the new version of the code should be faster than the old?

The numbers are correct: what looks like a bug? That the iteration with itertools.islice skips one element? If so, can you please check whether simple iteration (for _ in reader: ...) is also broken?

Cheers,
Enrico

Hi @eguiraud ,
“it should be much faster” - I mean that you can copy my code and check anything you want, because the circle “suggestion at Cern → check in Moscow → new suggestion at Cern” is definitely longer than if we skip the Moscow part.
Yes, exactly, one element is skipped. This is the problem and a bug (unless it is stated in documentation, which is not the case).
No, islice just takes first 10 elements (I don’t want to output them all). It can not effect the first element taken.
Hope that suits you fine. Thank you very much!

Ah, I see, well it’s not necessarily faster if you take into account my own latency (due to other posts, other bugs to fix, other features to implement, etc.), so thanks for your help :smiley:

This is definitely a bug, now reported as Python iteration over a TTreeReader is broken · Issue #8183 · root-project/root · GitHub . Our PyROOT experts will take a look as soon as possible. In the meanwhile you can use while reader.Next() instead of the for loop to get the correct behavior (or use RDataFrame instead, which has the performance advantage of pushing the loop to C++).

Cheers,
Enrico

Dear Enrico,

many thanks for the bug report! I would not be able to make such a succinct example of that bug in any reasonable time.

As for me - I use tree iteration and will switch to branch iteration as you suggested, so I’m not affected by that bug. Anyway I’m glad to know how to use other methods, and it’s great that there will be one bug less (hopefully) soon.
As for this thread, it looks like all is answered, so it can be closed.

Cheers,
Yaroslav

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.