TMVA cannot interpret some valid RDataFrame expressions as input variables

TMVA
1

Hi,

I am training a BDT using TMVA, and I noticed that some variables defined as RDataFrame expressions cannot be interpreted by TMVA, even though they work correctly within the RDataFrame workflow.

For example, I originally defined an alias as:

aliases['LowestQGLIdx'] = {
    'expr': 'Take(Nonzero(CleanJet_qgl >= 0), Argsort(CleanJet_qgl[CleanJet_qgl >= 0]))'
}

Since I suspected Take() might be the issue, I rewrote it as:

aliases['LowestQGLIdx'] = {
    'expr': 'Nonzero(CleanJet_qgl >= 0)[Argsort(CleanJet_qgl[CleanJet_qgl >= 0])]'
}

However, TMVA still fails with the following error:

Error in TTreeFormula::Compile: Bad numerical expression: "LowestQGLIdx"
Expression LowestQGLIdx could not be resolved to a valid formula.
***> abort program execution

I encountered a similar issue with another variable, but in that case I was able to resolve it by rewriting the expression using TTreeFormula syntax.

Originally:

aliases['bVeto'] = {
    'expr': 'Sum(CleanJet_pt > 20. && abs(CleanJet_eta) < 2.5 && Take(Jet_btag{}, CleanJet_jetIdx) > {}) == 0'.format(bAlgo, bWP)
}

Rewritten as:

aliases['bVeto'] = {
    'expr': 'Sum$(CleanJet_pt > 20. && abs(CleanJet_eta) < 2.5 && (Jet_btag{}[CleanJet_jetIdx] > {})) == 0'.format(bAlgo, bWP)
}

This version works with TMVA.

My understanding is that TMVA evaluates variables using TTreeFormula, which supports a more limited expression syntax than RDataFrame. Is that the reason why expressions involving Take(), Nonzero(), Argsort(), or vector indexing fail? If so, is there a recommended way to define such variables so they can be used as TMVA input variables?

Any suggestions would be appreciated.

2

Hi,

Your understanding is exactly right. TMVA does not use RDataFrame to evaluate your input variables. When you register a variable/target/spectator/cut expression, TMVA hands the expression string to a TTreeFormula and evaluates it per event.

TTreeFormula is the old TTree draw/selection engine. It knows branch names, the standard C math functions, and its own special $ constructs and indexing semantics that you see in the docs. It has no knowledge of RVec and therefore none of Take(), Nonzero(), Argsort(), or RVec-style boolean-mask indexing (vec[vec >= 0]). Those live entirely in the RDataFrame/ROOT::VecOps world, which is a completely separate code path.

Recommended approach: since you’re already building these columns in RDataFrame, don’t ask TMVA to re-parse them, but materialize them and feed TMVA a plain branch. Define the columns in RDF and Snapshot to a tree/file. Something like:

df = ROOT.RDataFrame("Events", "input.root")
df = df.Define("LowestQGLIdx",
               "Take(Nonzero(CleanJet_qgl >= 0), Argsort(CleanJet_qgl[CleanJet_qgl >= 0]))")
df = df.Define("bVeto", "...")   # full RDF/RVec syntax, no restrictions
df.Snapshot("Events", "for_tmva.root")

Then point TMVA at for_tmva.root and register the variable by its plain column name ("LowestQGLIdx"), not the expression. Now TTreeFormula only has to read a branch, which it can always do, and you get to use the full RVec syntax you’re used to.

One caveat regardless of route: a TMVA input variable must resolve to one scalar per event. Take(Nonzero(...), Argsort(...)) returns a vector, so even after snapshotting, TMVA won’t accept it as a single BDT input. You’ll want to reduce it to a scalar (e.g. the first element / the index you actually want) in the Define.

One more thing worth considering: if you’re computing your features in RDataFrame anyway, it’s worth asking whether the TTreeFormula+TMVA usage is buying you anything at all. The moment you move the feature computation into RDF Defines (which the snapshot approach above already does), you can train the BDT directly with XGBoost, which usually trains faster and gives better models than TMVA, and is also more robust to missing values and all these good things:

import numpy as np
from xgboost import XGBClassifier

# Compute features with the full RDF/RVec syntax, pull them out as arrays
data = ROOT.RDataFrame("Events", "input.root") \
           .Define("LowestQGLIdx", "...") \
           .Define("bVeto", "...") \
           .AsNumpy(["LowestQGLIdx", "bVeto", ...])

x = np.vstack([data[v] for v in variables]).T
bdt = XGBClassifier(max_depth=3, n_estimators=500).fit(x, y, sample_weight=w)

And you don’t lose ROOT-native C++ inference by doing this: TMVA can serialize an XGBoost model and evaluate it with its fast tree-inference engine, TMVA::Experimental::RBDT:

# after fitting
ROOT.TMVA.Experimental.SaveXGBoost(bdt, "myBDT", "bdt.root", num_inputs=x.shape[1])

// C++ / RDataFrame application
TMVA::Experimental::RBDT bdt("myBDT", "bdt.root");
auto df2 = df.Define("y", TMVA::Experimental::Compute<N, float>(bdt), {"var1", "var2", ...});

See the tutorials tmva101_Training.py (XGBoost training + SaveXGBoost) and tmva103_Application.C (RBDT event-by-event, batch, and RDataFrame inference). Roughly the same effort you’re now spending contorting expressions into TTreeFormula syntax gets you onto a path where that whole class of problem disappears. The caveat is that RBDT is still experimental, so the capability and interface might change.

I hope that helps, and let us know which solution you’re going for in the end! Than in itself is also valuable feedback.

Cheers,
Jonas