# Add new column to RDataFrame

**URL:** <https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962>\
**Category:** ROOT\
**Tags:** rdataframe\
**Created:** [July 2, 2019, 1:29pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962 "2019-07-02T13:29:48Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![riccardomanzoni](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/riccardomanzoni/32/14004_2.png) [@riccardomanzoni](https://root-forum.cern.ch/u/riccardomanzoni)\
**Post date:** [July 2, 2019, 1:29pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/1 "2019-07-02T13:29:49Z")

</div>

Hi!

I have the following problem with RDF: I need to add a column to the RDF, and the content of the column cannot be easily computed from other columns already contained in the RDF.  
What is the neatest way to do this (granting that the dirty ways are not very satisfactory)?

To give some context: this new column consist of the prediction of some machine learning algorithm (that it’s good to keep detached and standalone), so the simple example

```auto
auto df_with_define = df.Define("newColumn", "x*x + y*y");

```

does not apply.

Thanks,  
Riccardo

---

<div class="post-metadata">

**Author:** ![eguiraud](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/eguiraud/32/32217_2.png) [@eguiraud](https://root-forum.cern.ch/u/eguiraud)\
**Post date:** [July 2, 2019, 2:06pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/2 "2019-07-02T14:06:26Z")

</div>

Hi,  
from your snippet it looks like you use C++. The easier way, then, is to use lambda captures:

```auto
double some_other_variable = 42;
auto df2 = df.Define("newColumn", [=] { return some_other_variable; });

```

or

```auto
auto ml_model = SomeMachineLearningTool(...);
auto df2 = df.Define("prediction",
                     [&ml_model] (double column_value) { return ml_model.predict(column_value); },
                     {"column_name"});

```

Cheers,  
Enrico

---

<div class="post-metadata">

**Author:** ![riccardomanzoni](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/riccardomanzoni/32/14004_2.png) [@riccardomanzoni](https://root-forum.cern.ch/u/riccardomanzoni)\
**Post date:** [July 2, 2019, 2:46pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/3 "2019-07-02T14:46:41Z")

</div>

Thanks Enrico.  
Actually I am using python, I just copy-pasted the example from the RDF docs.  
Riccardo

---

<div class="post-metadata">

**Author:** ![swunsch](https://root-forum.cern.ch/letter_avatar_proxy/v4/letter/s/50afbb/32.png) [@swunsch](https://root-forum.cern.ch/u/swunsch)\
**Post date:** [July 2, 2019, 3:30pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/4 "2019-07-02T15:30:55Z")

</div>

Hi!

Unfortunately it is non trivial to call Python code from inside C++. The simplest solution is to compute the ML values beforehand and attach them to the actual dataset as a friend tree.

Does this work for you?

Best  
Stefan

---

<div class="post-metadata">

**Author:** ![riccardomanzoni](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/riccardomanzoni/32/14004_2.png) [@riccardomanzoni](https://root-forum.cern.ch/u/riccardomanzoni)\
**Post date:** [July 3, 2019, 9:17am UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/5 "2019-07-03T09:17:15Z")

</div>

Hi Stefan,

resorting to friend trees is how we currently circumvent this limitation.  
This is what we deem as “not very satisfactory” 😉

From my very naive point of view of newcomer to data frames, I think that the possibility to easily stitch an additional column to an existing data frame - in python - is a critical feature.

Cheers,  
Riccardo

---

<div class="post-metadata">

**Author:** ![eguiraud](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/eguiraud/32/32217_2.png) [@eguiraud](https://root-forum.cern.ch/u/eguiraud)\
**Post date:** [July 3, 2019, 9:55am UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/6 "2019-07-03T09:55:55Z")

</div>

Hi Riccardo,  
it might be possible, but (currently) not so performant. We are working on better python integration.

See this post and the following: [Rdataframe define column of same constant value](https://root-forum.cern.ch/t/rdataframe-define-column-of-same-constant-value/34851/8) . You can use `TPython::Eval` inside a Define/Filter lambda to get a python value into a dataframe expression.

If the result of the model evaluation fits in memory, the most performant way is still to do a single evaluation of the whole dataset, and then index the result. Something like (haven’t tested it):

```auto
predictions = ml_model.predict(data)
df = df.Define("prediction", 'float(TPython::Eval("predictions[rdfentry_]"))')

```

---

<div class="post-metadata">

**Author:** ![riccardomanzoni](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/riccardomanzoni/32/14004_2.png) [@riccardomanzoni](https://root-forum.cern.ch/u/riccardomanzoni)\
**Post date:** [July 3, 2019, 12:51pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/7 "2019-07-03T12:51:33Z")

</div>

Hi Enrico,

here’s a minimal version of my attempt

```auto
import ROOT
from ROOT import TFile, TTree
from array import array
import numpy as np

f = TFile( 'test.root', 'recreate' )
t = TTree( 'tree', 'tree without histos' )

nevents = 1000

n = array( 'i', [0] )
d = array( 'f', [0.] )
t.Branch( 'mynum', n, 'mynum/I' )
t.Branch( 'myval', d, 'myval/F' )
 
for i in range(nevents):
    n[0] = i
    d[0] = np.random.normal()
    t.Fill()
 
f.Write()
f.Close()

predictions = np.random.normal(100, 1, nevents)

df = ROOT.RDataFrame('tree', 'test.root')

# https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/6
df = df.Define("prediction", 'float(TPython::Eval("predictions[rdfentry_]"))')

hh = df.Histo1D('prediction')
hh.Draw()

```

as far as I understand, it fails because `rdfentry_` in not accessible from within `Eval`.  
Then I thought I could just pass the whole numpy array to `TPython::Eval` (that is I removed `[rdfentry_]`)but it seems that `Define` can only handle single floats, not arrays / vectors of floats.

Thanks

---

<div class="post-metadata">

**Author:** ![eguiraud](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/eguiraud/32/32217_2.png) [@eguiraud](https://root-forum.cern.ch/u/eguiraud)\
**Post date:** [July 3, 2019, 12:59pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/8 "2019-07-03T12:59:43Z")

</div>

Hi,  
what’s the error you get with `rdfentry_`?

Also, `Define` should be able to handle any non-pointer, non-reference C++ type.

Cheers,  
Enrico

---

<div class="post-metadata">

**Author:** ![riccardomanzoni](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/riccardomanzoni/32/14004_2.png) [@riccardomanzoni](https://root-forum.cern.ch/u/riccardomanzoni)\
**Post date:** [July 3, 2019, 1:04pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/9 "2019-07-03T13:04:01Z")

</div>

here’s what I get with the example above

```auto
In [4]: hh.Draw()
   ...:
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<string> in <module>()

NameError: name 'rdfentry_' is not defined
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
TypeError: a float is required
---------------------------------------------------------------------------

```

wheras, if I change the relevant line to

```auto
df = df.Define("prediction", 'float(TPython::Eval("predictions"))')

```

I get

```auto
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
TypeError: only size-1 arrays can be converted to Python scalars
---------------------------------------------------------------------------

```

---

<div class="post-metadata">

**Author:** ![eguiraud](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/eguiraud/32/32217_2.png) [@eguiraud](https://root-forum.cern.ch/u/eguiraud)\
**Post date:** [July 3, 2019, 1:41pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/10 "2019-07-03T13:41:50Z")

</div>

Alright, thanks!

Regarding the first problem: as most untested code, it was wrong 😅 This works on ROOT v6.16:

```python
>>> import ROOT
>>> a = range(10)
>>> df = ROOT.RDataFrame(10)
>>> df = df.Define("x", 'auto to_eval = std::string("a[") + std::to_string(rdfentry_) + "]"; return float(TPython::Eval(to_eval.c_str()));')
>>> display = df.Display()
>>> display.Print()
x | 
0.00000f | 
1.00000f | 
2.00000f | 
3.00000f | 
4.00000f | 

```

(works, but might be quite slow depending on the usecase).

Regarding the second error: it’s because you are casting that `TPython::Eval` result (which is a collection) to a `float` – of course that can’t work. I don’t think `TPython::Eval` supports converting python collections to C++ collections, see docs for the returned object [TPyReturn](https://root.cern.ch/doc/master/classTPyReturn.html).

Cheers,  
Enrico

---

<div class="post-metadata">

**Author:** ![riccardomanzoni](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/riccardomanzoni/32/14004_2.png) [@riccardomanzoni](https://root-forum.cern.ch/u/riccardomanzoni)\
**Post date:** [July 3, 2019, 1:48pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/11 "2019-07-03T13:48:37Z")

</div>

I’ve just tried it out, it works perfectly!  
Now, the toy example is sorted out, I hope this is still fast enough for the real life case!

Thanks a lot,  
Riccardo

---

<div class="post-metadata">

**Author:** ![eguiraud](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/eguiraud/32/32217_2.png) [@eguiraud](https://root-forum.cern.ch/u/eguiraud)\
**Post date:** [July 3, 2019, 2:04pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/12 "2019-07-03T14:04:20Z")

</div>

Good luck!

Yes, it turns out that calling python from C++ from python is ugly. Friend trees are nicer, probably faster.  
@swunsch has a nicer solution for the new, experimental PyROOT (available if you build ROOT yourself) that might be closer to the elegance/performance sweet spot.

---

<div class="post-metadata">

**Author:** ![swunsch](https://root-forum.cern.ch/letter_avatar_proxy/v4/letter/s/50afbb/32.png) [@swunsch](https://root-forum.cern.ch/u/swunsch)\
**Post date:** [July 3, 2019, 3:12pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/13 "2019-07-03T15:12:59Z")

</div>

Hi!

Probably it does not help you, but let me show you what we already have for experimental PyROOT in 6.18 (and planned to be “standard” in 6.20):

```python
import ROOT

class AwesomeModel:
    def predict(self, x):
        return x[0] * x[1]

model = AwesomeModel()

@ROOT.DeclareCppCallable(["float"] * 2, "float")
def predictModel(var1, var2):
    return model.predict([var1, var2])

df = ROOT.ROOT.RDataFrame(10).Define("x", "CppCallable::predictModel(var1, var2)")
print(df.AsNumpy())

```

From the technical side: The problem is that this will be always not thread safe and also interferes with the global interpreter lock of Python in the multi-threaded case.

From the ML/algorithmic side: As soon as you use neural network, a batch inference will always be much much faster than event-by-event unless you filter your dataset massively beforehand. So the friend tree solution would be most suitable.

In case you’re just executing something “simple” in python, you can use the solution above at the speed of C++ since we use numba if possible to jit the thingy into compiled code 🙂

Best  
Stefan

Edit: Even though the code above is not suited for multi-threading, we have protected the calls with a lock!

---

<div class="post-metadata">

**Author:** ![system](https://root-forum.cern.ch/uploads/default/original/3X/e/f/efe4f24398fd8ffeb76bccb6ecc46d9d9c003376.png) [@system](https://root-forum.cern.ch/u/system)\
**Post date:** [July 17, 2019, 3:21pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/14 "2019-07-17T15:21:27Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![eguiraud](https://root-forum.cern.ch/user_avatar/root-forum.cern.ch/eguiraud/32/32217_2.png) [@eguiraud](https://root-forum.cern.ch/u/eguiraud)\
**Post date:** [January 8, 2021, 1:50pm UTC](https://root-forum.cern.ch/t/add-new-column-to-rdataframe/34962/15 "2021-01-08T13:50:08Z")

</div>

In case someone stumbles upon this post: `DeclareCppCallable` is now in production as `ROOT.Numba.Declare`, a tutorial is available [here](https://root.cern/doc/master/pyroot004__NumbaDeclare_8py.html).
