Filter data frame using external flags

[rooter_03]

Hi,

From:

Adding data from an external container to a DataFrame

Hi @rooter_03 , I would like to provide a generic RDF solution for “adding the contents of a numpy array as an additional RDF column” but I am having some conceptual problems making it work in case the array is added after a filter is applied and the event loop is multi-threaded. Without that case solved I don’t think we can add this in upstream ROOT. A review of the different cases follows, let me know if I’m missing something: RDF with no filters In this case something like my first solutio…

One of the uses of adding a container to a dataframe would be to add a variable that can be used to remove entries:

df = add_column(df, arr_val, name)
df = df.Filter('var > 0.3')

where var values are arr_val. However, there are many caveats as seen in the post above. Would it be more feasible to have a function that filters based on an external container?

arr_flg = numexpr.evaluate('var>0.3', {'var' : arr_val})
df = df.Filter(arr_flg)

i.e. the filtering would take an array of bools instead of a string.

Cheers.

---

Please read tips for efficient and successful posting and posting code

ROOT Version: Not Provided
Platform: Not Provided
Compiler: Not Provided

---

[Axel]

@rooter_03

add a variable that can be used to remove entries

@rooter_03

to have a function that filters based on an external container

Can you give some more details of which limitation you’re running into? Maybe by writing down the if statement in a pseudo for loop?

My guess is that you expect one entry of arr_val to filter one entry of the input (“event”) - is that correct? In that case the most stable and performant solution is to create a friend tree that contains the flag for each entry of the input tree. You have to make sure that the order is the same, of course: when writing the friend tree you’ll need to run this without multithreading, once. Then you can just add that branch from the auxiliary friend tree that contains your filter, and filter on its branch - also in multithreading.

[eguiraud]

Hi,
I think the conceptual issues described in the post you link are still present with a Filter(arr_flg) method in the scenarios in which upstream filters or Range calls are present.

If the arr_flg array describes a selection of entries over the whole dataset (no upstream filters/ranges) then transforming that array into a TEntryList and attaching it to the TTree/TChain is a viable workaround. Or indeed you can use a friend tree as Axel suggests.

Also see my errata at Adding data from an external container to a DataFrame - #15 by eguiraud

Cheers,
Enrico