Using TTreeFormula in Proof without massive memory leaks

ROOT
1

Hi,

I have written a TSelector based off the one shown here: root.cern.ch/root/html532/src/T … w.cxx.html

It basically has a bunch of TTreeFormula that I want to loop over and evaluate for each entry of a tree.

I’ve found that when I run on PROOF, my TSelector starts using up lots of memory very quickly, compared to running without PROOF where it doesn’t use much at all.

In the above selector, the following logic takes place:

Init() : Compile the variables - i.e. create the TTreeFormula
Notify() : call UpdateFormulaLeaves on the manager

When I run on PROOF, I seem to see the following behaviour:

Notify() Gets called first at the start of each file
Init() Gets called next at the start of each file

When not running on PROOF, the Init method only gets called once, at the start of the job

I added some code to my TSelector to monitor memory usage. I saw a small increase in usage between the Notify and the Init method:

14:43:59 1625181 Wrk-0.0 | Info in TProofMultiDraw::Notify: virtual memory used = 398080
14:44:03 1625181 Wrk-0.0 | Info in TProofMultiDraw::Init: virtual memory used = 399624

But then I get a big increase in memory during the re-call to CompileVariables:

14:44:04 1625181 Wrk-0.0 | Info in TProofMultiDraw::Init: virtual memory used after recompile = 413744

This was after I already updated the code so that each time CompileVariables is called, it first tries to delete all existing TTreeFormula (i.e. it calls the ClearFormula method).
The problem seems to be that the creation of lots of TTreeFormula leads to memory leaks.

While resolving the leaking in TTreeFormula creation would be the best solution, for now it would suffice if I could use the existing TTreeFormula and get them to use the new tree. I tried using the ‘SetTree’ method to change the tree in all the formula but that lead to a crash.

So, in summary, what is the correct way to update/delete TTreeFormula, specifically in the context of a PROOF selector?

Thanks
Will

2

Dear Will,

Things to be done once per job should normally go in SlaveBegin. Or I would add a flag and do them only once in Init.
Can you provide a simple example reproducing the problem to debug?

G Ganis

3

Hi,

Well I could like to do the TTreeFormula creation once per worker, but the problem is that I dont have a recipe for ‘updating’ the formula when the TTree changes. I tried doing ‘SetTree’ on the formula but that didnt seem to work and lead to a crash when the first event of the next tree was processed.

I actually discovered the leak seems to be caused by some code I had for trying to check if the formula in the TTreeFormula was valid. I found and reported a bug in the recommended way (GetNdim()), which has now been fixed but I have to wait for our software to get updated to a fixed ROOT release, so I was doing:

TTreeFormula* f = new TTreeFormula("f","some*formula",tree);
if(f->Compile("some*formula")) {
	//is a bad formula
	exit();
}
delete f;
f = new TTreeFormula("f","some*formula*",tree);

It seems that using the Compile method and then deleting the formula like I was doing above was leaking memory. I can live without this safety net for now until I get the GetNdim() fix.

But it would still be good to know if there’s a way to update a TTreeFormula to use a new tree instead of having to delete the formula and recreate it with the new tree at the start of each file.

Cheers
Will

4

Hi Will,

You can likely use

    treeformula->SetTree(newtree);
    treeformula->UpdateFormulaLeaves();

Cheers,
Philippe.

5

I tried that too but it crashed still on the first event of the next file :frowning:

6

Are you able to use a TChain instead of multiple TTree? (I.e. this works for TChains … I am not sure why it does not work when done by hand …)

7

PROOF seems to provide me with TTrees rather than a TChain.

8

right … I have forgotten :frowning: that you were inside PROOT per see …

So back to SetTree/UpdateFormulaLeaves … can you reproduce the problem outside of Proof?

9

ok I’ll see what I can do and get back to you.

10

Hi…
@will_cern
I am trying to do some similar thing… In my case treeformula ->EvalInstance() is not working inside Process of tselector on PROOF.
https://root-forum.cern.ch/t/ttreeformula-not-working-in-proof/34658?u=chinmay
Did you have same problem ? Can you share relevant part of selector ?