Applying cut to a Tree (large dataset)

[sala]

Hi experts,
I need some help on loading data from a tree.
My goal is to load data from a tree applying a selection “BDT_response>0”.
I used to correctly load my OriginalTree and then do

TTree * tree = OriginalTree->CopyTree("BDT_response>0");

But now I have a very large dataset, and when I try to do this I get the error

Fatal in <TBufferFile::AutoExpand>: Request to expand to a negative size, 
likely due to an integer overflow: 0x800329ee for a max of 0x7ffffffe.
aborting

I think the error is due to the size of the dataset. Is there any way to avoid this?

I’ve already tried this:

TFile* file = TFile::Open("TEST.root");
TTree* originalTree = (TTree*)file->Get("mytree");
gROOT->cd(); 
TEntryList *entry = new TEntryList("BDT"); 
originalTree->Draw(">>BDT","BDT_response>0","goff");

like suggested from @pcanal in a previous thread [CopyTree (with a selection), July 2006] , but I get the error

Info in <TSelectorDraw::AbortProcess>: An object of type 'TEntryList' has the same name as the requested event list (entry)

In this case, the code works but it just does not apply the selection and works on the full tree.

I hope I was clear in explaining what I need.
Thanks a lot!

Andrea

[pcanal]

You meant

@sala

originalTree->Draw(">>BDT","BDT_response>0","goff entrylist");

[sala]

Hi @pcanal,
thanks for your answer!
The tip you gave me removed the error, but my fit is still not working.
When I load a RooDataSet right after loading the tree, do I have to do

RooDataSet data("data","data",RooArgSet(x),Import(*tree));

or maybe something else? Because with this line it seems to load the whole tree and not just the selected data. What am I missing? Do I have to pass the entry list in some way?

Thanks and have a good weekend,
Andrea

[Diyon335]

hey @sala ,
I am not sure whether your step of creating an EntryList and etc is the same as I am about to suggest, but maybe it might work for you:

1. create a temporary root file
2. go to its directory
3. do what you need to do with cuts and etc
4. in the end, delete the file

I work with pyROOT but in principle should be the same:

file = ROOT.TFile.Open("temp.root","RECREATE")
file.cd()

#apply cuts here
#draw a histogram maybe? idk, whatever you want

file.Close()
del file
os.system("rm -rf temp.root")

hope it helps!

Diyon

[sala]

Hey @Diyon335,
thanks for answering!
Actually, I’m not quite sure if I fully understood your point.
If I build another TFile, that means I have to build a temporary TTree on that file, and in order to load data on that new tree I would have to copy from an existing tree, and so we’re back to square one…

Please tell me if I’m missing something about your answer.
P.S. What I Have to do after creating the selected tree is load a RooDataSet and perform a RooFit.

Cheers,
Andrea

[Diyon335]

Hey @sala ,
Ah yes I forgot the most important thing which was to explain my point. Believe it or not, I am doing the exact same thing as you as I am typing this haha

But anyway, you do not have to do anything with this temporary root file at all. By asking the script to change directory to this file’s directory, you are preventing the flooding of memory. This is because copying a tree without giving it a space to do this floods the memory. Don’t write any tree to the new file. Simply make it, change directory to where it is, apply the selection with CopyTree and do your fitting and etc and just delete the file

Another thing you could do, is also use the other parameters of tree#CopyTree(). As seen here https://root.cern.ch/doc/master/classTTree.html#a2b57854ba133da5fc523931aa0427e21
You could implement a for loop and iterate in steps of, say 1000 events. like:

1. tree.CopyTree(“BDT_response > 0”, option = “”, nentries = 1000, firstentry = 0)
2. tree.CopyTree(“BDT_response > 0”, option = “”, nentries = 2000, firstentry = 1001) and so on

then maybe fill a completely new tree and do things with that?

Does it make sense?

[sala]

Oh yes @Diyon335 now it does make sense! I will try tomorrow morning and get back to you.
Thanks!
A.

[sala]

In the end I made two separate scripts, in the first one I skim the Tree and in the second one I do the fit… and it works! Thanks @Diyon335 for giving me the idea.
Cheers
Andrea