Hello Adye,
Short reply:
I didn’t try your code since I’m more interested in the method you provided in the last reply. But I just read that it’s doing something similar.
I’ve also tried setattr for all the branches at the start of looping, sth like:
self.val = self.b_val[0]
And the time just increased 2~3 times for me (yesterday the number was from 15 secs to 38 secs), because copying does take time. but it’s still good that doing this will only copy it once, if we use entry.val, every time it’s called, I suspect there’s a copy happening.
Actually, I don’t know what is vanilla PyRoot. I guess you can use getitem directly, instead of [], but yeah, with more typing… or add a wrapper like def r(s.val, i=0): return s.val.__getitem__(i) to save some typing. Also, in your PySelectorBase.py, I can see you also used [] in MakeAttributeValue_CacheAddress? does that part also work under vanilla PyROOT?
===================================
The following text are written in a convoluted way because I edited and summarized it at different time, sorry.
More detailed test I did:
So what I’m doing is use GetBranch and GetEntry for all of them like you do, but mixed SetBranchAddress in Init().
In total I did four tests.
- slowest:
SetBranchStatus() for branches used
For entry in tree:
entry.val
- medium time consumption: is your method:
<GetBranch() for branches used>
For entry in xrange(tree.GetEntries()):
for br in branches:
br.GetEntry(entry)
# I suspect this line will cause time no matter SetBranchStatus / Get only the branches
# because it contains copying.
entry.val
- my method:
<The pieces of code to automatically setup array.array type>
For entry in xrange(tree.GetEntries()):
for br in branches:
br.GetEntry(entry)
# this is a bit inconvenient:
self.val[0]
# in case the val is a vector: (that is what one is expected to do..)
self.val[1]
- based on method 3, but after br.GetEntry(entry), I setattr for things inside branches, to avoid calling [0] later. That is 2-3 times slower than method 3.
Following are time comparison between method 2 and 3:
I’m running on 566346 events, reading over 7 branches, Module [apply_cut] and [fill_hist] are done once per event.
In [apply_cut] module, I’m reading 1 or 2 (let’s say 99% of the time it’s 2) of the variables.
In [fill_hist] module, I’m reading 5 of the variables.
In the loop, I’m also reading 2 variables. So the rest of the time are spent on looping.
Due to my time analysis, if I use your method, GetBranch, and get the value still by tree.val, the filling of histogram took ~50% of the time. The rest are mostly spent on looping(retrieving variables by writing to memory by setattr()). you can see follows:
######## start my time analysis for [HistogramProcessor]
Module [total] spent time: 95.9731440544, 100.00% wrt to total time
Module [register_hist] spent time: 0.547441959381, 0.57% wrt to total time
Module [apply_cut] spent time: 16.7197921276, 17.42% wrt to total time
Module [fill_hist] spent time: 48.0654530525, 50.08% wrt to total time
Module [write] spent time: 0.037024974823, 0.04% wrt to total time
######## finish time analysis for [HistogramProcessor]
I’m curious about this behavior since tree.x seems to call some wrapper and do copy again and again, even with some existing variable… so the more you fill the more time you spent. You can see that fill_hist is ~ 3 times the apply_cut, while 5 branches / 2 branches =2.5 ~ 3.
But if I use SetBranchAddress, I guess the variable are copied out only when GetBranch, so i guess that’s why it saves more time. Time analysis follows:
######## start my time analysis for [HistogramProcessor]
Module [total] spent time: 27.0103001595, 100.00% wrt to total time
Module [register_hist] spent time: 0.278868913651, 1.03% wrt to total time
Module [apply_cut] spent time: 1.43743085861, 5.32% wrt to total time
Module [fill_hist] spent time: 7.2391602993, 26.80% wrt to total time
Module [write] spent time: 0.0350120067596, 0.13% wrt to total time
######## finish time analysis for [HistogramProcessor]
And those two code uses the same histogram bookkeeping utility, just changed reading of branch.
).