Thanks for the suggestions - they’re quite useful. I hadn’t considered TTree friends - I’ll look into this to see how it’ll work for us.
Can you have an in memory only TTree, or a ‘virtual’ file with a tree? If so, is it allowed link more than one TTree to the same local class objects (e.g. my event class) for writing?
With TTree::Draw you can choose the start and end index in the tree for the loop (nentries & firstentry), so I had envisioned drawing just one event.
I look at energy deposition throughout an accelerator and magnets and detector backgrounds. For a given event we make an energy deposition histogram (anywhere from 10 to 250k bins!) following the axis of an accelerator. The model is done in Geant4 and for a given event we can have say 10^4 to 10^7 energy deposition hits that are filled into the histogram, so this is a massive reduction. We also typically need to simulate 10^4 to 10^7 events for the result to converge.
Later, we want to investigate the origin of losses and perform some analysis - e.g. what’s the energy deposition caused by this bit of initial phase space in the beam. So, we can simply combine the histograms while performing a cut. We don’t use a “profile” so we can perform cuts, the variance and errors are calculated correctly and are also calculated in a numerically stable way. We also need to combine means from multiple runs with the correct variance.
We may also do cuts based on the integral of the histograms etc. Storing all the original energy deposition hits is impractical in data volume and highly inefficient to histogram each time we do an analysis.
One upcoming example is using a TH2Poly with say just a few bins in a calculated phase space set of variables that makes it very easy to classify events for later analysis.
We don’t have one model - our same application (BDSIM) is used to model many accelerators and beam lines so the specific analysis really varies from user to user, which is why I don’t want to hard code any loops etc for any specific variables. I want to be able to flexibly define a histogram (e.g. line for TTree::Draw) that I can make at simulation time and fill into the events to reduce data and reduce the weight of later analysis.
For a lot of our simulations we usually run close to memory limits (avg GB / cpu) so I’m trying to avoid having anything accumulate in memory per event.