At the moment, we have an unpacker which can convert raw recorded data (served over TCP in real real time in detector specific formats which only the unpacker knows how to handle) into a TTree written to a root file.
To do a near line analysis, I could have a separate pyroot script which takes the file with the tree and creates some histograms.
However, this would mean that every time I wanted to look at the latest data, I would need to first run the unpacker, write a new .root file for a bit and then run the pyroot script on that newly written file.
It would be more elegant if I could have the unpacker continuously serialize ttree entries to some pipe as they come in, and the pyroot process process them as they are read from the pipe, so I can see the the data online.
Our unpacker also provides other output formats besides root files with a ttree, which support pipe communications, but parsing them from python would likely take some work (i.e. a custom C module). By comparison, reading in a TTree from a root file in pyroot is rather effortless.
The reason why I would favor pipes as a means of transmitting the data is that it seems to be the simple and extensible (e.g. if I ever need to put the pyroot process on a different computer, I can route the pipe communication through socat trivially). Another option would be to have the TTree in shared memory, but then I would need to use some semaphore to indicate if the tree is currently being written by the unpacker or being read by the python script, and yield execution for every event.
I would also not want to merge the unpacking step with the histogramming script – it would run counter to unix philosophy (do one thing), not reflect our organizational structure (Conway’s law) and require compromises on the correct languages for the respective jobs.
If there is some halfway simply way to drag the ttree through the pipe (e.g. “for every event, loop over the baskets, serialize them, then on the receiving side, update the basket contents with this trick”), I would try it, if it is more complex (“write a custom class inheriting from TFile”), then I would rather try to parse the other format from python.
While searching a bit more, I found TWebFile, but from my understanding this is used to fetch a pre-created file from a web server, which is not what I want.