In order to do event based parallelism in ATLAS, we have a mother
process that forks worker child processes to individual cores, each of
which then processes a number of events individually. In order that
these child processes not interfere with each others’ I/O, they each
write their outputs into individual directories. At the end of the
job, the outputs are merged. In order to maximize memory sharing, this
forking happens after the first event is processed, by which time many
output files have already been opened. So what we do is clone these
files into the individual worker directories, and reset file
descriptors to point to the appropriate places.
We need to do this in a way that’s transparent to the users, so that
if they hold a pointer to something in the original file, this pointer
still needs to be valid after the forking has migrated the files to
the new locations.
We have a service (the THistSvc in Gaudi) that manages all Histograms,
Trees and Graphs that are registered with it. It holds all the
pointers to the individual TTrees, THNs, and TGraphs. When the mother
process clones the original TFiles in the individual worker
directories, we update all these objects that are registered with the
service, so that the pointers point to the objects in the new files,
but the value of the pointers haven’t changed. This way, users can
still keep using a pointer that they got before the forking.
When dealing with TFiles that contain TTrees and THNs, this isn’t a
problem as after we have copied the file structure to the new
location, we can just do a SetDirectory() that reflects the new TFile.
However, there’s no such method in TGraphs.
What I’ve tried doing is cloning the TGraph in the new file (either
via the constructor or a Clone() method), then doing a memcpy to
overwrite the original pointer with the clone, so that the original
pointer will reference a TGraph in the new file, but this doesn’t
work.
Can anyone suggest a solution?
cheers, Charles.