File closing running python multiprocess

Dear experts,

I’m not sure if this is the right forum for this question, so in case this should go to ROOT support please let me know.
Here my problem: Lately I get quite a lot of errors while the program shuts down and writes/merges root files created in some job which is executed via python multiprocess and subprocess modules. The amount of error messages increased significantly after switching to root v5.34.08 (used 5.34.00 before). In principle I have two use cases:

  1. run two synchronised jobs, i.e. run job A first which writes out a root file (if this helps I’m running hist2workspace) and after this job is done a second jobs start (once there is an open slot in the queue) reading in this file, the later complains from time to time about not closed files and tries to recover the keys. This somehow seems to depend on the no. of pool processes, which makes no sense to me.

  2. run one job on many cores and use TFileMerger to merge the output files. At least this worked fine in 5.34.00, but this version must not be used for my purpose (limit setting).

Does anyone has an idea what could go wrong or has an idea how this can be avoided? The code seems to run after the recovery, however I don’t know how reliable the results are.

Many thanks in advance!

Marcus

Marcus,

first off, python multiprocessing and C++ don’t mix. This is b/c the multiprocessing module makes many assumptions that are simply no longer true after code outside the control of the python interpreter is loaded. So, the best would be to defer “import ROOT” until the worker has started (i.e. after the fork). This is even more true for opening of files.

That said, if you want more detailed feedback, I’d have to see the code or at least the error messages that you get …

Cheers,
Wim