I have a code which runs in one minute with PROOF-lite (16 workers). I have to run it 50 times on the same files (each time the same code but with a different option).
What’s the best way to do it? So far I tried
- running sequentially 50 PROOF-lite sessions; it’s the safest but takes 50 minutes
- sending 50 LSF jobs, each one running PROOF-lite; it’s not that safe, as I get some jobs failing, and I’m not sure neither how to tell PROOF not to use all the 64 available cores on the LSF nodes, nor what will happen with PROOF if two batch jobs share the same CPU
One example of what I would like to do is: set (with PoD or elsewhere, the simpler the better) N workers, and share them among 50 different PROOF-lite sessions (i.e. completely separate! I can’t share my TSelector since I have 50 different configurations).
Goal: avoid having to wait 50 minutes.