Is there a way to control (limit) the number of threads used by the DNN CPU training ?
My use case is I am running trainings on a local batch system, which consists of nodes with 24 cores (48 w/ hyperthreading) and need to be able to control TMVA to not attempt to use 48 threads on these machines, as the batch system is (in principle) only giving me one core per job slot.
I tried turning threading off completely with the ‘Multithreading=False’ option, but that did not seem to have any effect. I would also like to not turn off threading completely, but limit it to a number < the number of cores (say 4).
I just heard back from Simon and indeed BLAS is using multithreading so in part we need to specify how many threads BLAS is using. However there is also an internal part of the DNN that uses a thread pool that for now is fixed size (hardware dependent).
One could fix this by recompiling root and changing this value. (If you’d like I can provide further instructions here).
The seems like a very reasonable feature request, so we’ll be looking into a solution. Thanks for letting us know!
After some helpful discussion with @xvallspl I think it is should be possible to put EnableImplicitMT( num_threads ); at the beginning of e.g. the TMVAClassification.C script. Using this together with the environment variable to set the number of threads that blas is using should allow you to limit the number of threads spawned.
Note however that the recommended usage of EnableImplicitMT is without an argument to let the system determine the optimal number of threads. So if this works for you it would be preferable.
One small comment though is would it not make sense for the call EnableImplicitMT to also set the bias ENV VAR ? As long as the call is made early enough in the script/application it should still be early enough to take effect, and it would be more user friendly than having to set the same thing twice in two different ways.