Putting SOFIE code into a library

I’ve been experimenting with putting the c++ code generated by the TMVA_SOFIE_KERAS tutorial into a library so that it can be linked with other codes. I was able to do that by making 2 modifications:

  • separating the Session constructor and ‘infer’ function declarations from their implementations
  • explicitly compiling SOFIE_common.cxx and putting it into the library as well
    This raises 2 questions
  • why does the SOFIE code inline all the functions? Is there a way to have it emit separate .h and .cxx output?
  • why isn’t SOFIE_common.cxx part of libTMVA.so? is there a root build option that will put it there?
    thanks, Dave

I guess @moneta can help you.

Hi Dave,

The idea is to have everything in a .h file that can be included in the code performing the inference, so no extra dependency is needed apart from BLAS.
For this reasons the functions used for inference are included in SOFIE_common.h. You can of course create a separate library if you prefer where you include the needed file.
SOFIE_common.cxx should be instead used for generating the code by SOFIE, if this is not the case is maybe an issue we need to fix. It is part of the libROOTTMVASofie.so library, and we did on purpose have it different than TMVA in case one day we want to release as an independent library.

If you have other questions or feedback, please let us know. Apologies for the delay in answering,

Best,

Lorenzo

Hi Lorenzo,
Thanks for your reply. Adding libROOTTMVASofie.so to my build resolved the missing dependencies, so I don’t need SOFIE_common.cxx anymore.

I’m now trying to use SOFIE to create the inference function from a very simple Keras model (4 Dense layers) I created and trained in a Jupyter notebook, saved using model.save(“mymodel.h5”).
Processing that using SOFIE::PyKeras::Parse(“mymodel.h5”) I get the following error:

Error in TRint::HandleTermInput(): std::runtime_error caught: None error: Models not initialized with batch-size are not yet supported in TMVA SOFIE

batch_size in Keras is related to the training gradient-descent, I tried setting that explicitly but it had no effect. I’m guessing that term means something different in this context. Do you have a suggestion
what I might try?

thanks, Dave

Hi Dave,

I think we have the fix for this but only in ROOT Master, where in case the batch size is not defined in the model we set it by default to 1. Otherwise one can set a batch size by passing the value in RModel::Generate. Unfortunately this is available only in ROOT master. For the time being you would need to probably save the correct batch size in the .h5 model. I think this can be done using the Input object, see Input object.
I can try to make an example for you if you want.

However, we don’t support the batch size as a parameter of the generated code. Is this something would you like to have it ?

Cheers

Lorenzo

Hi Lorenzo,
I was able to get around this problem by explicitly specifying the batch_size in my model Input stage, as opposed to just the fit function. Generate then produces
functional code.

The next problem I ran into is that the inference function then uses that batch_size; ie if I train with batch_size=32, Infer expects input
data from 32 samplings, and produces an output vector of size 32. This doesn’t fit my use case at all, and I wonder, are there use cases where this would
be desirable? What benefit does it provide? It seems an unnatural coupling between training optimization and inference.

I got around this problem by creating 2 models: a dummy model with batch_size=1, and one with the batch_size=32, which I actually train. Then, I copy
the weights from the trained model to the dummy before saving it. Processing the dummy h5 file with SOFIE creates an inference that operates on a single sample of inputs.
I see also that, at the head of root, there is a function RModel::Initialize that allows overriding the batch_size; will that allow me create a single sample inference

function from my batch_size=32 trained model if I call that before I call Generate? That would be a more elegant solution.

One last thing I noticed: If I run the SOFIE inference generation in root 6.26/6, the output code and data statement list the layers in a random order (not what’s in the h5 file).
But if I run in 6.26/10 the h5 order is respected. Was that part of a bug fix between /6 and /10?

Thanks again for your help. Just FYI, now that it’s proven to work, we plan to replace our ANN inference functions in the Mu2e experiment reconstruction code with SOFIE-
generated codes.

Dave

Hi Dave,

Thanks a lot on the feedback on SOFIE and your interest in using it for the Mu2e experiment.
For the batch size issue, we have now in the master the possibility to provide the batch size when generating the model code. This is possible when the model does not contain a fixed batch size, i.e. it is not specified in the input shape when defining the model, but it is provided only when training it.
This will be part of the new release 6.28 happening in about one month. If you would like to have this functionality in 6.26, I could maybe backport it for the next patch version of 6.26.

Concerning the fix of the layers order, I don’t think we have a fix for this in 6.26/10.
Are you sure are you using also the same Keras/Tensorflow version in your case ?
I could investigate this issue further if you could send me the h5 file.

Best regards

Lorenzo