Book multiple keras methods with different models (with different input sizes)

TMVA
1

Currently trying to implement keras DNNs as an alternative MVA (besides BDTs) for our analysis.

For the BDTs we optimize both which variables we want to input as well as all of the hyperparameters of the BDTs and this works perfectly.

For the DNNs i was trying to do the same but have run into a problem. When i try to include DNNs with different input sizes it always errors during the training step for some DNN due to the input dimensions not matching, while test runs with just that DNN work perfectly fine.

For example if i book DNNs with 2,1,2 input variables the first DNN runs fine but trying to run the training of the second DNN gives an error. If my input sizes are 2,3 then it errors at the first.

For the first case the message looks like this:
Exception: Error when checking model input: expected dense_input_8 to have shape (None, 2) but got array with shape (665723, 1)

My assumption is that each loading of a model overwrites the last one and then it tries to use that for ALL of the methods, which obviously does not work. Is that just the behaviour that one can not compare different models within one run or is there something i am doing wrong?

Using ROOT 6.08/06

The “full” output of the booking + training is here:

INFO           MVAAnalysis.py:258  in main                Prepared Inputs
INFO              MVAUtils.py:305  in BookMethods         Nvars: 2
INFO              MVAUtils.py:325  in BookMethods         Saved model model_nVars_2_nLayers_3_nNodes_32_Droput_20.h5
INFO              MVAUtils.py:328  in BookMethods         TMVAFactory.BookMethod(<ROOT.TMVA::DataLoader object ("TMVA_Pol_DNNTest") at 0x807fce0>,25,DNN_First__AddDY_3N,!H:FilenameModel=model_nVars_2_nLayers_3_nNodes_32_Droput_20.h5:NumEpochs=2:BatchSize=128:SaveBestOnly=true:TriesEarlyStopping=5)
Factory                  : Booking method: DNN_First__AddDY_3N
                         :
                         : Load model from file: model_nVars_2_nLayers_3_nNodes_32_Droput_20.h5
INFO              MVAUtils.py:305  in BookMethods         Nvars: 1
INFO              MVAUtils.py:325  in BookMethods         Saved model model_nVars_1_nLayers_3_nNodes_32_Droput_20.h5
INFO              MVAUtils.py:328  in BookMethods         TMVAFactory.BookMethod(<ROOT.TMVA::DataLoader object ("TMVA_Pol_DNNTest") at 0x61bcf60>,25,DNN_First__Nominal,!H:FilenameModel=model_nVars_1_nLayers_3_nNodes_32_Droput_20.h5:NumEpochs=2:BatchSize=128:SaveBestOnly=true:TriesEarlyStopping=5)
Factory                  : Booking method: DNN_First__Nominal
                         :
                         : Load model from file: model_nVars_1_nLayers_3_nNodes_32_Droput_20.h5
INFO              MVAUtils.py:305  in BookMethods         Nvars: 2
INFO              MVAUtils.py:325  in BookMethods         Saved model model_nVars_2_nLayers_3_nNodes_32_Droput_20.h5
INFO              MVAUtils.py:328  in BookMethods         TMVAFactory.BookMethod(<ROOT.TMVA::DataLoader object ("TMVA_Pol_DNNTest") at 0x7b05d50>,25,DNN_First__AddCosThetaLepW,!H:FilenameModel=model_nVars_2_nLayers_3_nNodes_32_Droput_20.h5:NumEpochs=2:BatchSize=128:SaveBestOnly=true:TriesEarlyStopping=5)
Factory                  : Booking method: DNN_First__AddCosThetaLepW
                         :
                         : Load model from file: model_nVars_2_nLayers_3_nNodes_32_Droput_20.h5
INFO           MVAAnalysis.py:262  in main                After book methods
Factory                  : Train all methods
DataSetFactory           : [TMVA_Pol_DNNTest] : Number of events in input trees
                         :
                         :
                         : Dataset[TMVA_Pol_DNNTest] : No weight renormalisation applied: use original global and event weights
DataSetInfo              : Correlation matrix (Background):
                         : ------------------------------------------------------------
                         :                      min(abs(DY_3N),3.2) min(abs(DY_3Z),3.2)
                         : min(abs(DY_3N),3.2):              +1.000              +0.809
                         : min(abs(DY_3Z),3.2):              +0.809              +1.000
                         : ------------------------------------------------------------
DataSetInfo              : Correlation matrix (Signal):
                         : ------------------------------------------------------------
                         :                      min(abs(DY_3N),3.2) min(abs(DY_3Z),3.2)
                         : min(abs(DY_3N),3.2):              +1.000              +0.744
                         : min(abs(DY_3Z),3.2):              +0.744              +1.000
                         : ------------------------------------------------------------
DataSetFactory           : [TMVA_Pol_DNNTest] :
                         :
Factory                  : [TMVA_Pol_DNNTest] : Create Transformation "I" with events from all classes.
                         :
                         : Transformation, Variable selection :
                         : Input : variable 'DY_3N' <---> Output : variable 'DY_3N'
                         : Input : variable 'DY_3Z' <---> Output : variable 'DY_3Z'
TFHandler_Factory        : Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :    DY_3N:     1.2243    0.84580   [ 3.5486e-06     3.2000 ]
                         :    DY_3Z:     1.1763    0.79172   [ 3.7230e-06     3.2000 ]
                         : -----------------------------------------------------------
                         : Ranking input variables (method unspecific)...
IdTransformation         : Ranking result (top variable is best ranked)
                         : ------------------------------
                         : Rank : Variable  : Separation
                         : ------------------------------
                         :    1 : DY_3Z     : 9.953e-02
                         :    2 : DY_3N     : 5.635e-02
                         : ------------------------------
Factory                  : Train method: DNN_First__AddDY_3N for Classification
                         :
                         : Option SaveBestOnly: Only model weights with smallest validation loss will be stored
                         : Option TriesEarlyStopping: Training will stop after 5 number of epochs with no improvement of validation loss
Train on 665723 samples, validate on 666622 samples
Epoch 1/2
665723/665723 [==============================] - 16s - loss: 0.0022 - acc: 0.8103 - val_loss: 0.0021 - val_acc: 0.8115om inf to 0.00213, saving model to TMVA_Pol_DNNTest/weights/TrainedModel_DNN_First__AddDY_3N.h5
Epoch 2/2
665723/665723 [==============================] - 16s - loss: 0.0021 - acc: 0.8106 - val_loss: 0.0021 - val_acc: 0.8115om 0.00213 to 0.00212, saving model to TMVA_Pol_DNNTest/weights/TrainedModel_DNN_First__AddDY_3N.h5
                         : Elapsed time for training with 665723 events: 34.9 sec
                         : Creating xml weight file: TMVA_Pol_DNNTest/weights/TMVA_Pol_DNNTest_DNN_First__AddDY_3N.weights.xml
                         : Creating standalone class: TMVA_Pol_DNNTest/weights/TMVA_Pol_DNNTest_DNN_First__AddDY_3N.class.C
Factory                  : Training finished
                         :
DataSetFactory           : [TMVA_Pol_DNNTest] : Number of events in input trees
                         :
                         :
                         : Dataset[TMVA_Pol_DNNTest] : No weight renormalisation applied: use original global and event weights
DataSetInfo              : Correlation matrix (Background):
                         : ----------------------------------------
                         :                      min(abs(DY_3Z),3.2)
                         : min(abs(DY_3Z),3.2):              +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (Signal):
                         : ----------------------------------------
                         :                      min(abs(DY_3Z),3.2)
                         : min(abs(DY_3Z),3.2):              +1.000
                         : ----------------------------------------
DataSetFactory           : [TMVA_Pol_DNNTest] :
                         :
Factory                  : Train method: DNN_First__Nominal for Classification
                         :
                         : Option SaveBestOnly: Only model weights with smallest validation loss will be stored
                         : Option TriesEarlyStopping: Training will stop after 5 number of epochs with no improvement of validation loss
<WARNING>                : Failed to run python code: history = model.fit(trainX, trainY, sample_weight=trainWeights, batch_size=batchSize, nb_epoch=numEpochs, verbose=verbose, validation_data=(valX, valY, valWeights), callbacks=callbacks)
<WARNING>                : Python error message:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_88/x86_64-slc6-gcc49-opt/lib/python2.7/site-packages/Keras-1.1.0-py2.7.egg/keras/models.py", line 620, in fit
    sample_weight=sample_weight)
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_88/x86_64-slc6-gcc49-opt/lib/python2.7/site-packages/Keras-1.1.0-py2.7.egg/keras/engine/training.py", line 1034, in fit
    batch_size=batch_size)
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_88/x86_64-slc6-gcc49-opt/lib/python2.7/site-packages/Keras-1.1.0-py2.7.egg/keras/engine/training.py", line 961, in _standardize_user_data
    exception_prefix='model input')
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_88/x86_64-slc6-gcc49-opt/lib/python2.7/site-packages/Keras-1.1.0-py2.7.egg/keras/engine/training.py", line 108, in standardize_input_data
    str(array.shape))
Exception: Error when checking model input: expected dense_input_6 to have shape (None, 2) but got array with shape (665723, 1)
2

Hi,
I am not sure I have fully understood the problem. Are you trying to book using the factory several DNN methods based on Keras, but using as input different number of variables.
This, I don’t think works in TMVA, given a data loader, you define the number of variable to use, by calling DataLoader::AddVariable.
What you can easily do is create for each different Keras method a different DataLoader. This should work fine.
If not, or if I have mis-understood something, please let me know

Cheers

Lorenzo

3

Thanks for the reply Lorenzo,

i am booking different DNNs with the factory the exact same way i already do it when i am using BDTs. For that we use different data loaders for each method.

Which you can see here where i just print the command that books the methods:

TMVAFactory.BookMethod(<ROOT.TMVA::DataLoader object ("TMVA_Pol_DNNTest") at 0x807fce0>,25,DNN_First__AddDY_3N,!H:FilenameModel=model_nVars_2_nLayers_3_nNodes_32_Droput_20.h5:NumEpochs=2:BatchSize=128:SaveBestOnly=true:TriesEarlyStopping=5)
TMVAFactory.BookMethod(<ROOT.TMVA::DataLoader object ("TMVA_Pol_DNNTest") at 0x61bcf60>,25,DNN_First__Nominal,!H:FilenameModel=model_nVars_1_nLayers_3_nNodes_32_Droput_20.h5:NumEpochs=2:BatchSize=128:SaveBestOnly=true:TriesEarlyStopping=5)
TMVAFactory.BookMethod(<ROOT.TMVA::DataLoader object ("TMVA_Pol_DNNTest") at 0x7b05d50>,25,DNN_First__AddCosThetaLepW,!H:FilenameModel=model_nVars_2_nLayers_3_nNodes_32_Droput_20.h5:NumEpochs=2:BatchSize=128:SaveBestOnly=true:TriesEarlyStopping=5)

You can see that the data loader for each method is at a different address. As this already works for the BDTs i do not think that the problem lies there.

It is specifically with the DNNs where the problem arises. Where it seems (not sure though) that i uses the same model for all the methods despite me having generated and specified a different model for each method that can work withthe number of inputs that should come from the dataloader.

4

Hi,
If you have different DataLoader, then it should work. If you have a reproducible, it will be easier to debug, otherwise I can easily do one.
Cheers

Lorenzo

5

No, sorry, i do not have a short reproducible piece.

6

So i checked and it is 100% different data loaders. And for each method the dimensions that are given to the model according to the error message are exactly those that i expect. It’s just the model that is used does not match that expectation.

7

Thank you for the information!
I will then try to reproduce your problem and let you know!
Best

Lorenzo

8

Hi,

do you have an update on this? Is it something i did wrong or is it due to TMVA or might have already been fixed in a newer version?

Cheers
Jan-Eric

9

Hi,
Apologies for the delay. I still need to create a reproducible example. I will try to do it today,
Lorenzo

10

Hi,
I have tested to run two Keras models with different inputs using two dataloaders. It works fine, it is possible the problem you are having is caused by a very old ROOT version (6.08). Can you try to use the latest ROOT, 6.26 ?

Best regards

Lorenzo

11

Updated to 6.24 now and it seems that this issue has disappeared.

Now i just have to fix all the errors that come from setting up the environment in a completely different why and switching to python3 as well as all the other things that got changed.

But i think i have solved most of those now as well.

Will come back if i encounter another problem that i can not solve.