factory.TrainAllMethods() crashes with TypeError when using PyKeras

I’m trying to run a custom pykeras model similar to the tutorial here: https://github.com/lmoneta/tmva-tutorial/blob/master/notebooks/TMVA_CNN_Classification_py.ipynb

When the training is called on the methods

factory.TrainAllMethods()

After the completion of all epochs, I hit this error:

Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
reshape_2 (Reshape)          (None, 1, 32, 32)         0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 6, 28, 28)         156       
_________________________________________________________________
activation_6 (Activation)    (None, 6, 28, 28)         0         
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 6, 14, 14)         0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 16, 10, 10)        2416      
_________________________________________________________________
activation_7 (Activation)    (None, 16, 10, 10)        0         
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 16, 5, 5)          0         
_________________________________________________________________
flatten_2 (Flatten)          (None, 400)               0         
_________________________________________________________________
dense_4 (Dense)              (None, 120)               48120     
_________________________________________________________________
activation_8 (Activation)    (None, 120)               0         
_________________________________________________________________
dense_5 (Dense)              (None, 84)                10164     
_________________________________________________________________
activation_9 (Activation)    (None, 84)                0         
_________________________________________________________________
dense_6 (Dense)              (None, 2)                 170       
_________________________________________________________________
activation_10 (Activation)   (None, 2)                 0         
=================================================================
Total params: 61,026
Trainable params: 61,026
Non-trainable params: 0
_________________________________________________________________
Train on 12800 samples, validate on 3200 samples
Epoch 1/10
12800/12800 [==============================] - 3s 250us/step - loss: 0.1691 - accuracy: 0.9316 - val_loss: 0.1962 - val_accuracy: 0.9169

Epoch 00001: val_loss improved from inf to 0.19617, saving model to trained_model_cnn.h5
Epoch 2/10
12800/12800 [==============================] - 3s 231us/step - loss: 0.2114 - accuracy: 0.9129 - val_loss: 0.1964 - val_accuracy: 0.9181

Epoch 00002: val_loss did not improve from 0.19617
Epoch 3/10
12800/12800 [==============================] - 3s 242us/step - loss: 0.1637 - accuracy: 0.9341 - val_loss: 0.1147 - val_accuracy: 0.9581

Epoch 00003: val_loss improved from 0.19617 to 0.11468, saving model to trained_model_cnn.h5
Epoch 4/10
12800/12800 [==============================] - 3s 239us/step - loss: 0.1260 - accuracy: 0.9522 - val_loss: 0.2123 - val_accuracy: 0.9141

Epoch 00004: val_loss did not improve from 0.11468
Epoch 5/10
12800/12800 [==============================] - 3s 243us/step - loss: 0.1569 - accuracy: 0.9366 - val_loss: 0.1028 - val_accuracy: 0.9622

Epoch 00005: val_loss improved from 0.11468 to 0.10278, saving model to trained_model_cnn.h5
Epoch 6/10
12800/12800 [==============================] - 3s 245us/step - loss: 0.1121 - accuracy: 0.9590 - val_loss: 0.0921 - val_accuracy: 0.9659

Epoch 00006: val_loss improved from 0.10278 to 0.09208, saving model to trained_model_cnn.h5
Epoch 7/10
12800/12800 [==============================] - 3s 256us/step - loss: 0.1077 - accuracy: 0.9581 - val_loss: 0.0944 - val_accuracy: 0.9638

Epoch 00007: val_loss did not improve from 0.09208
Epoch 8/10
12800/12800 [==============================] - 3s 266us/step - loss: 0.1572 - accuracy: 0.9387 - val_loss: 0.1088 - val_accuracy: 0.9566

Epoch 00008: val_loss did not improve from 0.09208
Epoch 9/10
12800/12800 [==============================] - 3s 264us/step - loss: 0.0757 - accuracy: 0.9710 - val_loss: 0.1003 - val_accuracy: 0.9622

Epoch 00009: val_loss did not improve from 0.09208
Epoch 10/10
12800/12800 [==============================] - 3s 268us/step - loss: 0.0815 - accuracy: 0.9691 - val_loss: 0.0712 - val_accuracy: 0.9747

Epoch 00010: val_loss improved from 0.09208 to 0.07119, saving model to trained_model_cnn.h5
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<string> in <module>

TypeError: 'dict_keys' object is not subscriptable

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-9-4bb1800441f7> in <module>
----> 1 factory.TrainAllMethods()

Exception: void TMVA::Factory::TrainAllMethods() =>
    FATAL error (C++ exception of type runtime_error)

Factory                  : Train all methods
Factory                  : Train method: PyKeras for Classification
                         : 
                         : Failed to run python code
                         : ================================================================
                         : H e l p   f o r   M V A   m e t h o d   [ PyKeras ] :
                         : 
                         : Keras is a high-level API for the Theano and Tensorflow packages.
                         : This method wraps the training and predictions steps of the Keras
                         : Python package for TMVA, so that dataloading, preprocessing and
                         : evaluation can be done within the TMVA system. To use this Keras
                         : interface, you have to generate a model with Keras first. Then,
                         : this model can be loaded and trained in TMVA.
                         : 
                         : 
                         : <Suppress this message by specifying "!H" in the booking option>
                         : ================================================================
                         : 
                         : Split TMVA training data in 12800 training events and 3200 validation events
                         : Training Model Summary
                         : Option SaveBestOnly: Only model weights with smallest validation loss will be stored
Getting his:0
<WARNING>                : Failed to run python code: copy_string=history.history.keys()[0]
<WARNING>                : Python error message:
<FATAL>                         : Failed to run python code
***> abort program execution

How can I resolve this issue?
Thanks :slight_smile:

This is most probably a question for @etejedor

I’ll ask for the help of @moneta or @kialbert here

Hi Anirudh,

Which ROOT version of ROOT are you using ? I think this issue is coming in Python3 and I remember to have it fixed with this commit

It might be needed to use the ROOT master version

Lorenzo

1 Like

Hi Lorenzo,

I guessed something like this, but couldn’t pin point the fix. My current version is 6.20/02.

Thanks a lot! Made my day, was stuck with this issue for the whole day yesterday :slight_smile:

Will build again from the master version to get to 6.21

Hi Anirudth

Thank you for reporting this issue. I will commit that fix also in 6.20 patches

Cheers

Lorenzo

You’re welcome. I hope I can make many more contributions to ROOT & TMVA in the future.

Cheers,
Anirudh