/cms/multilepton-3/olena/signalFull$ python Spectator_debug_multiClass.py 
Using TensorFlow backend.
<class 'ROOT.TTree'>
DataSetInfo              : [output/spectator/dataset_pymva] : Added class "signal1"
                         : Add Tree TreeS of type signal1 with 10000 events
<class 'ROOT.TTree'>
DataSetInfo              : [output/spectator/dataset_pymva] : Added class "signal2"
                         : Add Tree TreeS of type signal2 with 10000 events
<class 'ROOT.TTree'>
DataSetInfo              : [output/spectator/dataset_pymva] : Added class "background"
                         : Add Tree TreeB of type background with 10000 events
                         : Dataset[output/spectator/dataset_pymva] : Class index : 0  name : signal1
                         : Dataset[output/spectator/dataset_pymva] : Class index : 1  name : signal2
                         : Dataset[output/spectator/dataset_pymva] : Class index : 2  name : background
Spectator_debug_multiClass.py:68: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(64, kernel_initializer="glorot_normal", activation="relu", input_dim=4)`
  model.add(Dense(64, activation='relu', init='glorot_normal', input_dim=4))
Spectator_debug_multiClass.py:69: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(32, activation="relu", kernel_initializer="glorot_normal")`
  model.add(Dense(32, init='glorot_normal', activation='relu'))
Spectator_debug_multiClass.py:70: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(3, activation="softmax", kernel_initializer="glorot_uniform")`
  model.add(Dense(3, init='glorot_uniform', activation='softmax'))
2020-11-10 12:50:13.220759: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 64)                320       
_________________________________________________________________
dense_2 (Dense)              (None, 32)                2080      
_________________________________________________________________
dense_3 (Dense)              (None, 3)                 99        
=================================================================
Total params: 2,499
Trainable params: 2,499
Non-trainable params: 0
_________________________________________________________________
Factory                  : Booking method: PyKerasCat
                         : 
                         : Adding sub-classifier: PyKeras::Category_PyKeras_1
DataSetInfo              : [Category_PyKeras_1_dsi] : Added class "signal1"
DataSetInfo              : [Category_PyKeras_1_dsi] : Added class "signal2"
DataSetInfo              : [Category_PyKeras_1_dsi] : Added class "background"
Category_PyKeras_1       : [Category_PyKeras_1_dsi] : Create Transformation "N" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
                         : Load model from file: output/spectator/dataset_pymva/weights/spectator_test_model.h5
                         : Adding sub-classifier: PyKeras::Category_PyKeras_2
DataSetInfo              : [Category_PyKeras_2_dsi] : Added class "signal1"
DataSetInfo              : [Category_PyKeras_2_dsi] : Added class "signal2"
DataSetInfo              : [Category_PyKeras_2_dsi] : Added class "background"
Category_PyKeras_2       : [Category_PyKeras_2_dsi] : Create Transformation "N" with events from all classes.
                         : 
                         : Transformation, Variable selection : 
                         : Input : variable 'var1' <---> Output : variable 'var1'
                         : Input : variable 'var2' <---> Output : variable 'var2'
                         : Input : variable 'var3' <---> Output : variable 'var3'
                         : Input : variable 'var4' <---> Output : variable 'var4'
                         : Load model from file: output/spectator/dataset_pymva/weights/spectator_test_model.h5
Factory                  : Train all methods
DataSetFactory           : [output/spectator/dataset_pymva] : Number of events in input trees
                         : 
                         : 
                         : 
                         : Number of training and testing events
                         : ---------------------------------------------------------------------------
                         : signal1    -- training events            : 5000
                         : signal1    -- testing events             : 5000
                         : signal1    -- training and testing events: 10000
                         : signal2    -- training events            : 5000
                         : signal2    -- testing events             : 5000
                         : signal2    -- training and testing events: 10000
                         : background -- training events            : 5000
                         : background -- testing events             : 5000
                         : background -- training and testing events: 10000
                         : 
DataSetInfo              : Correlation matrix (signal1):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.368  +0.378  +0.391
                         :    var2:  +0.368  +1.000  +0.388  +0.386
                         :    var3:  +0.378  +0.388  +1.000  +0.389
                         :    var4:  +0.391  +0.386  +0.389  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (signal2):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.388  +0.392  +0.394
                         :    var2:  +0.388  +1.000  +0.377  +0.399
                         :    var3:  +0.392  +0.377  +1.000  +0.388
                         :    var4:  +0.394  +0.399  +0.388  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (background):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.364  +0.370  +0.381
                         :    var2:  +0.364  +1.000  +0.371  +0.383
                         :    var3:  +0.370  +0.371  +1.000  +0.372
                         :    var4:  +0.381  +0.383  +0.372  +1.000
                         : ----------------------------------------
DataSetFactory           : [output/spectator/dataset_pymva] :  
                         : 
Factory                  : Train method: PyKerasCat for Multiclass classification
                         : 
                         : Train all sub-classifiers for Classification ...
DataSetFactory           : [Category_PyKeras_1_dsi] : Number of events in input trees
                         : Dataset[Category_PyKeras_1_dsi] :     signal1    requirement: "abs(eta)<=1.3"
                         : Dataset[Category_PyKeras_1_dsi] :     signal1         -- number of events passed: 5123   / sum of weights: 5123 
                         : Dataset[Category_PyKeras_1_dsi] :     signal1         -- efficiency             : 0.5123
                         : Dataset[Category_PyKeras_1_dsi] :     signal2    requirement: "abs(eta)<=1.3"
                         : Dataset[Category_PyKeras_1_dsi] :     signal2         -- number of events passed: 5123   / sum of weights: 5123 
                         : Dataset[Category_PyKeras_1_dsi] :     signal2         -- efficiency             : 0.5123
                         : Dataset[Category_PyKeras_1_dsi] :     background requirement: "abs(eta)<=1.3"
                         : Dataset[Category_PyKeras_1_dsi] :     background      -- number of events passed: 5134   / sum of weights: 5134 
                         : Dataset[Category_PyKeras_1_dsi] :     background      -- efficiency             : 0.5134
                         : Dataset[Category_PyKeras_1_dsi] :  you have opted for interpreting the requested number of training/testing events
                         :  to be the number of events AFTER your preselection cuts
                         : 
                         : Dataset[Category_PyKeras_1_dsi] :  you have opted for interpreting the requested number of training/testing events
                         :  to be the number of events AFTER your preselection cuts
                         : 
                         : Dataset[Category_PyKeras_1_dsi] :  you have opted for interpreting the requested number of training/testing events
                         :  to be the number of events AFTER your preselection cuts
                         : 
                         : Number of training and testing events
                         : ---------------------------------------------------------------------------
                         : signal1    -- training events            : 2561
                         : signal1    -- testing events             : 2561
                         : signal1    -- training and testing events: 5122
                         : Dataset[Category_PyKeras_1_dsi] : signal1    -- due to the preselection a scaling factor has been applied to the numbers of requested events: 0.5123
                         : signal2    -- training events            : 2561
                         : signal2    -- testing events             : 2561
                         : signal2    -- training and testing events: 5122
                         : Dataset[Category_PyKeras_1_dsi] : signal2    -- due to the preselection a scaling factor has been applied to the numbers of requested events: 0.5123
                         : background -- training events            : 2567
                         : background -- testing events             : 2567
                         : background -- training and testing events: 5134
                         : Dataset[Category_PyKeras_1_dsi] : background -- due to the preselection a scaling factor has been applied to the numbers of requested events: 0.5134
                         : 
DataSetInfo              : Correlation matrix (signal1):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  -0.017  +0.004  +0.001
                         :    var2:  -0.017  +1.000  -0.019  -0.003
                         :    var3:  +0.004  -0.019  +1.000  -0.012
                         :    var4:  +0.001  -0.003  -0.012  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (signal2):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  +0.011  +0.004  +0.001
                         :    var2:  +0.011  +1.000  -0.005  +0.026
                         :    var3:  +0.004  -0.005  +1.000  -0.011
                         :    var4:  +0.001  +0.026  -0.011  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (background):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  -0.014  -0.037  +0.004
                         :    var2:  -0.014  +1.000  -0.017  +0.002
                         :    var3:  -0.037  -0.017  +1.000  -0.033
                         :    var4:  +0.004  +0.002  -0.033  +1.000
                         : ----------------------------------------
DataSetFactory           : [Category_PyKeras_1_dsi] :  
                         : 
                         : Train method: Category_PyKeras_1 for Classification
                         : 
                         : ================================================================
                         : H e l p   f o r   M V A   m e t h o d   [ Category_PyKeras_1 ] :
                         : 
                         : Keras is a high-level API for the Theano and Tensorflow packages.
                         : This method wraps the training and predictions steps of the Keras
                         : Python package for TMVA, so that dataloading, preprocessing and
                         : evaluation can be done within the TMVA system. To use this Keras
                         : interface, you have to generate a model with Keras first. Then,
                         : this model can be loaded and trained in TMVA.
                         : 
                         : 
                         : <Suppress this message by specifying "!H" in the booking option>
                         : ================================================================
                         : 
TFHandler_Category_PyK...: Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:   0.054587    0.28302   [    -1.0000     1.0000 ]
                         :     var2:  -0.021796    0.28407   [    -1.0000     1.0000 ]
                         :     var3:  -0.036253    0.27460   [    -1.0000     1.0000 ]
                         :     var4:  -0.010989    0.28279   [    -1.0000     1.0000 ]
                         : -----------------------------------------------------------
TFHandler_Category_PyK...: Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:   0.054587    0.28302   [    -1.0000     1.0000 ]
                         :     var2:  -0.021796    0.28407   [    -1.0000     1.0000 ]
                         :     var3:  -0.036253    0.27460   [    -1.0000     1.0000 ]
                         :     var4:  -0.010989    0.28279   [    -1.0000     1.0000 ]
                         : -----------------------------------------------------------
                         : Split TMVA training data in 6152 training events and 1537 validation events
                         : Option SaveBestOnly: Only model weights with smallest validation loss will be stored
Train on 6152 samples, validate on 1537 samples
Epoch 1/5
6152/6152 [==============================] - 1s 94us/step - loss: 0.8630 - acc: 0.5237 - val_loss: 0.8185 - val_acc: 0.5452

Epoch 00001: val_loss improved from inf to 0.81852, saving model to output/spectator/dataset_pymva/weights/spectator_test_model1.h5
Epoch 2/5
6152/6152 [==============================] - 0s 66us/step - loss: 0.8166 - acc: 0.5454 - val_loss: 0.8174 - val_acc: 0.5537

Epoch 00002: val_loss improved from 0.81852 to 0.81742, saving model to output/spectator/dataset_pymva/weights/spectator_test_model1.h5
Epoch 3/5
6152/6152 [==============================] - 0s 64us/step - loss: 0.8158 - acc: 0.5506 - val_loss: 0.8156 - val_acc: 0.5530

Epoch 00003: val_loss improved from 0.81742 to 0.81560, saving model to output/spectator/dataset_pymva/weights/spectator_test_model1.h5
Epoch 4/5
6152/6152 [==============================] - 0s 68us/step - loss: 0.8144 - acc: 0.5478 - val_loss: 0.8162 - val_acc: 0.5556

Epoch 00004: val_loss did not improve from 0.81560
Epoch 5/5
6152/6152 [==============================] - 0s 68us/step - loss: 0.8129 - acc: 0.5551 - val_loss: 0.8148 - val_acc: 0.5524

Epoch 00005: val_loss improved from 0.81560 to 0.81482, saving model to output/spectator/dataset_pymva/weights/spectator_test_model1.h5
                         : Elapsed time for training with 7689 events: 2.71 sec         
                         : Dataset[Category_PyKeras_1_dsi] : Create results for training
                         : Dataset[Category_PyKeras_1_dsi] : Multiclass evaluation of Category_PyKeras_1 on training sample
                         : Dataset[Category_PyKeras_1_dsi] : Elapsed time for evaluation of 7689 events: 2.2 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Training finished
DataSetFactory           : [Category_PyKeras_2_dsi] : Number of events in input trees
                         : Dataset[Category_PyKeras_2_dsi] :     signal1    requirement: "abs(eta)>1.3"
                         : Dataset[Category_PyKeras_2_dsi] :     signal1         -- number of events passed: 4877   / sum of weights: 4877 
                         : Dataset[Category_PyKeras_2_dsi] :     signal1         -- efficiency             : 0.4877
                         : Dataset[Category_PyKeras_2_dsi] :     signal2    requirement: "abs(eta)>1.3"
                         : Dataset[Category_PyKeras_2_dsi] :     signal2         -- number of events passed: 4877   / sum of weights: 4877 
                         : Dataset[Category_PyKeras_2_dsi] :     signal2         -- efficiency             : 0.4877
                         : Dataset[Category_PyKeras_2_dsi] :     background requirement: "abs(eta)>1.3"
                         : Dataset[Category_PyKeras_2_dsi] :     background      -- number of events passed: 4866   / sum of weights: 4866 
                         : Dataset[Category_PyKeras_2_dsi] :     background      -- efficiency             : 0.4866
                         : Dataset[Category_PyKeras_2_dsi] :  you have opted for interpreting the requested number of training/testing events
                         :  to be the number of events AFTER your preselection cuts
                         : 
                         : Dataset[Category_PyKeras_2_dsi] :  you have opted for interpreting the requested number of training/testing events
                         :  to be the number of events AFTER your preselection cuts
                         : 
                         : Dataset[Category_PyKeras_2_dsi] :  you have opted for interpreting the requested number of training/testing events
                         :  to be the number of events AFTER your preselection cuts
                         : 
                         : Number of training and testing events
                         : ---------------------------------------------------------------------------
                         : signal1    -- training events            : 2438
                         : signal1    -- testing events             : 2438
                         : signal1    -- training and testing events: 4876
                         : Dataset[Category_PyKeras_2_dsi] : signal1    -- due to the preselection a scaling factor has been applied to the numbers of requested events: 0.4877
                         : signal2    -- training events            : 2438
                         : signal2    -- testing events             : 2438
                         : signal2    -- training and testing events: 4876
                         : Dataset[Category_PyKeras_2_dsi] : signal2    -- due to the preselection a scaling factor has been applied to the numbers of requested events: 0.4877
                         : background -- training events            : 2433
                         : background -- testing events             : 2433
                         : background -- training and testing events: 4866
                         : Dataset[Category_PyKeras_2_dsi] : background -- due to the preselection a scaling factor has been applied to the numbers of requested events: 0.4866
                         : 
DataSetInfo              : Correlation matrix (signal1):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  -0.005  +0.002  -0.039
                         :    var2:  -0.005  +1.000  +0.011  -0.004
                         :    var3:  +0.002  +0.011  +1.000  -0.021
                         :    var4:  -0.039  -0.004  -0.021  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (signal2):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  -0.012  -0.015  +0.000
                         :    var2:  -0.012  +1.000  +0.019  +0.004
                         :    var3:  -0.015  +0.019  +1.000  -0.024
                         :    var4:  +0.000  +0.004  -0.024  +1.000
                         : ----------------------------------------
DataSetInfo              : Correlation matrix (background):
                         : ----------------------------------------
                         :             var1    var2    var3    var4
                         :    var1:  +1.000  -0.039  +0.015  +0.013
                         :    var2:  -0.039  +1.000  -0.002  +0.009
                         :    var3:  +0.015  -0.002  +1.000  -0.033
                         :    var4:  +0.013  +0.009  -0.033  +1.000
                         : ----------------------------------------
DataSetFactory           : [Category_PyKeras_2_dsi] :  
                         : 
                         : Train method: Category_PyKeras_2 for Classification
                         : 
                         : ================================================================
                         : H e l p   f o r   M V A   m e t h o d   [ Category_PyKeras_2 ] :
                         : 
                         : Keras is a high-level API for the Theano and Tensorflow packages.
                         : This method wraps the training and predictions steps of the Keras
                         : Python package for TMVA, so that dataloading, preprocessing and
                         : evaluation can be done within the TMVA system. To use this Keras
                         : interface, you have to generate a model with Keras first. Then,
                         : this model can be loaded and trained in TMVA.
                         : 
                         : 
                         : <Suppress this message by specifying "!H" in the booking option>
                         : ================================================================
                         : 
TFHandler_Category_PyK...: Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:  -0.012741    0.27756   [    -1.0000     1.0000 ]
                         :     var2:   0.023274    0.26681   [    -1.0000     1.0000 ]
                         :     var3: -0.0046050    0.29587   [    -1.0000     1.0000 ]
                         :     var4:   0.032257    0.29912   [    -1.0000     1.0000 ]
                         : -----------------------------------------------------------
TFHandler_Category_PyK...: Variable        Mean        RMS   [        Min        Max ]
                         : -----------------------------------------------------------
                         :     var1:  -0.012741    0.27756   [    -1.0000     1.0000 ]
                         :     var2:   0.023274    0.26681   [    -1.0000     1.0000 ]
                         :     var3: -0.0046050    0.29587   [    -1.0000     1.0000 ]
                         :     var4:   0.032257    0.29912   [    -1.0000     1.0000 ]
                         : -----------------------------------------------------------
                         : Split TMVA training data in 5848 training events and 1461 validation events
                         : Option SaveBestOnly: Only model weights with smallest validation loss will be stored
Train on 5848 samples, validate on 1461 samples
Epoch 1/5
5848/5848 [==============================] - 1s 101us/step - loss: 0.8558 - acc: 0.5443 - val_loss: 0.8113 - val_acc: 0.5524

Epoch 00001: val_loss improved from inf to 0.81130, saving model to output/spectator/dataset_pymva/weights/spectator_test_model2.h5
Epoch 2/5
5848/5848 [==============================] - 0s 69us/step - loss: 0.8098 - acc: 0.5653 - val_loss: 0.8096 - val_acc: 0.5483

Epoch 00002: val_loss improved from 0.81130 to 0.80964, saving model to output/spectator/dataset_pymva/weights/spectator_test_model2.h5
Epoch 3/5
5848/5848 [==============================] - 0s 64us/step - loss: 0.8102 - acc: 0.5578 - val_loss: 0.8079 - val_acc: 0.5483

Epoch 00003: val_loss improved from 0.80964 to 0.80795, saving model to output/spectator/dataset_pymva/weights/spectator_test_model2.h5
Epoch 4/5
5848/5848 [==============================] - 0s 68us/step - loss: 0.8073 - acc: 0.5640 - val_loss: 0.8072 - val_acc: 0.5558

Epoch 00004: val_loss improved from 0.80795 to 0.80717, saving model to output/spectator/dataset_pymva/weights/spectator_test_model2.h5
Epoch 5/5
5848/5848 [==============================] - 0s 66us/step - loss: 0.8070 - acc: 0.5587 - val_loss: 0.8053 - val_acc: 0.5462

Epoch 00005: val_loss improved from 0.80717 to 0.80529, saving model to output/spectator/dataset_pymva/weights/spectator_test_model2.h5
                         : Elapsed time for training with 7309 events: 2.68 sec         
                         : Dataset[Category_PyKeras_2_dsi] : Create results for training
                         : Dataset[Category_PyKeras_2_dsi] : Multiclass evaluation of Category_PyKeras_2 on training sample
                         : Dataset[Category_PyKeras_2_dsi] : Elapsed time for evaluation of 7309 events: 2.69 sec       
                         : Creating multiclass response histograms...
                         : Creating multiclass performance histograms...
                         : Training finished
                         : Begin ranking of input variables...
                         : No variable ranking supplied by classifier: Category_PyKeras_1
                         : No variable ranking supplied by classifier: Category_PyKeras_2
                         : Elapsed time for training with 15000 events: 10.4 sec         
                         : Dataset[output/spectator/dataset_pymva] : Create results for training
                         : Dataset[output/spectator/dataset_pymva] : Multiclass evaluation of PyKerasCat on training sample
                         : Dataset[output/spectator/dataset_pymva] : Elapsed time for evaluation of 15000 events: 0.473 sec       
                         : Creating multiclass response histograms...

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================

Thread 6 (Thread 0x7f60ac0f6700 (LWP 21406)):
#0  0x00007f60cf08ba35 in pthread_cond_wait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f60c5452a4c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc-8.2.0-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc/8.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f6097066e37 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#4  0x00007f6097067779 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#5  0x00007f6097066582 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#6  0x00007f60c54584c0 in std::execute_native_thread_routine_compat (__p=<optimized out>) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc/8.2.0/libstdc++-v3/src/c++11/thread.cc:94
#7  0x00007f60cf087ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f60ce6a78dd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f60ab8f5700 (LWP 21405)):
#0  0x00007f60cf08ba35 in pthread_cond_wait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f60c5452a4c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc-8.2.0-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc/8.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f6097066e37 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#4  0x00007f6097067779 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#5  0x00007f6097066582 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#6  0x00007f60c54584c0 in std::execute_native_thread_routine_compat (__p=<optimized out>) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc/8.2.0/libstdc++-v3/src/c++11/thread.cc:94
#7  0x00007f60cf087ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f60ce6a78dd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f60a8bea700 (LWP 21404)):
#0  0x00007f60cf08ba35 in pthread_cond_wait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f60c5452a4c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc-8.2.0-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc/8.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f6097066e37 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#4  0x00007f6097067779 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#5  0x00007f6097066582 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#6  0x00007f60c54584c0 in std::execute_native_thread_routine_compat (__p=<optimized out>) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc/8.2.0/libstdc++-v3/src/c++11/thread.cc:94
#7  0x00007f60cf087ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f60ce6a78dd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f60a83e9700 (LWP 21399)):
#0  0x00007f60cf08ba35 in pthread_cond_wait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f60c5452a4c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc-8.2.0-build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc/8.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f6097066e37 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#4  0x00007f6097067779 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#5  0x00007f6097066582 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/tensorflow/libtensorflow_framework.so
#6  0x00007f60c54584c0 in std::execute_native_thread_routine_compat (__p=<optimized out>) at /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/build/contrib/gcc-8.2.0/src/gcc/8.2.0/libstdc++-v3/src/c++11/thread.cc:94
#7  0x00007f60cf087ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f60ce6a78dd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f60b607e700 (LWP 21370)):
#0  0x00007f60cf08db3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007f60cf08dbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f60cf08dc6b in sem_wait

GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007f60cf3d2a88 in PyThread_acquire_lock (lock=lock
entry=0x141c530, waitflag=waitflag
entry=1) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/thread_pthread.h:324
#4  0x00007f60cf3917a6 in PyEval_RestoreThread (tstate=tstate
entry=0x43e2cf0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:359
#5  0x00007f60b6081546 in floatsleep (secs=<optimized out>) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Modules/timemodule.c:1057
#6  time_sleep (self=<optimized out>, args=<optimized out>) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Modules/timemodule.c:206
#7  0x00007f60cf39b41b in call_function (oparg=<optimized out>, pp_stack=0x7f60b607d5d8) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4372
#8  PyEval_EvalFrameEx (f=f
entry=0x7f60b62a1050, throwflag=throwflag
entry=0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3009
#9  0x00007f60cf39c022 in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals
entry=0x0, args=args
entry=0x7f60b62960e8, argcount=<optimized out>, kws=kws
entry=0x7f60cf851068, kwcount=0, defs=0x0, defcount=0, closure=0x0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3604
#10 0x00007f60cf315944 in function_call (func=0x7f60b6b1c938, arg=0x7f60b62960d0, kw=0x7f60b629b398) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Objects/funcobject.c:523
#11 0x00007f60cf2ec0f3 in PyObject_Call (func=func
entry=0x7f60b6b1c938, arg=arg
entry=0x7f60b62960d0, kw=kw
entry=0x7f60b629b398) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Objects/abstract.c:2547
#12 0x00007f60cf3933fd in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7f60b607d850, func=0x7f60b6b1c938) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4686
#13 PyEval_EvalFrameEx (f=f
entry=0x7f60b6b35b00, throwflag=throwflag
entry=0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3048
#14 0x00007f60cf399e3f in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7f60b607d958, func=0x7f60b629ab90) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4457
#15 call_function (oparg=<optimized out>, pp_stack=0x7f60b607d958) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4392
#16 PyEval_EvalFrameEx (f=f
entry=0x7f60b0000910, throwflag=throwflag
entry=0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3009
#17 0x00007f60cf399e3f in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0x7f60b607da68, func=0x7f60b629acf8) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4457
#18 call_function (oparg=<optimized out>, pp_stack=0x7f60b607da68) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4392
#19 PyEval_EvalFrameEx (f=f
entry=0x7f60b62a0210, throwflag=throwflag
entry=0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3009
#20 0x00007f60cf39c022 in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals
entry=0x0, args=args
entry=0x7f60b62960a8, argcount=<optimized out>, kws=kws
entry=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3604
#21 0x00007f60cf31587b in function_call (func=0x7f60b629ac08, arg=0x7f60b6296090, kw=0x0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Objects/funcobject.c:523
#22 0x00007f60cf2ec0f3 in PyObject_Call (func=func
entry=0x7f60b629ac08, arg=arg
entry=0x7f60b6296090, kw=kw
entry=0x0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Objects/abstract.c:2547
#23 0x00007f60cf2f9fdc in instancemethod_call (func=0x7f60b629ac08, arg=0x7f60b6296090, kw=0x0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Objects/classobject.c:2600
#24 0x00007f60cf2ec0f3 in PyObject_Call (func=func
entry=0x7f60b7405a00, arg=arg
entry=0x7f60cf851050, kw=0x0) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Objects/abstract.c:2547
#25 0x00007f60cf391e53 in PyEval_CallObjectWithKeywords (func=0x7f60b7405a00, arg=0x7f60cf851050, kw=<optimized out>) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4241
#26 0x00007f60cf3d73b2 in t_bootstrap (boot_raw=0x43cf020) at /mnt/build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/build/externals/Python-2.7.15/src/Python/2.7.15/Modules/threadmodule.c:620
#27 0x00007f60cf087ea5 in start_thread () from /lib64/libpthread.so.0
#28 0x00007f60ce6a78dd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f60cf891740 (LWP 21336)):
#0  0x00007f60ce66e4b9 in waitpid () from /lib64/libc.so.6
#1  0x00007f60ce5ebf62 in do_system () from /lib64/libc.so.6
#2  0x00007f60ce5ec311 in system () from /lib64/libc.so.6
#3  0x00007f60c5a64713 in TUnixSystem::StackTrace() () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libCore.so
#4  0x00007f60c5a66fe4 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libCore.so
#5  <signal handler called>
#6  0x00007f60b4f8fe7c in TMVA::ResultsMulticlass::CreateMulticlassHistos(TString, int, int) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libTMVA.so
#7  0x00007f60b4e7e6c4 in TMVA::MethodBase::AddMulticlassOutput(TMVA::Types::ETreeType) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libTMVA.so
#8  0x00007f60b4e7d898 in TMVA::MethodBase::TrainMethod() () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libTMVA.so
#9  0x00007f60b4e2b73b in TMVA::Factory::TrainAllMethods() () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libTMVA.so
#10 0x00007f60b71d302a in ?? ()
#11 0x0000000000000000 in ?? ()
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#6  0x00007f60b4f8fe7c in TMVA::ResultsMulticlass::CreateMulticlassHistos(TString, int, int) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libTMVA.so
#7  0x00007f60b4e7e6c4 in TMVA::MethodBase::AddMulticlassOutput(TMVA::Types::ETreeType) () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libTMVA.so
#8  0x00007f60b4e7d898 in TMVA::MethodBase::TrainMethod() () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libTMVA.so
#9  0x00007f60b4e2b73b in TMVA::Factory::TrainAllMethods() () from /cvmfs/sft.cern.ch/lcg/views/LCG_95/x86_64-centos7-gcc8-opt/lib/libTMVA.so
#10 0x00007f60b71d302a in ?? ()
#11 0x0000000000000000 in ?? ()
===========================================================