How set set up configuration of DNN and CNN

TMVA
1

I have six input variables putting into the DNN and CNN model. How to set up the configuration, like layout and training? My code is below, but it terminates when I run the code. The BDT method works well, while DNN and CNN not. what’s wrong and how to fix it? Thank you for help.

#include <cstdlib>
#include <iostream>
#include <map>
#include <string>

#include "TChain.h"
#include "TFile.h"
#include "TTree.h"
#include "TString.h"
#include "TObjString.h"
#include "TSystem.h"
#include "TROOT.h"

#include "TMVA/Factory.h"
#include "TMVA/DataLoader.h"
#include "TMVA/Tools.h"
#include "TMVA/TMVAGui.h"

void NN_tth(){

//Declare Factory
TMVA::Tools::Instance();
auto sigFile = TFile::Open("./ttHiggs0PToGG.root");
auto bkgFile=TFile::Open("./ttHiggs0MToGG.root");

 // for using Keras
gSystem->Setenv("KERAS_BACKEND","tensorflow"); 
TMVA::PyMethodBase::PyInitialize();


TString outfileName("NN_tth_Classification.root");
auto outputFile = TFile::Open(outfileName, "RECREATE");
TMVA::Factory factory("NN_tth_Classification", outputFile,
                      "!V:ROC:!Silent:Color:!DrawProgressBar:AnalysisType=Classification" );

TString delta_phi_pho,delta_phi_jet;
delta_phi_pho="delta_phi_pho:=(abs(pho1_phi-pho2_phi))*(abs(pho1_phi-pho2_phi)<3.14)+\
(2*3.14-abs(pho1_phi-pho2_phi))*(abs(pho1_phi-pho2_phi)>3.14)";
delta_phi_jet="delta_phi_jet:=(abs(jetPhi_1-jetPhi_2))*(abs(jetPhi_1-jetPhi_2)<3.14)+\
(2*3.14-abs(jetPhi_1-jetPhi_2))*(abs(jetPhi_1-jetPhi_2)>3.14)";

//Declare DataLoader(s)
TMVA::DataLoader loader("dataset");
loader.AddVariable("delta_phoj_eta:=pho1_eta-jetEta_1",'F');
loader.AddVariable("delta_phoj_phi:=pho1_phi-jetPhi_1",'F');
loader.AddVariable("delata_jj_eta:=jetEta_1-jetEta_2",'F');
loader.AddVariable("delta_pho_eta:=pho1_eta-pho2_eta",'F');
loader.AddVariable(delta_phi_pho,"delta_phi_pho","",'F');
loader.AddVariable(delta_phi_jet,"delta_phi_jet","",'F');

//Setup Dataset(s)
TTree *tsignal, *tbackground;
sigFile->GetObject("ttH_0P_125_13TeV_TTHHadronicTag", tsignal);
bkgFile->GetObject("ttH_0M_125_13TeV_TTHHadronicTag", tbackground);

TCut mycuts, mycutb;

Double_t signalWeight     = 1.0;
Double_t backgroundWeight = 1.0;
loader.AddSignalTree    (tsignal,     signalWeight);   //signal weight  = 1
loader.AddBackgroundTree(tbackground, backgroundWeight);   //background weight = 1 

loader.SetBackgroundWeightExpression( "weight" ); //Set individual event weights 
loader.SetSignalWeightExpression("weight");
loader.PrepareTrainingAndTestTree(mycuts, mycutb,
                                   "nTrain_Signal=3000:nTrain_Background=3000:SplitMode=Random:NormMode=NumEvents:!V" );
//Booking Methods
//Boosted Decision Trees
factory.BookMethod(&loader,TMVA::Types::kBDT, "BDT",
                   "!V:NTrees=200:MinNodeSize=2.5%:MaxDepth=2:BoostType=AdaBoost:AdaBoostBeta=0.5:UseBaggedBoost:BaggedSampleFraction=0.5:SeparationType=GiniIndex:nCuts=20" );

//Booking  Neural Network
//Here we book the new DNN of TMVA. If using master version you can use the new DL method

bool useDNN = true; 
bool useCNN = true; 
bool useKeras = false;

if (useDNN) { 
    
     TString layoutString ("Layout=TANH|128,TANH|128,TANH|128,LINEAR");

      // Training strategies.
      TString training0("LearningRate=1e-1,Momentum=0.9,Repetitions=1,"
                        "ConvergenceSteps=20,BatchSize=256,TestRepetitions=10,"
                        "WeightDecay=1e-4,Regularization=L2,"
                        "DropConfig=0.0+0.5+0.5+0.5, Multithreading=True");
      TString training1("LearningRate=1e-2,Momentum=0.9,Repetitions=1,"
                        "ConvergenceSteps=20,BatchSize=256,TestRepetitions=10,"
                        "WeightDecay=1e-4,Regularization=L2,"
                        "DropConfig=0.0+0.0+0.0+0.0, Multithreading=True");
      TString training2("LearningRate=1e-3,Momentum=0.0,Repetitions=1,"
                        "ConvergenceSteps=20,BatchSize=256,TestRepetitions=10,"
                        "WeightDecay=1e-4,Regularization=L2,"
                        "DropConfig=0.0+0.0+0.0+0.0, Multithreading=True");
      TString trainingStrategyString ("TrainingStrategy=");
      trainingStrategyString += training0 + "|" + training1 + "|" + training2;

      // General Options.                                                                                                                                                                
      TString dnnOptions ("!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:"
                          "WeightInitialization=XAVIERUNIFORM");
      dnnOptions.Append (":"); dnnOptions.Append (layoutString);
      dnnOptions.Append (":"); dnnOptions.Append (trainingStrategyString);

      dnnOptions += ":Architecture=CPU";
      factory.BookMethod(&loader, TMVA::Types::kDNN, "DNN_CPU", dnnOptions);

}

//Book Convolutional Neural Network in TMVA
if (useCNN) { 
    TString inputLayoutString("InputLayout=1|8|8");
                                                                                                
// Batch Layout                                                                                                                                     
    TString batchLayoutString("BatchLayout=256|1|64");
                                                   

	TString layoutString("Layout=CONV|10|3|3|1|1|1|1|RELU,CONV|10|3|3|1|1|1|1|RELU,MAXPOOL|2|2|1|1,"
                     "RESHAPE|FLAT,DENSE|64|TANH,DENSE|1|LINEAR");
                                                                                                                                              


   // Training strategies.                                                                                                                          
   TString training0("LearningRate=1e-1,Momentum=0.9,Repetitions=1,"
                     "ConvergenceSteps=20,BatchSize=256,TestRepetitions=5,"
                     "WeightDecay=1e-4,Regularization=None,"
                     "DropConfig=0.0+0.5+0.5+0.5, Multithreading=False");
 
   TString trainingStrategyString ("TrainingStrategy=");
   trainingStrategyString += training0; // + "|" + training1 + "|" + training2;   }
    
// General Options.                                                                                                                              
   TString cnnOptions ("!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=None:"
                       "WeightInitialization=XAVIERUNIFORM");

   cnnOptions.Append(":"); cnnOptions.Append(inputLayoutString);
   cnnOptions.Append(":"); cnnOptions.Append(batchLayoutString);
   cnnOptions.Append(":"); cnnOptions.Append(layoutString);
   cnnOptions.Append(":"); cnnOptions.Append(trainingStrategyString);
   cnnOptions.Append(":Architecture=CPU");

   //// New DL (CNN)                                                                                                                                


  factory.BookMethod(&loader, TMVA::Types::kDL, "DL_CNN_CPU", cnnOptions);
}

//Book Convolutional Neural Network in Keras using a generated model
if (useKeras) { 
   factory.BookMethod(&loader, TMVA::Types::kPyKeras, 
                       "PyKeras","H:!V:VarTransform=None:FilenameModel=model_cnn.h5:"
                       "FilenameTrainedModel=trained_model_cnn.h5:NumEpochs=20:BatchSize=256");
}

//Train Methods
factory.TrainAllMethods();
//Test and Evaluate Methods
factory.TestAllMethods();
factory.EvaluateAllMethods();
outputFile->Close(); // Save the output

//Plot ROC Curve
auto c1 = factory.GetROCCurve(&loader);
c1->Draw();
gPad->Print("Significance.png");
  
  // Launch the GUI for the root macros
if (!gROOT->IsBatch()) TMVA::TMVAGui(outfileName);

}
2

Hi,

The configuration for the DNN should work fine, which error message are you getting ?
For the CNN you have instead 6 input variables and the CNN expects an image of 8x8 variables , i.e 64.
Now with only 6 variables which are not really representing an image I am not sure it makes really sense using a CNN. But in case you need to use you need to specify correctly the inputLayoutString and batch LayoutString.
Supposing your having an image of size 3x2, you would do:

    TString inputLayoutString("InputLayout=1|3|2");                                                                                                                                                                                                                                 
    TString batchLayoutString("BatchLayout=256|1|6");

Also you might need to check the right padding to get sensible output from the convolutional layers.
Remember

output_size = (input_size - filter_size + 2 * padding)/stride + 1

so with a filter size of 3x3 you need to use stride=1 and padding=1 to get an output size equal to the input size. The parameters you have are fine if you want the same output.

One additional note. In the latest ROOT version it is recommended to use also for dense (fully connected) layer the more general MethodDL (TMVA::Types::kDL).

For using this you need to set the input layout and batch layout string as following

    TString inputLayoutString("InputLayout=1|1|6");                                                                                                                                                                                                                                 
    TString batchLayoutString("BatchLayout=1|256|6");

and for the network (3 layers of 128 neutrons + 1 output layer )

TString layoutString("Layout=DENSE|128,TANH,DENSE|128|TANH,DENSE|128|TANH,DENSE|1|LINEAR");

I would also suggest to use the ADAM optimiser with a smaller learning rate (e.g. 10E-3) and no need to dropout with a small network.

TString training0("LearningRate=1e-3,Repetitions=1,"
                     "ConvergenceSteps=20,BatchSize=256,TestRepetitions=5,"
                     "WeightDecay=1e-4,Regularization=None,"
                     "DropConfig=0.0+0.0+0.0+0.0, Optimizer=ADAM");

You can look at this notebook as an example:

Best Regards

Lorenzo