Segmentation fault using new method "AddVariablesArray"

TheZardoz · September 14, 2020, 1:42pm

Hi,
I tried to use the RNN from TMVA using the example : https://root.cern/doc/master/TMVA__RNN__Classification_8C.html

This example use the new function “AddVariablesArray”

For my case i used a tree with 500 samples of 50*3500 entries (the creator of the example used different number but the idea is the same).

However the program break into segmentation fault after loading the Signal event as follow :

fo in <TMVA_RNN_Classification>: Booking Keras GRU model
Factory : Booking method: PyKeras_GRU
:
: Using TensorFlow backend - setting special configuration options
: Using Tensorflow version 2
: Applying GPU option: gpu_options.allow_growth=True
: Load model from file: model_GRU.h5
Factory : Booking method: BDTG
:
: the option NegWeightTreatment=InverseBoostNegWeights does not exist for BoostType=Grad
: → change to new default NegWeightTreatment=Pray
: Building event vectors for type 2 Signal
: Dataset[dataset] : create input formulas for tree RNN_TWFS
: Using variable RNN_N_WF_0[0] from array expression RNN_N_WF_0 of size 50
: Using variable RNN_N_WF_1[0] from array expression RNN_N_WF_1 of size 50
: Using variable RNN_N_WF_2[0] from array expression RNN_N_WF_2 of size 50
: Using variable RNN_N_WF_3[0] from array expression RNN_N_WF_3 of size 50
: Using variable RNN_N_WF_4[0] from array expression RNN_N_WF_4 of size 50
: Using variable RNN_N_WF_5[0] from array expression RNN_N_WF_5 of size 50
: Using variable RNN_N_WF_6[0] from array expression RNN_N_WF_6 of size 50
: Using variable RNN_N_WF_7[0] from array expression RNN_N_WF_7 of size 50
: Using variable RNN_N_WF_8[0] from array expression RNN_N_WF_8 of size 50
: Using variable RNN_N_WF_9[0] from array expression RNN_N_WF_9 of size 50
: Using variable RNN_N_WF_10[0] from array expression RNN_N_WF_10 of size 50
: Using variable RNN_N_WF_11[0] from array expression RNN_N_WF_11 of size 50
: Using variable RNN_N_WF_12[0] from array expression RNN_N_WF_12 of size 50
: Using variable RNN_N_WF_13[0] from array expression RNN_N_WF_13 of size 50
: Using variable RNN_N_WF_14[0] from array expression RNN_N_WF_14 of size 50
: Using variable RNN_N_WF_15[0] from array expression RNN_N_WF_15 of size 50
: Using variable RNN_N_WF_16[0] from array expression RNN_N_WF_16 of size 50
: Using variable RNN_N_WF_17[0] from array expression RNN_N_WF_17 of size 50
: Using variable RNN_N_WF_18[0] from array expression RNN_N_WF_18 of size 50
: Using variable RNN_N_WF_19[0] from array expression RNN_N_WF_19 of size 50
: Using variable RNN_N_WF_20[0] from array expression RNN_N_WF_20 of size 50
: Using variable RNN_N_WF_21[0] from array expression RNN_N_WF_21 of size 50
: Using variable RNN_N_WF_22[0] from array expression RNN_N_WF_22 of size 50
: Using variable RNN_N_WF_23[0] from array expression RNN_N_WF_23 of size 50
: Using variable RNN_N_WF_24[0] from array expression RNN_N_WF_24 of size 50
: Using variable RNN_N_WF_25[0] from array expression RNN_N_WF_25 of size 50
: Using variable RNN_N_WF_26[0] from array expression RNN_N_WF_26 of size 50
: Using variable RNN_N_WF_27[0] from array expression RNN_N_WF_27 of size 50
: Using variable RNN_N_WF_28[0] from array expression RNN_N_WF_28 of size 50
: Using variable RNN_N_WF_29[0] from array expression RNN_N_WF_29 of size 50
: Using variable RNN_N_WF_30[0] from array expression RNN_N_WF_30 of size 50
: Using variable RNN_N_WF_31[0] from array expression RNN_N_WF_31 of size 50
: Using variable RNN_N_WF_32[0] from array expression RNN_N_WF_32 of size 50
: Using variable RNN_N_WF_33[0] from array expression RNN_N_WF_33 of size 50
: Using variable RNN_N_WF_34[0] from array expression RNN_N_WF_34 of size 50
: Using variable RNN_N_WF_35[0] from array expression RNN_N_WF_35 of size 50
: Using variable RNN_N_WF_36[0] from array expression RNN_N_WF_36 of size 50
: Using variable RNN_N_WF_37[0] from array expression RNN_N_WF_37 of size 50
: Using variable RNN_N_WF_38[0] from array expression RNN_N_WF_38 of size 50
: Using variable RNN_N_WF_39[0] from array expression RNN_N_WF_39 of size 50
: Using variable RNN_N_WF_40[0] from array expression RNN_N_WF_40 of size 50
: Using variable RNN_N_WF_41[0] from array expression RNN_N_WF_41 of size 50
: Using variable RNN_N_WF_42[0] from array expression RNN_N_WF_42 of size 50
: Using variable RNN_N_WF_43[0] from array expression RNN_N_WF_43 of size 50
: Using variable RNN_N_WF_44[0] from array expression RNN_N_WF_44 of size 50
: Using variable RNN_N_WF_45[0] from array expression RNN_N_WF_45 of size 50
: Using variable RNN_N_WF_46[0] from array expression RNN_N_WF_46 of size 50
: Using variable RNN_N_WF_47[0] from array expression RNN_N_WF_47 of size 50
: Using variable RNN_N_WF_48[0] from array expression RNN_N_WF_48 of size 50
: Using variable RNN_N_WF_49[0] from array expression RNN_N_WF_49 of size 50
: Using variable RNN_N_WF_50[0] from array expression RNN_N_WF_50 of size 50
: Using variable RNN_N_WF_51[0] from array expression RNN_N_WF_51 of size 50
: Using variable RNN_N_WF_52[0] from array expression RNN_N_WF_52 of size 50
: Using variable RNN_N_WF_53[0] from array expression RNN_N_WF_53 of size 50
: Using variable RNN_N_WF_54[0] from array expression RNN_N_WF_54 of size 50
: Using variable RNN_N_WF_55[0] from array expression RNN_N_WF_55 of size 50
: Using variable RNN_N_WF_56[0] from array expression RNN_N_WF_56 of size 50
: Using variable RNN_N_WF_57[0] from array expression RNN_N_WF_57 of size 50
: Using variable RNN_N_WF_58[0] from array expression RNN_N_WF_58 of size 50
: Using variable RNN_N_WF_59[0] from array expression RNN_N_WF_59 of size 50
: Using variable RNN_N_WF_60[0] from array expression RNN_N_WF_60 of size 50
: Using variable RNN_N_WF_61[0] from array expression RNN_N_WF_61 of size 50
: Using variable RNN_N_WF_62[0] from array expression RNN_N_WF_62 of size 50
: Using variable RNN_N_WF_63[0] from array expression RNN_N_WF_63 of size 50
: Using variable RNN_N_WF_64[0] from array expression RNN_N_WF_64 of size 50
: Using variable RNN_N_WF_65[0] from array expression RNN_N_WF_65 of size 50
: Using variable RNN_N_WF_66[0] from array expression RNN_N_WF_66 of size 50
: Using variable RNN_N_WF_67[0] from array expression RNN_N_WF_67 of size 50
: Using variable RNN_N_WF_68[0] from array expression RNN_N_WF_68 of size 50
: Using variable RNN_N_WF_69[0] from array expression RNN_N_WF_69 of size 50
: Using variable RNN_N_WF_70[0] from array expression RNN_N_WF_70 of size 50
: Using variable RNN_N_WF_71[0] from array expression RNN_N_WF_71 of size 50
: Using variable RNN_N_WF_72[0] from array expression RNN_N_WF_72 of size 50
: Using variable RNN_N_WF_73[0] from array expression RNN_N_WF_73 of size 50
: Using variable RNN_N_WF_74[0] from array expression RNN_N_WF_74 of size 50
: Using variable RNN_N_WF_75[0] from array expression RNN_N_WF_75 of size 50
: Using variable RNN_N_WF_76[0] from array expression RNN_N_WF_76 of size 50
ect …

…
: Using variable RNN_N_WF_492[0] from array expression RNN_N_WF_492 of size 50
: Using variable RNN_N_WF_493[0] from array expression RNN_N_WF_493 of size 50
: Using variable RNN_N_WF_494[0] from array expression RNN_N_WF_494 of size 50
: Using variable RNN_N_WF_495[0] from array expression RNN_N_WF_495 of size 50
: Using variable RNN_N_WF_496[0] from array expression RNN_N_WF_496 of size 50
: Using variable RNN_N_WF_497[0] from array expression RNN_N_WF_497 of size 50
: Using variable RNN_N_WF_498[0] from array expression RNN_N_WF_498 of size 50
: Using variable RNN_N_WF_499[0] from array expression RNN_N_WF_499 of size 50

*** Break *** segmentation violation
[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[/usr/lib/dyld] dyld::gLinkContext (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libTreePlayer.so] double TFormLeafInfoCollection::GetValueImpl(TLeaf*, int) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libTMVA.so] TMVA::DataSetFactory::BuildEventVector(TMVA::DataSetInfo&, TMVA::DataInputHandler&, std::__1::map<TMVA::Types::ETreeType, std::__1::vector<std::__1::vector<TMVA::Event*, std::__1::allocatorTMVA::Event* >, std::__1::allocator<std::__1::vector<TMVA::Event*, std::__1::allocatorTMVA::Event* > > >, std::__1::lessTMVA::Types::ETreeType, std::__1::allocator<std::__1::pair<TMVA::Types::ETreeType const, std::__1::vector<std::__1::vector<TMVA::Event*, std::__1::allocatorTMVA::Event* >, std::__1::allocator<std::__1::vector<TMVA::Event*, std::__1::allocatorTMVA::Event* > > > > > >&, std::__1::vector<TMVA::DataSetFactory::EventStats, std::__1::allocatorTMVA::DataSetFactory::EventStats >&) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libTMVA.so] TMVA::DataSetFactory::BuildInitialDataSet(TMVA::DataSetInfo&, TMVA::DataInputHandler&) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libTMVA.so] TMVA::DataSetFactory::CreateDataSet(TMVA::DataSetInfo&, TMVA::DataInputHandler&) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libTMVA.so] TMVA::DataSetManager::CreateDataSet(TString const&) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libTMVA.so] TMVA::DataSetInfo::GetDataSet() const (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libTMVA.so] TMVA::MethodBDT::ProcessOptions() (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libTMVA.so] TMVA::BookMethod(TMVA::DataLoader*, TString, TString, TString) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libTMVA.so] TMVA::BookMethod(TMVA::DataLoader*, TMVA::Types::EMVA, TString, TString) (no debug info)
[] (no debug info)
[] (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] cling::Interpreter::EvaluateInternal(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCling.so] TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libCore.so] TApplication::ExecuteFile(char const*, int*, bool) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libRint.so] TRint::ProcessLineNr(char const*, char const*, int*) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/lib/root/libRint.so] TRint::Run(bool) (no debug info)
[/usr/local/Cellar/root/6.22.00_1/bin/root.exe] main (no debug info)
[/usr/lib/system/libdyld.dylib] start (no debug info)
[] (no debug info)

Is the new method “AddVariablesArray” not confortable with vector of vector ?

Here the script i used (it’s the same from the example but modified a little)TMVA_RNN_Classification.C (13.8 KB)

jblomer · September 15, 2020, 7:12am

@moneta Can you help?

moneta · September 15, 2020, 2:11pm

Hi,
What is the type of the branch in the Tree : RNN_N_WF_... . It should be a collection, e.g. an std::vector or similar. Can you post a small snapshot of your data, just a copy with few events, so I can reproduce your error

Thank you

Lorenzo

TheZardoz · September 16, 2020, 10:56am

Hi,
The type of the branch is an std::vector . Here a small snapshot of my data

test_small_OUT_RNN_OUT_Alphas_Dataset_BiPo.root (1.6 MB) test_small_OUT_RNN_OUT_Neutron_AmBe_Complete_Source_Stack.root (1.7 MB)

moneta · September 16, 2020, 1:03pm

Hi,

It looks to me that in the Tree the std::vector have different sizes Are the vector size sometimes zero for some events ? That could explain the crash observed.
I would try to re-create a Tree where for each entry each vector has a size=50. If it is not the case I would add a vector padding with zero values

Lorenzo

TheZardoz · September 16, 2020, 3:13pm

Hi,

Normally all vector have the same size (50) except for the last one but we didn’t take into account this one in the RNN script.

Zardoz

moneta · September 17, 2020, 8:56am

Hi,
I can reproduce the crash in the data I have. The problem (in the data you have posted) is that the TTree’s have 500 entries but only for the first 50 entries the arrays have size=50.
You can check this by doing:

tree->Draw(Draw("Length$(RNN_N_WF_0)","Entry$>=50")

You can strip the tree and keep only the good entries, unfortunately you cannot use the tree’s as they are in TMVA, but you can fix this before using TMVA with `TTree::CloneTree``:

auto file = TFile::Open("test_small_OUT_RNN_OUT_Alphas_Dataset_BiPo.root");
auto * t = (TTree*) file->Get("RNN_Results/RNN_TWFS");
TFile fileOut("test_new_Alphas.root","NEW");
auto tnew = t->CloneTree(50);  // copy the first 50 entry of the tree
tnew->Write(); 
fileOut.Close();

after doing this your macro works fine with these data

Lorenzo

TheZardoz · September 17, 2020, 11:16am

Hi,

Indeed, it’s working after these modifications. Thank you very much !

moneta · September 17, 2020, 12:33pm

Good!
I have added anyway a check in TMVA for this case by printing an error message.
See https://github.com/root-project/root/pull/6372

Lorenzo