TMVA.CrossValidation crashes on destructor (PyROOT)

TMVA
1

I get a segmentation fault after performing a TMVA classification with CrossValidation, i.e.

*** Error in `python3': double free or corruption (out): 0x0000000007d1fc10 ***

which seems related to memory deallocation in the CrossValidation destructor. Please find attached a minimal example reproducing the error and the full log file (obtained with PyROOT, ROOT 6.24/06). The same code works fine when using classification with TMVA.Factory, as can be reproduced by calling train_classifier(cross_validation=False) in the last line of the minimal example.
Is this a bug, or I am making some trivial mistakes in my code?

Thanks in advance for your help!

PS: it seems that as a new user I cannot put links in my post. I attach the content of the minimal example verbatim instead:

#!/usr/bin/env python3

from ROOT import TChain, TFile, TTree, TCut, TMVA, gRandom
from array import array


def make_tree(label):
    f = TFile(f'{label}.root', 'recreate')
    tree = TTree('tree', '')
    var = array('f', [0.])
    eventNumber = array('i', [0])
    tree.Branch('var', var, 'var/F')
    tree.Branch('eventNumber', eventNumber, 'eventNumber/I')
    for i in range(int(1e3)):
        var[0] = gRandom.Gaus(0 if label == 'sig' else 1, 1)
        eventNumber[0] = i
        tree.Fill()
    tree.Write()
    f.Close()


def train_classifier(cross_validation = True):
    tree_sig = TChain('tree')
    tree_bkg = TChain('tree')
    tree_sig.AddFile('sig.root')
    tree_bkg.AddFile('bkg.root')

    out_file = TFile('cv-minimal.root', 'recreate')

    data_loader_name = 'dl-minimal'
    data_loader = TMVA.DataLoader(data_loader_name)
    data_loader.AddVariable('var', 'F')
    if cross_validation:
        data_loader.AddSpectator('eventNumber := eventNumber % 4096', 'I')
    data_loader.AddSignalTree(tree_sig)
    data_loader.AddBackgroundTree(tree_bkg)

    dataset_options = 'nTest_Signal=0'\
                      ':nTest_Background=0'\
                      ':!V'
    if not cross_validation:
        dataset_options += ':nTrain_Signal=0'\
                           ':nTrain_Background=0'
    data_loader.PrepareTrainingAndTestTree(TCut(), TCut(), dataset_options)

    method_options_mlp = 'H:!V:NeuronType=tanh:VarTransform=N:NCycles=600:'\
                         'HiddenLayers=N+5:TestRate=5:!UseRegulator'

    if cross_validation:
        cv_options = 'AnalysisType=Classification'\
                     ':NumFolds=2'\
                     ':SplitType=Deterministic'\
                     ':SplitExpr=(int(fabs([eventNumber]))%2)'\
                     ':!V'\
                     ':!Silent'
        cv = TMVA.CrossValidation('TMVACrossValidation', data_loader, out_file, cv_options)
        cv.BookMethod(TMVA.Types.kMLP, 'MLP', method_options_mlp)
        cv.Evaluate()
    else:
        factory_options = 'AnalysisType=Classification'
        factory = TMVA.Factory('TMVAClassification', out_file, factory_options)
        factory.BookMethod(data_loader, TMVA.Types.kMLP, 'MLP', method_options_mlp)
        factory.TrainAllMethods()
        factory.TestAllMethods()
        factory.EvaluateAllMethods()

    out_file.Close()


if __name__ == '__main__':
    make_tree('sig')
    make_tree('bkg')
    train_classifier(False)

and part of the log:

Factory                  : e[1mThank you for using TMVA!e[0m
                         : e[1mFor citation information, please visit: http://tmva.sf.net/citeTMVA.htmle[0m
                         : Evaluation done.
 *** Break *** abort

*** Error in `python3': double free or corruption (out): 0x0000000007d1fc10 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f31f6514329]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.24.06-3455f/x86_64-centos7-gcc11-opt/lib/libTMVA.so(_ZN4TMVA15CrossValidationD1Ev+0x9f)[0x7f31d42307ef]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.24.06-3455f/x86_64-centos7-gcc11-opt/lib/libTMVA.so(_ZN4TMVA15CrossValidationD0Ev+0x9)[0x7f31d4230c79]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libcppyy3_9.so(_ZN8CPyCppyy17op_dealloc_nofreeEPNS_11CPPInstanceE+0x59)[0x7f31efc60c39]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libcppyy3_9.so(+0x59d51)[0x7f31efc60d51]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x10dd2f)[0x7f31f74cad2f]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x68e05)[0x7f31f7425e05]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x68c58)[0x7f31f7425c58]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x56f3)[0x7f31f742c273]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x19590e)[0x7f31f755290e]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalCodeWithName+0x4e)[0x7f31f7552c4e]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(PyEval_EvalCodeEx+0x3b)[0x7f31f7552c9b]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(PyEval_EvalCode+0x1b)[0x7f31f7552ccb]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x1d5d6e)[0x7f31f7592d6e]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(PyRun_SimpleFileExFlags+0x193)[0x7f31f7594783]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(Py_RunMain+0x758)[0x7f31f75b1b48]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(Py_BytesMain+0x47)[0x7f31f75b2007]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f31f64b5555]
python3[0x40108e]
======= Memory map: ========
00400000-00401000 r--p 00000000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
00401000-00402000 r-xp 00001000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
00402000-00403000 r--p 00002000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
00403000-00404000 r--p 00002000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
00404000-00405000 rw-p 00003000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
0183b000-08bf1000 rw-p 00000000 00:00 0                                  [heap]
7f317fbf1000-7f318188e000 rw-p 00000000 00:00 0 
7f318188e000-7f31a188e000 rw-p 00000000 00:00 0 
7f31a188e000-7f31a188f000 ---p 00000000 00:00 0 
7f31a188f000-7f31a208f000 rw-p 00000000 00:00 0 
7f31a208f000-7f31aa08f000 rw-p 00000000 00:00 0 
7f31aa08f000-7f31aa090000 ---p 00000000 00:00 0 
7f31aa090000-7f31aa890000 rw-p 00000000 00:00 0 
7f31aa890000-7f31aa891000 ---p 00000000 00:00 0 
7f31aa891000-7f31ab091000 rw-p 00000000 00:00 0 
7f31ab091000-7f31bb091000 rw-p 00000000 00:00 0 
7f31bb091000-7f31bb092000 ---p 00000000 00:00 0 
7f31bb092000-7f31bb892000 rw-p 00000000 00:00 0 
7f31bb892000-7f31cb892000 rw-p 00000000 00:00 0 
7f31cbdaa000-7f31cc3fa000 r--s 00000000 fc:01 58739049                   /var/lib/sss/mc/group
7f31cc3fa000-7f31ce097000 rw-p 00000000 00:00 0 
7f31ce097000-7f31ce09a000 r--p 00000000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce09a000-7f31ce0c6000 r-xp 00003000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce0c6000-7f31ce0de000 r--p 0002f000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce0de000-7f31ce0df000 r--p 00046000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce0df000-7f31ce0e0000 rw-p 00047000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce0e0000-7f31ce0ec000 r--p 00000000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce0ec000-7f31ce151000 r-xp 0000c000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce151000-7f31ce178000 r--p 00071000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce178000-7f31ce17f000 r--p 00097000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce17f000-7f31ce180000 rw-p 0009e000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce180000-7f31ce19f000 r--p 00000000 00:38 247103163                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libgfortran.so.5.0.0
7f31ce19f000-7f31ce3f8000 r-xp 0001f000 00:38 247103163                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libgfortran.so.5.0.0*** Error in `python3': free(): invalid next size (fast): 0x0000000007d1fbd0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f31f6514329]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCling.so(+0x3065f24)[0x7f31ed27bf24]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCling.so(+0x305c521)[0x7f31ed272521]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCling.so(+0x69cfc3)[0x7f31ea8b2fc3]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCling.so(+0x69d299)[0x7f31ea8b3299]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCling.so(_ZN6TClingD1Ev+0x258)[0x7f31ea7f0aa8]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCling.so(_ZN6TClingD0Ev+0x9)[0x7f31ea7f0d79]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCore.so(_ZN5TROOTD1Ev+0x443)[0x7f31ef8262b3]
/lib64/libc.so.6(+0x39ce9)[0x7f31f64ccce9]
/lib64/libc.so.6(+0x39d37)[0x7f31f64ccd37]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCore.so(_ZN11TUnixSystem4ExitEib+0x1a)[0x7f31ef97b01a]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCore.so(_ZN11TUnixSystem15DispatchSignalsE8ESignals+0x41)[0x7f31ef97ed01]
/lib64/libpthread.so.0(+0xf630)[0x7f31f6f79630]
/lib64/libc.so.6(gsignal+0x37)[0x7f31f64c9387]
/lib64/libc.so.6(abort+0x148)[0x7f31f64caa78]
/lib64/libc.so.6(+0x78f67)[0x7f31f650bf67]
/lib64/libc.so.6(+0x81329)[0x7f31f6514329]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.24.06-3455f/x86_64-centos7-gcc11-opt/lib/libTMVA.so(_ZN4TMVA15CrossValidationD1Ev+0x9f)[0x7f31d42307ef]
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.24.06-3455f/x86_64-centos7-gcc11-opt/lib/libTMVA.so(_ZN4TMVA15CrossValidationD0Ev+0x9)[0x7f31d4230c79]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libcppyy3_9.so(_ZN8CPyCppyy17op_dealloc_nofreeEPNS_11CPPInstanceE+0x59)[0x7f31efc60c39]
/cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libcppyy3_9.so(+0x59d51)[0x7f31efc60d51]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x10dd2f)[0x7f31f74cad2f]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x68e05)[0x7f31f7425e05]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x68c58)[0x7f31f7425c58]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x56f3)[0x7f31f742c273]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x19590e)[0x7f31f755290e]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalCodeWithName+0x4e)[0x7f31f7552c4e]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(PyEval_EvalCodeEx+0x3b)[0x7f31f7552c9b]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(PyEval_EvalCode+0x1b)[0x7f31f7552ccb]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(+0x1d5d6e)[0x7f31f7592d6e]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(PyRun_SimpleFileExFlags+0x193)[0x7f31f7594783]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(Py_RunMain+0x758)[0x7f31f75b1b48]
/cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/../lib/libpython3.9.so.1.0(Py_BytesMain+0x47)[0x7f31f75b2007]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f31f64b5555]
python3[0x40108e]
======= Memory map: ========
00400000-00401000 r--p 00000000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
00401000-00402000 r-xp 00001000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
00402000-00403000 r--p 00002000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
00403000-00404000 r--p 00002000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
00404000-00405000 rw-p 00003000 00:38 247170758                          /cvmfs/sft.cern.ch/lcg/releases/Python/3.9.6-b0f98/x86_64-centos7-gcc11-opt/bin/python3.9
0183b000-08bf1000 rw-p 00000000 00:00 0                                  [heap]
7f3178000000-7f3178046000 rw-p 00000000 00:00 0 
7f3178046000-7f317c000000 ---p 00000000 00:00 0 
7f318188e000-7f31a188e000 rw-p 00000000 00:00 0 
7f31a188e000-7f31a188f000 ---p 00000000 00:00 0 
7f31a188f000-7f31a208f000 rw-p 00000000 00:00 0 
7f31a208f000-7f31aa08f000 rw-p 00000000 00:00 0 
7f31aa08f000-7f31aa090000 ---p 00000000 00:00 0 
7f31aa090000-7f31aa890000 rw-p 00000000 00:00 0 
7f31aa890000-7f31aa891000 ---p 00000000 00:00 0 
7f31aa891000-7f31ab091000 rw-p 00000000 00:00 0 
7f31ab091000-7f31bb091000 rw-p 00000000 00:00 0 
7f31bb091000-7f31bb092000 ---p 00000000 00:00 0 
7f31bb092000-7f31bb892000 rw-p 00000000 00:00 0 
7f31bb892000-7f31cb892000 rw-p 00000000 00:00 0 
7f31cbdaa000-7f31cc3fa000 r--s 00000000 fc:01 58739049                   /var/lib/sss/mc/group
7f31ce097000-7f31ce09a000 r--p 00000000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce09a000-7f31ce0c6000 r-xp 00003000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce0c6000-7f31ce0de000 r--p 0002f000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce0de000-7f31ce0df000 r--p 00046000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce0df000-7f31ce0e0000 rw-p 00047000 00:38 247102577                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libquadmath.so.0.0.0
7f31ce0e0000-7f31ce0ec000 r--p 00000000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce0ec000-7f31ce151000 r-xp 0000c000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce151000-7f31ce178000 r--p 00071000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce178000-7f31ce17f000 r--p 00097000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce17f000-7f31ce180000 rw-p 0009e000 00:38 248702855                  /cvmfs/sft.cern.ch/lcg/releases/freetype/2.10.0-08d63/x86_64-centos7-gcc11-opt/lib/libfreetype.so.6.17.0
7f31ce180000-7f31ce19f000 r--p 00000000 00:38 247103163                  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.1.0-e80bf/x86_64-centos7/lib64/libgfortran.so.5.0.0Aborted (core dumped)

===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================

Thread 2 (Thread 0x7f31d50e0700 (LWP 5399) "python3"):
#0  0x00007f31f6f75de2 in pthread_cond_timedwait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f31f75516e7 in PyCOND_TIMEDWAIT (us=<optimized out>, mut=0x7f31f7764f70 <_PyRuntime+432>, cond=0x7f31f7764f40 <_PyRuntime+384>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/condvar.h:73
#2  take_gil (tstate=tstate
entry=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval_gil.h:247
#3  0x00007f31f7551b2e in PyEval_RestoreThread (tstate=tstate
entry=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:467
#4  0x00007f31f75f83f2 in pysleep (secs=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Modules/timemodule.c:2037
#5  time_sleep (self=<optimized out>, obj=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Modules/timemodule.c:365
#6  0x00007f31f74b3a73 in cfunction_vectorcall_O (func=0x7f31f78cba90, args=0x7f31d51c1ec0, nargsf=<optimized out>, kwnames=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Objects/methodobject.c:512
#7  0x00007f31f742d892 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f31d51c1ec0, callable=0x7f31f78cba90, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/cpython/abstract.h:118
#8  PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f31d51c1ec0, callable=0x7f31f78cba90) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/cpython/abstract.h:127
#9  call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:5072
#10 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:3487
#11 0x00007f31f755290e in _PyEval_EvalFrame (throwflag=0, f=0x7f31d51c1d40, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/internal/pycore_ceval.h:40
#12 _PyEval_EvalCode (tstate=tstate
entry=0x684d5a0, _co=_co
entry=0x7f31d51069d0, globals=<optimized out>, locals=locals
entry=0x0, args=<optimized out>, argcount=1, kwnames=0x0, kwargs=0x7f31efd13cc0, kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7f31d5103b20, qualname=0x7f31d5107420) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:4327
#13 0x00007f31f746f441 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Objects/call.c:396
#14 0x00007f31f742b274 in do_call_core (kwdict=0x7f31d5104dc0, callargs=0x7f31efd13ca0, func=0x7f31f78c64c0, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:5120
#15 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:3580
#16 0x00007f31f7425bbb in _PyEval_EvalFrame (throwflag=0, f=0x7f31f77d4740, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/internal/pycore_ceval.h:40
#17 function_code_fastcall (tstate=0x684d5a0, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Objects/call.c:330
#18 0x00007f31f742d7d0 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f31d51707b8, callable=0x7f31e02c4160, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/cpython/abstract.h:118
#19 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f31d51707b8, callable=0x7f31e02c4160) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/cpython/abstract.h:127
#20 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:5072
#21 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:3504
#22 0x00007f31f7425bbb in _PyEval_EvalFrame (throwflag=0, f=0x7f31d5170640, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/internal/pycore_ceval.h:40
#23 function_code_fastcall (tstate=0x684d5a0, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Objects/call.c:330
#24 0x00007f31f742d7d0 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f31e02b8a78, callable=0x7f31e02c4430, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/cpython/abstract.h:118
#25 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f31e02b8a78, callable=0x7f31e02c4430) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/cpython/abstract.h:127
#26 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:5072
#27 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:3504
#28 0x00007f31f7425bbb in _PyEval_EvalFrame (throwflag=0, f=0x7f31e02b8900, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/internal/pycore_ceval.h:40
#29 function_code_fastcall (tstate=0x684d5a0, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Objects/call.c:330
#30 0x00007f31f74722ec in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7f31d50dfe68, callable=0x7f31e02c41f0, tstate=0x684d5a0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/cpython/abstract.h:118
#31 method_vectorcall (method=<optimized out>, args=0x7f31f7914058, nargsf=<optimized out>, kwnames=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Objects/classobject.c:61
#32 0x00007f31f75f93a9 in t_bootstrap (boot_raw=0x7f31d5109720) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Modules/_threadmodule.c:1040
#33 0x00007f31f75a0bd7 in pythread_wrapper (arg=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/thread_pthread.h:236
#34 0x00007f31f6f71ea5 in start_thread () from /lib64/libpthread.so.0
#35 0x00007f31f6591b0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f31f7950740 (LWP 5372) "python3"):
#0  0x00007f31f6558659 in waitpid () from /lib64/libc.so.6
#1  0x00007f31f64d5f62 in do_system () from /lib64/libc.so.6
#2  0x00007f31f64d6311 in system () from /lib64/libc.so.6
#3  0x00007f31ef98187c in TUnixSystem::StackTrace() () from /cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCore.so
#4  0x00007f31efbe8dd3 in (anonymous namespace)::TExceptionHandlerImp::HandleException(int) () from /cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libcppyy_backend3_9.so
#5  0x00007f31ef97ed01 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libCore.so
#6  <signal handler called>
#7  0x00007f31f64c9387 in raise () from /lib64/libc.so.6
#8  0x00007f31f64caa78 in abort () from /lib64/libc.so.6
#9  0x00007f31f650bf67 in __libc_message () from /lib64/libc.so.6
#10 0x00007f31f6514329 in _int_free () from /lib64/libc.so.6
#11 0x00007f31d42307ef in TMVA::CrossValidation::~CrossValidation() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.24.06-3455f/x86_64-centos7-gcc11-opt/lib/libTMVA.so
#12 0x00007f31d4230c79 in TMVA::CrossValidation::~CrossValidation() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.24.06-3455f/x86_64-centos7-gcc11-opt/lib/libTMVA.so
#13 0x00007f31efc60c39 in CPyCppyy::op_dealloc_nofree(CPyCppyy::CPPInstance*) () from /cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libcppyy3_9.so
#14 0x00007f31efc60d51 in CPyCppyy::op_dealloc(CPyCppyy::CPPInstance*) () from /cvmfs/sft.cern.ch/lcg/views/LCG_101_LHCB_7/x86_64-centos7-gcc11-opt/lib/libcppyy3_9.so
#15 0x00007f31f74cad2f in subtype_dealloc (self=0x7f31d48afec0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Objects/typeobject.c:1337
#16 0x00007f31f7425e05 in _Py_DECREF (op=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/object.h:430
#17 frame_dealloc (f=0x3492030) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Objects/frameobject.c:582
#18 0x00007f31f7425c58 in _Py_DECREF (op=0x3492030) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/object.h:430
#19 function_code_fastcall (tstate=0x1841160, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Objects/call.c:338
#20 0x00007f31f742c273 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x18aadf0, callable=0x7f31d5110670, tstate=0x1841160) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/cpython/abstract.h:118
#21 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x18aadf0, callable=0x7f31d5110670) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/cpython/abstract.h:127
#22 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x1841160) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:5072
#23 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:3518
#24 0x00007f31f755290e in _PyEval_EvalFrame (throwflag=0, f=0x18aac80, tstate=0x1841160) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Include/internal/pycore_ceval.h:40
#25 _PyEval_EvalCode (tstate=0x1841160, _co=_co
entry=0x7f31efd277c0, globals=globals
entry=0x7f31f77d7840, locals=locals
entry=0x7f31f77d7840, args=args
entry=0x0, argcount=argcount
entry=0, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:4327
#26 0x00007f31f7552c4e in _PyEval_EvalCodeWithName (_co=_co
entry=0x7f31efd277c0, globals=globals
entry=0x7f31f77d7840, locals=locals
entry=0x7f31f77d7840, args=args
entry=0x0, argcount=argcount
entry=0, kwnames=kwnames
entry=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:4359
#27 0x00007f31f7552c9b in PyEval_EvalCodeEx (_co=_co
entry=0x7f31efd277c0, globals=globals
entry=0x7f31f77d7840, locals=locals
entry=0x7f31f77d7840, args=args
entry=0x0, argcount=argcount
entry=0, kws=kws
entry=0x0, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:4375
#28 0x00007f31f7552ccb in PyEval_EvalCode (co=co
entry=0x7f31efd277c0, globals=globals
entry=0x7f31f77d7840, locals=locals
entry=0x7f31f77d7840) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/ceval.c:826
#29 0x00007f31f7592d6e in run_eval_code_obj (locals=0x7f31f77d7840, globals=0x7f31f77d7840, co=0x7f31efd277c0, tstate=0x1841160) at /build/jenkins/workspace/lcg_release_pipeline/build/externals/Python-3.9.6/src/Python/3.9.6/Python/pythonrun.c:1219
#30 run_mod (mod=<optimized out>, filename=filename
...
2

Can you run with valgrind, see Valgrind and ROOT - and post the output?

3

Thank you for your reply!

Please find attached the output of valgrind (BTW I’m using LCG_102 with x86_64-centos7-gcc8-opt)

valgrind.txt (1.0 MB)

4

OK that did it - thanks, @tpajero . This is the culprit:

==4850== Invalid write of size 8
==4850==    at 0x3147EFCC: TMVA::CvSplitKFoldsExpr::Eval(unsigned int, TMVA::Event const*) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.26.04-edd28/x86_64-centos7-gcc8-opt/lib/libTMVA.so)
==4850==    by 0x31480E00: TMVA::CvSplitKFolds::SplitSets(std::vector<TMVA::Event*, std::allocator<TMVA::Event*> >&, unsigned int, unsigned int) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.26.04-edd28/x86_64-centos7-gcc8-opt/lib/libTMVA.so)
==4850==    by 0x314818D5: TMVA::CvSplitKFolds::MakeKFoldDataSet(TMVA::DataSetInfo&) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.26.04-edd28/x86_64-centos7-gcc8-opt/lib/libTMVA.so)
==4850==    by 0x31479B36: TMVA::CrossValidation::Evaluate() (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.26.04-edd28/x86_64-centos7-gcc8-opt/lib/libTMVA.so)

==4850==  Address 0x2ae12b68 is 8 bytes before a block of size 8 alloc'd
==4850==    at 0x4C2BE3B: operator new(unsigned long) (vg_replace_malloc.c:422)
==4850==    by 0x31480498: TMVA::CvSplitKFoldsExpr::CvSplitKFoldsExpr(TMVA::DataSetInfo&, TString) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.26.04-edd28/x86_64-centos7-gcc8-opt/lib/libTMVA.so)
==4850==    by 0x314816DE: TMVA::CvSplitKFolds::MakeKFoldDataSet(TMVA::DataSetInfo&) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.26.04-edd28/x86_64-centos7-gcc8-opt/lib/libTMVA.so)
==4850==    by 0x31479B36: TMVA::CrossValidation::Evaluate() (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.26.04-edd28/x86_64-centos7-gcc8-opt/lib/libTMVA.so)

This might be a bug in the cross validation. @moneta will be able to cross check - but he is only back next week. Can you ping us then, please? (We currently really don’t have anyone else who will be able to do this…)

5

Hi @Axel thank you for looking into this!
Sure thing, I’ll ping Lorenzo next week

6

Hi,
Have you tried running with the latest ROOT version? I don’t have this error on the master ROOT version

Lorenzo

7

Hi @moneta,

I’ve tried with ROOT v6.26.06 (source /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.26.06/x86_64-centos8-gcc85-opt/bin/thisroot.sh on lxplus8.cern.ch) and the error is still there. But I can try to install locally on my Mac Arm the master version and try with that if you think that the bug has been fixed meanwhile.

Tommaso

8

Hi,
If it fails in 6.26.06, the problem should be also in the master. I will investigate running on a Linux machine.

Lorenzo

9

I have open a PR fixing this problem, see [tmva] Fix maximum value for indexing the NumFolds parameter by lmoneta · Pull Request #11267 · root-project/root · GitHub

There is a simple workaround for you to avoid this bug. In the constructor of the CrossValidation, as option, define the split expression passing “NumFolds” as parameter instead of its actual value (2):

'SplitExpr=(int(fabs([eventNumber]))%int([NumFolds]))'

Cheers

Lorenzo

10

Many thanks for your prompt help @moneta!

Best,

Tommaso