Processing and merging TTrees on the fly with PyROOT

I wanted to continue this topic (https://root-forum.cern.ch/t/merging-ttrees-on-the-fly-with-ttree-mergetrees-in-pyroot/), but haven’t found a way to open it.

I’ve got a problem when I’m trying to process TTrees and merge the results using TTree.MergeTrees() function.
Here are the example code and test data:
DatasetConvert.py (2.1 KB)
test.root (492.9 KB)

The code is

import os
from ROOT import TFile, TTree, TList, TLorentzVector, TMath
from array import array

def ConvertTree(inputTree, datasetWeight=1):
	
	outputTree = TTree('TMVA_input', '')

	mJJ = array('d',[0])
	deltaYJJ = array('d',[0])
	deltaYJPh = array('d',[0])
	deltaRJJ = array('d',[0])

	outputTree.Branch("mJJ", mJJ, "mJJ/D")
	outputTree.Branch("deltaYJJ", deltaYJJ, "deltaYJJ/D")
	outputTree.Branch("deltaYJPh", deltaYJPh, "deltaYJPh/D")
	outputTree.Branch("deltaRJJ", deltaRJJ, "deltaRJJ/D")

	print '{0} entries in a tree'.format(inputTree.GetEntriesFast())
	for eventIdx, event in enumerate(inputTree):

		if event.n_jet >= 2:

			metTlv = TLorentzVector()
			metTlv.SetPtEtaPhiE(event.metTST_pt, 0, event.metTST_phi, event.metTST_pt)
			phTlv = TLorentzVector()
			phTlv.SetPtEtaPhiM(event.ph_pt, event.ph_eta, event.ph_phi, 0)
			leadJetTlv = TLorentzVector()
			leadJetTlv.SetPtEtaPhiE(event.jet_lead_pt, event.jet_lead_eta, event.jet_lead_phi, event.jet_lead_E)
			subleadJetTlv = TLorentzVector()
			subleadJetTlv.SetPtEtaPhiE(event.jet_sublead_pt, event.jet_sublead_eta, event.jet_sublead_phi, event.jet_sublead_E)

			mJJ[0] = (leadJetTlv + subleadJetTlv).M()
			deltaYJJ[0] = TMath.Abs(leadJetTlv.Rapidity() - subleadJetTlv.Rapidity())
			deltaYJPh[0] = TMath.Abs(leadJetTlv.Rapidity() - phTlv.Rapidity())
			deltaRJJ[0] = leadJetTlv.DeltaR(subleadJetTlv)
			outputTree.Fill()
	return outputTree

pathList = ('test.root', ) 

outputFile = TFile('output.root', 'recreate')

treeList = TList()
pyInputFileList = list()
pyOutputTreeList = list()
pyInputTreeList = list()

for path in pathList:
	print path
	inputFile = TFile(path, 'read')
	inputTree = inputFile.Get('output_tree')
	outputFile.cd()
	outputTree = ConvertTree(inputTree)
	treeList.Add(outputTree)
	pyInputFileList.append(inputFile)
	pyOutputTreeList.append(outputTree)
	pyInputTreeList.append(inputTree)

print 'cycle ended'
outputFile.cd()
print 'cd() to output file'
outputTree = TTree.MergeTrees(treeList)
print 'merged trees'
outputFile.Write()
print 'file written'
outputFile.Close()
print 'file closed'

I’m running it on the lxplus machines with the following commands:

setupATLAS
lsetup "root 6.14.04-x86_64-slc6-gcc62-opt"
python DatasetConvert.py

and the output I get is

test.root
5000 entries in a tree
cycle ended
cd() to output file

after that everything freezes and the terminal stops responding to any commands.

What could be the problem here?
Thanks in advance.

Hi @apetukho

Could it be that it is just taking some time to do the merge? What happens if you run it with a smaller list (e.g. two trees) to see if it finishes?

Hi, @etejedor

I’m running it on 1 tree with 5000 events and it could stay like that for more than 10 minutes, so I don’t think that’s the case.
What’s more interesting, I went to test it again (same script, same input file, same setup) and got completely different (thought bad) results:

test.root
5000 entries in a tree
cycle ended
cd() to output file
merged trees
*** glibc detected *** python: malloc(): smallbin double linked list corrupted: 0x000000000548de50 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75e5e)[0x7fdaf989be5e]
/lib64/libc.so.6(+0x7a528)[0x7fdaf98a0528]
/lib64/libc.so.6(__libc_malloc+0x5c)[0x7fdaf98a0b1c]
/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64/libstdc++.so.6(_Znwm+0x18)[0x7fdaef34ba08]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libCling.so(+0x4557e2)[0x7fdae92f07e2]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libCore.so(_ZN9TFunction19GetListOfMethodArgsEv+0x39)[0x7fdaef93c229]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN5Cppyy16GetMethodArgTypeB5cxx11Eli+0x4a)[0x7fdaf32d718a]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN6PyROOT13TMethodHolder11GetPriorityEv+0x8d)[0x7fdaf3300e2d]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(+0x63cc7)[0x7fdaf32e3cc7]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZSt16__merge_adaptiveIN9__gnu_cxx17__normal_iteratorIPPN6PyROOT10PyCallableESt6vectorIS4_SaIS4_E                                              EEElS5_NS0_5__ops15_Iter_comp_iterIPFiS4_S4_EEEEvT_SF_SF_T0_SG_T1_SG_T2_+0x8a)[0x7fdaf32e7daa]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZSt22__stable_sort_adaptiveIN9__gnu_cxx17__normal_iteratorIPPN6PyROOT10PyCallableESt6vectorIS4_S                                              aIS4_EEEES5_lNS0_5__ops15_Iter_comp_iterIPFiS4_S4_EEEEvT_SF_T0_T1_T2_+0x97)[0x7fdaf32e83c7]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(+0x659ab)[0x7fdaf32e59ab]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x43)[0x7fdafa4b6af3]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5bab)[0x7fdafa56e02b]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc)[0x7fdafa5723ec]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x19)[0x7fdafa5724e9]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0x8a)[0x7fdafa5968ba]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xd5)[0x7fdafa597c95]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(Py_Main+0xc61)[0x7fdafa5ae3d1]
/lib64/libc.so.6(__libc_start_main+0x100)[0x7fdaf9844d20]
python[0x400641]
======= Memory map: ========
00400000-00401000 r-xp 00000000 00:2d 729050740                          /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/python2.7
00600000-00601000 r--p 00000000 00:2d 729050740                          /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/python2.7
00601000-00602000 rw-p 00001000 00:2d 729050740                          /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/python2.7
01b89000-0561f000 rw-p 00000000 00:00 0                                  [heap]
7fdad0000000-7fdad0021000 rw-p 00000000 00:00 0
7fdad0021000-7fdad4000000 ---p 00000000 00:00 0
7fdad46c5000-7fdad8000000 rw-p 00000000 00:00 0
7fdad8000000-7fdad8045000 rw-p 00000000 00:00 0
7fdad8045000-7fdadc000000 ---p 00000000 00:00 0
7fdadcb92000-7fdadcb9c000 r-xp 00000000 00:2d 729049084                  /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7fdadcb9c000-7fdadcd9b000 ---p 0000a000 00:2d 729049084                  /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7fdadcd9b000-7fdadcd9c000 r--p 00009000 00:2d 729049084                  /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7fdadcd9c000-7fdadcd9e000 rw-p 0000a000 00:2d 729049084                  /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7fdadcd9e000-7fdadcde5000 r-xp 00000000 00:2d 724557331                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7fdadcde5000-7fdadcfe5000 ---p 00047000 00:2d 724557331                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7fdadcfe5000-7fdadcfe8000 r--p 00047000 00:2d 724557331                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7fdadcfe8000-7fdadcfe9000 rw-p 0004a000 00:2d 724557331                  /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7fdadcfe9000-7fdadcfea000 rw-p 00000000 00:00 0
7fdadcfea000-7fdadcfeb000 ---p 00000000 00:00 0
7fdadcfeb000-7fdadd9eb000 rw-p 00000000 00:00 0
7fdadd9eb000-7fdadd9ef000 r-xp 00000000 00:2d 729049179                  /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7fdadd9ef000-7fdaddbee000 ---p 00004000 00:2d 729049179                  /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7fdaddbee000-7fdaddbef000 r--p 00003000 00:2d 729049179                  /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7fdaddbef000-7fdaddbf1000 rw-p 00004000 00:2d 729049179                  /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7fdaddbf1000-7fdaddc31000 rw-p 00000000 00:00 0
7fdaddc31000-7fdaddc36000 r-xp 00000000 fc:01 40233                      /usr/lib64/libXfixes.so.3.1.0
7fdaddc36000-7fdadde35000 ---p 00005000 fc:01 40233                      /usr/lib64/libXfixes.so.3.1.0
7fdadde35000-7fdadde36000 rw-p 00004000 fc:01 40233                      /usr/lib64/libXfixes.so.3.1.0Aborted (core dumped)

I do not see anything wrong with the code a priori, perhaps @pcanal ?

In the previous post you said that if you do it in two steps (first you write the trees in one script, then you merge in another) then it works. Is this still the case? is that an option?

Also in my previous post I proposed a simplified solution to do everything in one go that works. Did you find out what is the difference that makes it crash your case with respect to the proposed solution?

Here’s what I’ve got starting from the simplest working version of the script.

The last working version of the script is:

from ROOT import TFile, TTree, TList, TLorentzVector
from array import array

def ConvertTree(tree):
        outputTree = TTree('TMVA_input', '')

        metTST = array('d', [0])

        outputTree.Branch("metTST", metTST, "metTST/D")

        for event in tree:
                if event.n_jet >= 2:
                        metTlv = TLorentzVector()
                        metTlv.SetPtEtaPhiE(event.metTST_pt, 0, event.metTST_phi, event.metTST_pt)
                        metTST = metTlv.Pt()
                        outputTree.Fill()
        return outputTree

for i in range(50):
        print i
        pathList = ['test.root',]
        treeList = TList()
        outputFile = TFile('MergeTest.root', 'recreate')
        pyfilelist = []
        pytreelist = []

        for path in pathList:
                print("Path", path)
                inputFile = TFile(path, 'read')
                pyfilelist.append(inputFile) # Make this TFile survive the loop!
                inputTree = inputFile.Get('output_tree')
                pytreelist.append(inputTree) # Make this TTree survive the loop!
                outputFile.cd()
                outputTree = ConvertTree(inputTree) #instead of extensive processing
                treeList.Add(outputTree)

        outputFile.cd()
        outputTree = TTree.MergeTrees(treeList)
        outputFile.Write()
        outputFile.Close()

But when I add another TLorentzVector phTLV:

from ROOT import TFile, TTree, TList, TLorentzVector
from array import array

def ConvertTree(tree):
        outputTree = TTree('TMVA_input', '')

        metTST = array('d', [0])

        outputTree.Branch("metTST", metTST, "metTST/D")

        for event in tree:
                if event.n_jet >= 2:
                        metTlv = TLorentzVector()
                        metTlv.SetPtEtaPhiE(event.metTST_pt, 0, event.metTST_phi, event.metTST_pt)
                        phTlv = TLorentzVector() #NEW
                        phTlv.SetPtEtaPhiM(event.ph_pt, event.ph_eta, event.ph_phi, 0) #NEW
                      
                        metTST = metTlv.Pt()
                        outputTree.Fill()
        return outputTree

for i in range(50):
        print i
        pathList = ['test.root',]
        treeList = TList()
        outputFile = TFile('MergeTest.root', 'recreate')
        pyfilelist = []
        pytreelist = []

        for path in pathList:
                print("Path", path)
                inputFile = TFile(path, 'read')
                pyfilelist.append(inputFile) # Make this TFile survive the loop!
                inputTree = inputFile.Get('output_tree')
                pytreelist.append(inputTree) # Make this TTree survive the loop!
                outputFile.cd()
                outputTree = ConvertTree(inputTree) #instead of extensive processing
                treeList.Add(outputTree)

        outputFile.cd()
        outputTree = TTree.MergeTrees(treeList)
        outputFile.Write()
        outputFile.Close()

it starts to crash sometimes (1-2 times per 100 iterations) with an error like the following

*** glibc detected *** python: malloc(): smallbin double linked list corrupted: 0x000000000556f4d0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75e5e)[0x7f4b6c52ae5e]
/lib64/libc.so.6(+0x7a528)[0x7f4b6c52f528]
/lib64/libc.so.6(__libc_malloc+0x5c)[0x7f4b6c52fb1c]
/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64/libstdc++.so.6(_Znwm+0x18)[0x7f4b61fdaa08]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libTree.so(_ZN7TBranch8StreamerER7TBuffer+0x5c)[0x7f4b64f5ba4c]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN11TBufferFile16WriteObjectClassEPKvPK6TClassb+0x12c)[0x7f4b6363b1ac]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN9TBufferIO14WriteObjectAnyEPKvPK6TClassb+0x126)[0x7f4b6363fd36]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libCore.so(_ZN9TObjArray8StreamerER7TBuffer+0xd8)[0x7f4b6257ad78]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN11TBufferFile14WriteFastArrayEPvPK6TClassiP15TMemberStreamer+0x8c)[0x7f4b6363582c]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN13TStreamerInfo14WriteBufferAuxIPPcEEiR7TBufferRKT_PKPNS_9TCompInfoEiiiii+0x4d88)[0x7f4b638a90b8]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN20TStreamerInfoActions18GenericWriteActionER7TBufferPvPKNS_14TConfigurationE+0x3d)[0x7f4b636f8bfd]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN11TBufferFile16WriteClassBufferEPK6TClassPv+0x196)[0x7f4b6363bb56]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libTree.so(_ZN5TTree8StreamerER7TBuffer+0x5e4)[0x7f4b64fc44a4]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN4TKeyC2EPK7TObjectPKciP10TDirectory+0x1b8)[0x7f4b636d3378]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN5TFile9CreateKeyEP10TDirectoryPK7TObjectPKci+0x35)[0x7f4b6367cb35]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN14TDirectoryFile12WriteTObjectEPK7TObjectPKcS4_i+0x202)[0x7f4b63674902]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libCore.so(_ZNK7TObject5WriteEPKcii+0x9d)[0x7f4b624fca7d]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN14TDirectoryFile5WriteEPKcii+0xd1)[0x7f4b63672a41]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libRIO.so(_ZN5TFile5WriteEPKcii+0x73)[0x7f4b6368c483]
[0x7f4b6d611052]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_Z8FastCalllPvS_S_+0x701)[0x7f4b65f6abf1]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN5Cppyy5CallIElPvS0_+0x1f)[0x7f4b65f6b83f]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN6PyROOT12TIntExecutor7ExecuteElPvPNS_12TCallContextE+0x2f)[0x7f4b65f6e2ff]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN6PyROOT13TMethodHolder8CallSafeEPvlPNS_12TCallContextE+0x72)[0x7f4b65f92782]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN6PyROOT13TMethodHolder4CallERPNS_11ObjectProxyEP7_objectS5_PNS_12TCallContextE+0x134)[0x7f4b65f91b24]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(+0x65412)[0x7f4b65f74412]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x43)[0x7f4b6d145af3]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5bab)[0x7f4b6d1fd02b]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc)[0x7f4b6d2013ec]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x19)[0x7f4b6d2014e9]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0x8a)[0x7f4b6d2258ba]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xd5)[0x7f4b6d226c95]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(Py_Main+0xc61)[0x7f4b6d23d3d1]
/lib64/libc.so.6(__libc_start_main+0x100)[0x7f4b6c4d3d20]
python[0x400641]
======= Memory map: ========
00400000-00401000 r-xp 00000000 00:1c 682027                             /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/python2.7
00600000-00601000 r--p 00000000 00:1c 682027                             /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/python2.7
00601000-00602000 rw-p 00001000 00:1c 682027                             /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/python2.7
01cf1000-09058000 rw-p 00000000 00:00 0                                  [heap]
7f4b40000000-7f4b40021000 rw-p 00000000 00:00 0
7f4b40021000-7f4b44000000 ---p 00000000 00:00 0
7f4b48000000-7f4b48045000 rw-p 00000000 00:00 0
7f4b48045000-7f4b4c000000 ---p 00000000 00:00 0
7f4b4f862000-7f4b4f86c000 r-xp 00000000 00:1c 26262956                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7f4b4f86c000-7f4b4fa6b000 ---p 0000a000 00:1c 26262956                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7f4b4fa6b000-7f4b4fa6c000 r--p 00009000 00:1c 26262956                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7f4b4fa6c000-7f4b4fa6e000 rw-p 0000a000 00:1c 26262956                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7f4b4fa6e000-7f4b4fab5000 r-xp 00000000 00:1c 68131                      /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7f4b4fab5000-7f4b4fcb5000 ---p 00047000 00:1c 68131                      /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7f4b4fcb5000-7f4b4fcb8000 r--p 00047000 00:1c 68131                      /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7f4b4fcb8000-7f4b4fcb9000 rw-p 0004a000 00:1c 68131                      /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7f4b4fcb9000-7f4b4fcba000 rw-p 00000000 00:00 0
7f4b4fcba000-7f4b4fcbb000 ---p 00000000 00:00 0
7f4b4fcbb000-7f4b506bb000 rw-p 00000000 00:00 0
7f4b506bb000-7f4b506bf000 r-xp 00000000 00:1c 17350810                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7f4b506bf000-7f4b508be000 ---p 00004000 00:1c 17350810                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7f4b508be000-7f4b508bf000 r--p 00003000 00:1c 17350810                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7f4b508bf000-7f4b508c1000 rw-p 00004000 00:1c 17350810                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7f4b508c1000-7f4b50901000 rw-p 00000000 00:00 0
7f4b50901000-7f4b50906000 r-xp 00000000 fc:01 40792                      /usr/lib64/libXfixes.so.3.1.0
7f4b50906000-7f4b50b05000 ---p 00005000 fc:01 40792                      /usr/lib64/libXfixes.so.3.1.0
7f4b50b05000-7f4b50b06000 rw-p 00004000 fc:01 40792                      /usr/lib64/libXfixes.so.3.1.0
7f4b50b06000-7f4b50b0f000 r-xp 00000000 fc:01 40794                      /usr/lib64/libXcursor.so.1.0.2
7f4b50b0f000-7f4b50d0f000 ---p 00009000 fc:01 40794                      /usr/lib64/libXcursor.so.1.0.2
7f4b50d0f000-7f4b50d10000 rw-p 00009000 fc:01 40794                      /usr/lib64/libXcursor.so.1.0.2
7f4b50d10000-7f4b50d66000 r--p 00000000 00:1c 68426                      /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/fonts/FreeSansBold.otf
7f4b50d66000-7f4b50d98000 r-xp 00000000 00:1c 420384                     /cvmfs/sft.cern.ch/lcg/releases/expat/2.2.5-fe6d4/x86_64-slc6-gcc62-opt/lib/libexpat.so.1.6.7
7f4b50d98000-7f4b50f98000 ---p 00032000 00:1c 420384                     /cvmfs/sft.cern.ch/lcg/releases/expat/2.2.5-fe6d4/x86_64-slc6-gcc62-opt/lib/libexpat.so.1.6.7
7f4b50f98000-7f4b50f9a000 r--p 00032000 00:1c 420384                     /cvmfs/sft.cern.ch/lcg/releases/expat/2.2.5-fe6d4/x86_64-slc6-gcc62-opt/lib/libexpat.so.1.6.7
7f4b50f9a000-7f4b50f9b000 rw-p 00034000 00:1c 420384                     /cvmfs/sft.cern.ch/lcg/releases/expat/2.2.5-fe6d4/x86_64-slc6-gcc62-opt/lib/libexpat.so.1.6.7
7f4b50f9b000-7f4b50fa4000 r-xp 00000000 fc:01 20176                      /usr/lib64/libXrender.so.1.3.0
7f4b50fa4000-7f4b511a3000 ---p 00009000 fc:01 20176                      /usr/lib64/libXrender.so.1.3.0
7f4b511a3000-7f4b511a4000 rw-p 00008000 fc:01 20176                      /usr/lib64/libXrender.so.1.3.0
7f4b511a4000-7f4b511e2000 r-xp 00000000 00:1c 411206                     /cvmfs/sft.cern.ch/lcg/releases/fontconfig/2.12.6-63041/x86_64-slc6-gcc62-opt/lib/libfontconfig.so.1.10.1
7f4b511e2000-7f4b513e1000 ---p 0003e000 00:1c 411206                     /cvmfs/sft.cern.ch/lcg/releases/fonAborted (core dumped)

Doing it in two steps might be an option, but doing it on the fly seems like a much simpler and cleaner solution.

Ok let’s start with the first code that crashes.

It just crashes because of the fact of creating a new TLorenzVector? It is not filled into any branch, so it really has no effect.

This should be metTST[0] = metTlv.Pt()

That’s right. Fixed it, going to do intensive tests tomorrow.
For now, there’s a new strange error caused by running this code in a loop. It that might not be connected to the main one, but it still puzzles me.
So, this code works perfectly fine

from ROOT import TFile, TTree, TList

def ConvertTree(tree):
	outputTree = inputTree.CloneTree() 
	return outputTree

for i in range(30):
	print i
	pathList = ['test.root',]
	treeList = TList()
	outputFile = TFile('MergeTest.root', 'recreate')
	pyfilelist = []
	pytreelist = []


	for path in pathList:
	        print("Path", path)
	        inputFile = TFile(path, 'read')
	        pyfilelist.append(inputFile) # Make this TFile survive the loop!
	        inputTree = inputFile.Get('output_tree')
	        pytreelist.append(inputTree) # Make this TTree survive the loop!
	        outputFile.cd()
	        outputTree = ConvertTree(inputTree) #instead of extensive processing
	        treeList.Add(outputTree)

	outputFile.cd()
	outputTree = TTree.MergeTrees(treeList)
	outputFile.Write()
	outputFile.Close()

but this one runs fine only if number of iterations is equal to 1.

from ROOT import TFile, TTree, TList
from array import array

def ConvertTree(tree):
        outputTree = TTree('TMVA_input', '')
        metTST = array('d', [0])
        outputTree.Branch("metTST", metTST, "metTST/D")

        for event in tree:
                outputTree.Fill()
        return outputTree

for i in range(2):
        print i
        pathList = ['test.root',]
        treeList = TList()
        outputFile = TFile('MergeTest.root', 'recreate')
        pyfilelist = []
        pytreelist = []

        for path in pathList:
                print("Path", path)
                inputFile = TFile(path, 'read')
                pyfilelist.append(inputFile) # Make this TFile survive the loop!
                inputTree = inputFile.Get('output_tree')
                pytreelist.append(inputTree) # Make this TTree survive the loop!
                outputFile.cd()
                outputTree = ConvertTree(inputTree) #instead of extensive processing
                print 'Got output tree'
                treeList.Add(outputTree)
                print 'Added output tree to TList'

        print 'cycle ended'
        outputFile.cd()
        print 'cd() to the output file'
        outputTree = TTree.MergeTrees(treeList)
        print 'Trees merged'
        outputFile.Write()
        print 'File Written'
        outputFile.Close()
        print 'File closed'

If it is anything more, than one, the script fails with the following output.

0
('Path', 'test.root')
Got output tree
Added output tree to TList
cycle ended
cd() to the output file
Trees merged
File Written
File closed
1
('Path', 'test.root')
Got output tree

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================

Thread 2 (Thread 0x7ff2390ff700 (LWP 10684)):
#0  0x00007ff255926a00 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007ff255c7e2c8 in PyThread_acquire_lock () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#2  0x00007ff255c3b7c6 in PyEval_RestoreThread () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:359
#3  0x00007ff239101d68 in time_sleep () from /cvmfs/sft.cern.ch/lcg/releases/LCG_94/Python/2.7.15/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
#4  0x00007ff255c44064 in PyEval_EvalFrameEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4372
#5  0x00007ff255c463ec in PyEval_EvalCodeEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3604
#6  0x00007ff255bbbddd in function_call () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#7  0x00007ff255b8aaf3 in PyObject_Call () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#8  0x00007ff255c40f67 in PyEval_EvalFrameEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4686
#9  0x00007ff255c423cf in PyEval_EvalFrameEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4457
#10 0x00007ff255c423cf in PyEval_EvalFrameEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4457
#11 0x00007ff255c463ec in PyEval_EvalCodeEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3604
#12 0x00007ff255bbbd0c in function_call () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#13 0x00007ff255b8aaf3 in PyObject_Call () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#14 0x00007ff255b9bfbc in instancemethod_call () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Objects/classobject.c:2600
#15 0x00007ff255b8aaf3 in PyObject_Call () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#16 0x00007ff255c3bea7 in PyEval_CallObjectWithKeywords () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4241
#17 0x00007ff255c841d2 in t_bootstrap () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#18 0x00007ff255920aa1 in start_thread () from /lib64/libpthread.so.0
#19 0x00007ff254fe2c4d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ff25613e700 (LWP 10626)):
#0  0x00007ff254fa68dd in waitpid () from /lib64/libc.so.6
#1  0x00007ff254f384e9 in do_system () from /lib64/libc.so.6
#2  0x00007ff254f38820 in system () from /lib64/libc.so.6
#3  0x00007ff24b050ebd in TUnixSystem::StackTrace() () from /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libCore.so
#4  0x00007ff24b053624 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libCore.so
#5  <signal handler called>
#6  0x00007ff24e9d7615 in PyROOT::TMethodHolder::ConvertAndSetArgs(_object*, PyROOT::TCallContext*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so
#7  0x00007ff24e9d6a6e in PyROOT::TMethodHolder::Call(PyROOT::ObjectProxy*&, _object*, _object*, PyROOT::TCallContext*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so
#8  0x00007ff24e9b9412 in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so
#9  0x00007ff255b8aaf3 in PyObject_Call () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#10 0x00007ff255c4202b in PyEval_EvalFrameEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4589
#11 0x00007ff255c463ec in PyEval_EvalCodeEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3604
#12 0x00007ff255c464e9 in PyEval_EvalCode () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:669
#13 0x00007ff255c6a8ba in PyRun_FileExFlags () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#14 0x00007ff255c6bc95 in PyRun_SimpleFileExFlags () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#15 0x00007ff255c823d1 in Py_Main () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#16 0x00007ff254f18d20 in __libc_start_main () from /lib64/libc.so.6
#17 0x0000000000400641 in _start ()
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#6  0x00007ff24e9d7615 in PyROOT::TMethodHolder::ConvertAndSetArgs(_object*, PyROOT::TCallContext*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so
#7  0x00007ff24e9d6a6e in PyROOT::TMethodHolder::Call(PyROOT::ObjectProxy*&, _object*, _object*, PyROOT::TCallContext*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so
#8  0x00007ff24e9b9412 in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so
#9  0x00007ff255b8aaf3 in PyObject_Call () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#10 0x00007ff255c4202b in PyEval_EvalFrameEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:4589
#11 0x00007ff255c463ec in PyEval_EvalCodeEx () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:3604
#12 0x00007ff255c464e9 in PyEval_EvalCode () at /mnt/build/jenkins/workspace/lcg_release_latest/BUILDTYPE/Release/COMPILER/gcc62binutils/LABEL/slc6/build/externals/Python-2.7.15/src/Python/2.7.15/Python/ceval.c:669
#13 0x00007ff255c6a8ba in PyRun_FileExFlags () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#14 0x00007ff255c6bc95 in PyRun_SimpleFileExFlags () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#15 0x00007ff255c823d1 in Py_Main () from /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0
#16 0x00007ff254f18d20 in __libc_start_main () from /lib64/libc.so.6
#17 0x0000000000400641 in _start ()
===========================================================

Do you know what is the line that makes it crash the second iteration? We are talking about the outer loop right?

So if I understand correctly, you want to invoke MergeTrees N times on N lists of K input trees to generate N output trees in N different files?

Yes.

It happens on the 2 iteration of the outer loop (for i in range(2)) on the treeList.Add(outputTree) line.

I have tested the code once again, this time with only 1 iteration.
The last working version is

from ROOT import TFile, TTree, TList, TLorentzVector
from array import array

def ConvertTree(tree):
        outputTree = TTree('TMVA_input', '')

        metTST = array('d', [0])

        outputTree.Branch("metTST", metTST, "metTST/D")

        for event in tree:
                if event.n_jet >= 2:
                        metTlv = TLorentzVector()
                        metTlv.SetPtEtaPhiE(event.metTST_pt, 0, event.metTST_phi, event.metTST_pt)
                        phTlv = TLorentzVector()
                        phTlv.SetPtEtaPhiM(event.ph_pt, event.ph_eta, event.ph_phi, 0)
                        leadJetTlv = TLorentzVector()
                        leadJetTlv.SetPtEtaPhiE(event.jet_lead_pt, event.jet_lead_eta, event.jet_lead_phi, event.jet_lead_E)
                        subleadJetTlv = TLorentzVector()
                        subleadJetTlv.SetPtEtaPhiE(event.jet_sublead_pt, event.jet_sublead_eta, event.jet_sublead_phi, event.jet_sublead_E)

                        metTST[0] = metTlv.Pt()
                        outputTree.Fill()
        return outputTree

for i in range(1):
        print i
        pathList = ['test.root',]
        treeList = TList()
        outputFile = TFile('MergeTest.root', 'recreate')
        pyfilelist = []
        pytreelist = []

        for path in pathList:
                print("Path", path)
                inputFile = TFile(path, 'read')
                pyfilelist.append(inputFile) # Make this TFile survive the loop!
                inputTree = inputFile.Get('output_tree')
                pytreelist.append(inputTree) # Make this TTree survive the loop!
                outputFile.cd()
                outputTree = ConvertTree(inputTree) #instead of extensive processing
                treeList.Add(outputTree)

        outputFile.cd()
        outputTree = TTree.MergeTrees(treeList)
        outputFile.Write()
        outputFile.Close()

But when I change the variable I’m writing to the branch to (leadJetTlv + subleadJetTlv).M()

from ROOT import TFile, TTree, TList, TLorentzVector, TMath
from array import array

def ConvertTree(tree):
        outputTree = TTree('TMVA_input', '')

        mJJ = array('d',[0])


        outputTree.Branch("mJJ", mJJ, "mJJ/D")

        for event in tree:
                if event.n_jet >= 2:
                        metTlv = TLorentzVector()
                        metTlv.SetPtEtaPhiE(event.metTST_pt, 0, event.metTST_phi, event.metTST_pt)
                        phTlv = TLorentzVector()
                        phTlv.SetPtEtaPhiM(event.ph_pt, event.ph_eta, event.ph_phi, 0)
                        leadJetTlv = TLorentzVector()
                        leadJetTlv.SetPtEtaPhiE(event.jet_lead_pt, event.jet_lead_eta, event.jet_lead_phi, event.jet_lead_E)
                        subleadJetTlv = TLorentzVector()
                        subleadJetTlv.SetPtEtaPhiE(event.jet_sublead_pt, event.jet_sublead_eta, event.jet_sublead_phi, event.jet_sublead_E)

                        mJJ[0] = (leadJetTlv + subleadJetTlv).M()
                        outputTree.Fill()
        return outputTree

for i in range(1):
        print i
        pathList = ['test.root',]
        treeList = TList()
        outputFile = TFile('MergeTest.root', 'recreate')
        pyfilelist = []
        pytreelist = []

        for path in pathList:
                print("Path", path)
                inputFile = TFile(path, 'read')
                pyfilelist.append(inputFile) # Make this TFile survive the loop!
                inputTree = inputFile.Get('output_tree')
                pytreelist.append(inputTree) # Make this TTree survive the loop!
                outputFile.cd()
                outputTree = ConvertTree(inputTree) #instead of extensive processing
                treeList.Add(outputTree)

        outputFile.cd()
        outputTree = TTree.MergeTrees(treeList)
        outputFile.Write()
        outputFile.Close()

The code starts to break sometimes on the outputTree = TTree.MergeTrees(treeList) line with the following message:

*** glibc detected *** python: malloc(): smallbin double linked list corrupted: 0x0000000005b65ff0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75e5e)[0x7fcd2b653e5e]
/lib64/libc.so.6(+0x7a528)[0x7fcd2b658528]
/lib64/libc.so.6(__libc_malloc+0x5c)[0x7fcd2b658b1c]
/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/lib64/libstdc++.so.6(_Znwm+0x18)[0x7fcd21103a08]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libTree.so(_ZN6TLeafD10SetAddressEPv+0x15c)[0x7fcd240cd29c]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libTree.so(_ZN7TBranch12ResetAddressEv+0x58)[0x7fcd2407ec28]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libTree.so(_ZN5TTree10MergeTreesEP5TListPKc+0x287)[0x7fcd240dec37]
[0x7fcd2c73a04c]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_Z8FastCalllPvS_S_+0x701)[0x7fcd25093bf1]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN5Cppyy5CallRElPvS0_+0x20)[0x7fcd250949b0]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN6PyROOT18TCppObjectExecutor7ExecuteElPvPNS_12TCallContextE+0x39)[0x7fcd25097cf9]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN6PyROOT13TMethodHolder8CallSafeEPvlPNS_12TCallContextE+0x72)[0x7fcd250bb782]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(_ZN6PyROOT13TMethodHolder7ExecuteEPvlPNS_12TCallContextE+0x29)[0x7fcd250bb1f9]
/cvmfs/sft.cern.ch/lcg/releases/LCG_94/ROOT/6.14.04/x86_64-slc6-gcc62-opt/lib/libPyROOT.so(+0x65d19)[0x7fcd2509dd19]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x43)[0x7fcd2c26eaf3]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5bab)[0x7fcd2c32602b]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc)[0x7fcd2c32a3ec]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x19)[0x7fcd2c32a4e9]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0x8a)[0x7fcd2c34e8ba]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xd5)[0x7fcd2c34fc95]
/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/../lib/libpython2.7.so.1.0(Py_Main+0xc61)[0x7fcd2c3663d1]
/lib64/libc.so.6(__libc_start_main+0x100)[0x7fcd2b5fcd20]
python[0x400641]
======= Memory map: ========
00400000-00401000 r-xp 00000000 00:31 69381916                           /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/python2.7
00600000-00601000 r--p 00000000 00:31 69381916                           /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/python2.7
00601000-00602000 rw-p 00001000 00:31 69381916                           /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/bin/python2.7
02302000-05d04000 rw-p 00000000 00:00 0                                  [heap]
7fcd00000000-7fcd00021000 rw-p 00000000 00:00 0
7fcd00021000-7fcd04000000 ---p 00000000 00:00 0
7fcd06363000-7fcd08000000 rw-p 00000000 00:00 0
7fcd08000000-7fcd08045000 rw-p 00000000 00:00 0
7fcd08045000-7fcd0c000000 ---p 00000000 00:00 0
7fcd0ccee000-7fcd0e98b000 rw-p 00000000 00:00 0
7fcd0e98b000-7fcd0e995000 r-xp 00000000 00:31 69380260                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7fcd0e995000-7fcd0eb94000 ---p 0000a000 00:31 69380260                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7fcd0eb94000-7fcd0eb95000 r--p 00009000 00:31 69380260                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7fcd0eb95000-7fcd0eb97000 rw-p 0000a000 00:31 69380260                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/array.so
7fcd0eb97000-7fcd0ebde000 r-xp 00000000 00:31 68768353                   /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7fcd0ebde000-7fcd0edde000 ---p 00047000 00:31 68768353                   /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7fcd0edde000-7fcd0ede1000 r--p 00047000 00:31 68768353                   /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7fcd0ede1000-7fcd0ede2000 rw-p 0004a000 00:31 68768353                   /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc62-opt/lib/libPhysics.so
7fcd0ede2000-7fcd0ede3000 rw-p 00000000 00:00 0
7fcd0ede3000-7fcd0ede4000 ---p 00000000 00:00 0
7fcd0ede4000-7fcd0f7e4000 rw-p 00000000 00:00 0
7fcd0f7e4000-7fcd0f7e8000 r-xp 00000000 00:31 69380355                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7fcd0f7e8000-7fcd0f9e7000 ---p 00004000 00:31 69380355                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7fcd0f9e7000-7fcd0f9e8000 r--p 00003000 00:31 69380355                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7fcd0f9e8000-7fcd0f9ea000 rw-p 00004000 00:31 69380355                   /cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-2cba0/x86_64-slc6-gcc62-opt/lib/python2.7/lib-dynload/time.so
7fcd0f9ea000-7fcd0fa2a000 rw-p 00000000 00:00 0
7fcd0fa2a000-7fcd0fa2f000 r-xp 00000000 fc:01 40792                      /usr/lib64/libXfixes.so.3.1.0
7fcd0fa2f000-7fcd0fc2e000 ---p 00005000 fc:01 40792                      /usr/lib64/libXfixes.so.3.1.0
7fcd0fc2e000-7fcd0fc2f000 rw-p 00004000 fc:01 40792                      /usr/lib64/libXfixes.so.3.1.0Aborted (core dumped)

Is this expression correct? Is the addition of two TLorentzVectors well defined from Python?

I really do not see any reason why just changing the branch you write in the output tree by an equivalent one causes a crash.

It certainly seems so. I’ve used the same environment and ROOT version on lxplus to run scripts with millions of such calculations and they’ve never crushed like that.

I can’t really tell why the change of:

                        metTST[0] = metTlv.Pt()

by

                       mJJ[0] = (leadJetTlv + subleadJetTlv).M()

makes any difference, but from the last stack trace you posted it looks like an issue in MergeTrees itself. Maybe @pcanal has some further insight?

I know it is not optimal, but just for you not to be blocked, you can always fall back to the solution where you write the trees in one script and you merge them in another.

The problem seems to be linked to object lifetime. Running with

valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp --suppressions=$ROOTSYS/etc/valgrind-root-python.supp python .... 

to pin-point the problem.

So what goes wrong with that code? Unfortunately, I’m not that experienced to make sense of the valgrind output.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Did you upload the valgrind output?