Looking up for exact location of files: OK (5 files) Looking up for exact location of files: OK (5 files) Validating files: OK (5 files) 0.4: caught exception triggered by signal '1' while processing dset:'TDSet:*', file:'root://chinmay//home/chinmay/tmp6/data/default/chinmay/0.0/chinmay-1519884325-5690/2/ProofOutFile_TestTrigger.root' - check logs for possible stacktrace - last event: 0 0.3: caught exception triggered by signal '1' while processing dset:'TDSet:*', file:'root://ubuntu//home/chinmay/tmp6/data/default/chinmay/0.3/ubuntu-1519884298-17917/2/ProofOutFile_TestTrigger.root' - check logs for possible stacktrace - last event: 0 Worker 'ubuntu.localdomain-0.3' has been removed from the active list +++ Message from top master at chinmay:1093 : marking ubuntu.localdomain:1093 (0.3) as bad +++ Reason: received kPROOF_FATAL +++ Most likely your code crashed on worker 0.3 at ubuntu.localdomain:1093. +++ Please check the session logs for error messages either using +++ the 'Show logs' button or executing +++ +++ root [] TProof::Mgr("chinmay:1093")->GetSessionLogs()->Display("0.3",0) Worker 'ubuntu.localdomain-0.4' has been removed from the active list +++ Message from top master at chinmay:1093 : marking ubuntu.localdomain:1093 (0.4) as bad +++ Reason: received kPROOF_FATAL +++ Most likely your code crashed on worker 0.4 at ubuntu.localdomain:1093. +++ Please check the session logs for error messages either using +++ the 'Show logs' button or executing +++ +++ root [] TProof::Mgr("chinmay:1093")->GetSessionLogs()->Display("0.4",0) 0.0: caught exception triggered by signal '1' while processing dset:'TDSet:*', file:'root://chinmay//home/chinmay/tmp6/data/default/chinmay/0.1/chinmay-1519884325-5707/2/ProofOutFile_TestTrigger.root' - check logs for possible stacktrace - last event: 0 0.1: caught exception triggered by signal '1' while processing dset:'TDSet:*', file:'root://chinmay//home/chinmay/tmp6/data/default/chinmay/0.2/chinmay-1519884326-5724/2/ProofOutFile_TestTrigger.root' - check logs for possible stacktrace - last event: 0 Worker 'chinmay-0.1' has been removed from the active list +++ Message from top master at chinmay:1093 : marking chinmay:1093 (0.1) as bad +++ Reason: received kPROOF_FATAL +++ Most likely your code crashed on worker 0.1 at chinmay:1093. +++ Please check the session logs for error messages either using +++ the 'Show logs' button or executing +++ +++ root [] TProof::Mgr("chinmay:1093")->GetSessionLogs()->Display("0.1",0) Worker 'chinmay-0.0' has been removed from the active list +++ Message from top master at chinmay:1093 : marking chinmay:1093 (0.0) as bad +++ Reason: received kPROOF_FATAL +++ Most likely your code crashed on worker 0.0 at chinmay:1093. +++ Please check the session logs for error messages either using +++ the 'Show logs' button or executing +++ +++ root [] TProof::Mgr("chinmay:1093")->GetSessionLogs()->Display("0.0",0) 0.2: caught exception triggered by signal '1' while processing dset:'TDSet:*', file:'root://ubuntu//home/chinmay/tmp6/data/default/chinmay/0.4/ubuntu-1519884298-17936/2/ProofOutFile_TestTrigger.root' - check logs for possible stacktrace - last event: 0 Worker 'chinmay-0.2' has been removed from the active list +++ Message from top master at chinmay:1093 : marking chinmay:1093 (0.2) as bad +++ Reason: received kPROOF_FATAL +++ Most likely your code crashed on worker 0.2 at chinmay:1093. +++ Please check the session logs for error messages either using +++ the 'Show logs' button or executing +++ +++ root [] TProof::Mgr("chinmay:1093")->GetSessionLogs()->Display("0.2",0) Mst-0: merging output objects ... done Mst-0: grand total: sent 4 objects, size: 390 bytes *** Break *** segmentation violation | session: chinmay.default.3622.status terminated by peer Info in : 0x398cce0:chinmay:0 got called ... fProof: 0x25b24d0, fSocket: 0x3913b80 (valid: 1) Info in : 0x398cce0: proof: 0x25b24d0 TXSlave::HandleError: 0x398cce0: DONE ... Info in : 0x382b7f0: reconnection attempts explicitly disabled! =========================================================== There was a crash (kSigSegmentationViolation). This is the entire stack trace of all threads: =========================================================== Thread 3 (Thread 0x7fc96b967700 (LWP 3619)): #0 0x00007fc98bb6788d in nanosleep () from /lib64/libc.so.6 #1 0x00007fc98bb67724 in sleep () from /lib64/libc.so.6 #2 0x00007fc96c20fd5c in GarbageCollectorThread (arg=0x382ef00, thr=) at /tmp/xrootd-4.5.0-26536/xrootd-4.5.0/src/XrdClient/XrdClientConnMgr.cc:93 #3 0x00007fc96c491bd8 in XrdSysThread_Xeq (myargs=0x380e020) at /tmp/xrootd-4.5.0-26536/xrootd-4.5.0/src/XrdSys/XrdSysPthread.cc:86 #4 0x00007fc98be6760a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc98bba1bbd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc96b166700 (LWP 3621)): #0 0x00007fc98bb9611d in poll () from /lib64/libc.so.6 #1 0x00007fc96c1f5046 in XrdClientSock::RecvRaw (this=0x25be460, buffer=0x7fc964000900, length=8, substreamid=, usedsubstreamid=) at /tmp/xrootd-4.5.0-26536/xrootd-4.5.0/src/XrdClient/XrdClientSock.cc:162 #2 0x00007fc96c219ede in XrdClientPhyConnection::ReadRaw (this=this entry=0x37887b0, buf=buf entry=0x7fc964000900, len=len entry=8, substreamid=substreamid entry=-1, usedsubstreamid=usedsubstreamid entry=0x7fc96b1615bc) at /tmp/xrootd-4.5.0-26536/xrootd-4.5.0/src/XrdClient/XrdClientPhyConnection.cc:362 #3 0x00007fc96c2281ff in XrdClientMessage::ReadRaw (this=this entry=0x7fc9640008c0, phy=phy entry=0x37887b0) at /tmp/xrootd-4.5.0-26536/xrootd-4.5.0/src/XrdClient/XrdClientMessage.cc:172 #4 0x00007fc96c21c773 in XrdClientPhyConnection::BuildMessage (this=this entry=0x37887b0, IgnoreTimeouts=IgnoreTimeouts entry=true, Enqueue=Enqueue entry=true) at /tmp/xrootd-4.5.0-26536/xrootd-4.5.0/src/XrdClient/XrdClientPhyConnection.cc:443 #5 0x00007fc96c21e64a in SocketReaderThread (arg=0x37887b0, thr=) at /tmp/xrootd-4.5.0-26536/xrootd-4.5.0/src/XrdClient/XrdClientPhyConnection.cc:79 #6 0x00007fc96c491bd8 in XrdSysThread_Xeq (myargs=0x2f4c390) at /tmp/xrootd-4.5.0-26536/xrootd-4.5.0/src/XrdSys/XrdSysPthread.cc:86 #7 0x00007fc98be6760a in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc98bba1bbd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc98d4c8940 (LWP 3605)): #0 0x00007fc98bb674eb in waitpid () from /lib64/libc.so.6 #1 0x00007fc98bae29ab in do_system () from /lib64/libc.so.6 #2 0x00007fc98cd482e0 in TUnixSystem::Exec (this=0x1a1f4f0, shellcmd=0x4a74240 "/home/chinmay/ROOT/root-6.08.04/PROOF_INSTALLATION/etc/gdb-backtrace.sh 3605 1>&2") at /home/chinmay/ROOT/root-6.08.04/core/unix/src/TUnixSystem.cxx:2118 #3 0x00007fc98cd48b52 in TUnixSystem::StackTrace (this=0x1a1f4f0) at /home/chinmay/ROOT/root-6.08.04/core/unix/src/TUnixSystem.cxx:2405 #4 0x00007fc98cd4c647 in TUnixSystem::DispatchSignals (this=0x1a1f4f0, sig=kSigSegmentationViolation) at /home/chinmay/ROOT/root-6.08.04/core/unix/src/TUnixSystem.cxx:3663 #5 0x00007fc98cd4461e in SigHandler (sig=kSigSegmentationViolation) at /home/chinmay/ROOT/root-6.08.04/core/unix/src/TUnixSystem.cxx:408 #6 0x00007fc98cd4c599 in sighandler (sig=11) at /home/chinmay/ROOT/root-6.08.04/core/unix/src/TUnixSystem.cxx:3640 #7 0x00007fc98cd3dd88 in textinput::TerminalConfigUnix::HandleSignal (this=0x7fc98d094300 , signum=11) at /home/chinmay/ROOT/root-6.08.04/core/textinput/src/textinput/TerminalConfigUnix.cpp:100 #8 0x00007fc98cd3dab2 in (anonymous namespace)::TerminalConfigUnix__handleSignal (signum=11) at /home/chinmay/ROOT/root-6.08.04/core/textinput/src/textinput/TerminalConfigUnix.cpp:36 #9 #10 0x00007fc96a4b80fe in SelSensitivity::Terminate() () from /home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/selectors/SelSensitivity_C.so #11 0x00007fc9712ff74b in TProofPlayerRemote::Finalize (this=0x392db10, force=false, sync=true) at /home/chinmay/ROOT/root-6.08.04/proof/proofplayer/src/TProofPlayer.cxx:2915 #12 0x00007fc9712fd020 in TProofPlayerRemote::Process (this=0x392db10, dset=0x36047c0, selector_file=0x2b816e0 "/home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/selectors/SelSensitivity.C+", option=0x7ffde6258509 "", nentries=-1, first=0) at /home/chinmay/ROOT/root-6.08.04/proof/proofplayer/src/TProofPlayer.cxx:2486 #13 0x00007fc971624cfa in TProof::Process (this=0x25b24d0, dset=0x36047c0, selector=0x2b816e0 "/home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/selectors/SelSensitivity.C+", option=0x7fc975ca2882 "", nentries=-1, first=0) at /home/chinmay/ROOT/root-6.08.04/proof/proof/src/TProof.cxx:5331 #14 0x00007fc975c9607a in MSimManagerProof::Sensitivity_iteration (this=this entry=0x7fc98d4eb028, Par_Cuts=Par_Cuts entry=0x7ffde6258880) at /home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/src/SimManager/MSimManagerProof.cxx:1057 #15 0x00007fc975c9753d in MSimManagerProof::Sensitivity (this=0x7fc98d4eb028, dataset=0x7fc98d2fc000 "TestTrigger", cutsfile=, friend_datasets=0x0, trgconf=0x7fc98d2fc018 "4CCNN", sizecut=50) at /home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/src/SimManager/MSimManagerProof.cxx:920 #16 0x00007fc98d2fd07a in ?? () #17 0x0000000000000016 in ?? () #18 0x00007fc98bb22bcc in free () from /lib64/libc.so.6 #19 0x00007fc986894356 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /home/chinmay/ROOT/root-6.08.04/PROOF_INSTALLATION/lib/libCling.so #20 0x00007fc986895bcd in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string, std::allocator > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /home/chinmay/ROOT/root-6.08.04/PROOF_INSTALLATION/lib/libCling.so #21 0x00007fc986895e48 in cling::Interpreter::process(std::__cxx11::basic_string, std::allocator > const&, cling::Value*, cling::Transaction**) () from /home/chinmay/ROOT/root-6.08.04/PROOF_INSTALLATION/lib/libCling.so #22 0x00007fc986918f37 in cling::MetaProcessor::process(char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /home/chinmay/ROOT/root-6.08.04/PROOF_INSTALLATION/lib/libCling.so #23 0x00007fc986731157 in HandleInterpreterException (metaProcessor=0x1c86fd0, input_line=0x4501bb0 "#line 1 \"ROOT_prompt_5\"\nmgr.Sensitivity(\"TestTrigger\",\"cutfile.txt\")", compRes= 0x7ffde62591bc: cling::Interpreter::kSuccess, result=0x7ffde62591c0) at /home/chinmay/ROOT/root-6.08.04/core/meta/src/TCling.cxx:1882 #24 0x00007fc986731e3f in TCling::ProcessLine (this=0x1a7ea20, line=0x44a50a0 "#line 1 \"ROOT_prompt_5\"\nmgr.Sensitivity(\"TestTrigger\",\"cutfile.txt\")", error=0x7ffde625949c) at /home/chinmay/ROOT/root-6.08.04/core/meta/src/TCling.cxx:2048 #25 0x00007fc98cbf23df in TApplication::ProcessLine (this=0x1a707a0, line=0x44a50a0 "#line 1 \"ROOT_prompt_5\"\nmgr.Sensitivity(\"TestTrigger\",\"cutfile.txt\")", sync=false, err=0x7ffde625949c) at /home/chinmay/ROOT/root-6.08.04/core/base/src/TApplication.cxx:1005 #26 0x00007fc98d0b7d34 in TRint::ProcessLineNr (this=0x1a707a0, filestem=0x7fc98d0c54e7 "ROOT_prompt_", line=0x3a495d0 "mgr.Sensitivity(\"TestTrigger\",\"cutfile.txt\")", error=0x7ffde625949c) at /home/chinmay/ROOT/root-6.08.04/core/rint/src/TRint.cxx:749 #27 0x00007fc98d0b7636 in TRint::HandleTermInput (this=0x1a707a0) at /home/chinmay/ROOT/root-6.08.04/core/rint/src/TRint.cxx:610 #28 0x00007fc98d0b51d3 in TTermInputHandler::Notify (this=0x2217df0) at /home/chinmay/ROOT/root-6.08.04/core/rint/src/TRint.cxx:132 #29 0x00007fc98d0b8f2f in TTermInputHandler::ReadNotify (this=0x2217df0) at /home/chinmay/ROOT/root-6.08.04/core/rint/src/TRint.cxx:124 #30 0x00007fc98cd467ca in TUnixSystem::CheckDescriptors (this=0x1a1f4f0) at /home/chinmay/ROOT/root-6.08.04/core/unix/src/TUnixSystem.cxx:1321 #31 0x00007fc98cd45c09 in TUnixSystem::DispatchOneEvent (this=0x1a1f4f0, pendingOnly=false) at /home/chinmay/ROOT/root-6.08.04/core/unix/src/TUnixSystem.cxx:1076 #32 0x00007fc98cc5e7fd in TSystem::InnerLoop (this=0x1a1f4f0) at /home/chinmay/ROOT/root-6.08.04/core/base/src/TSystem.cxx:408 #33 0x00007fc98cc5e5a9 in TSystem::Run (this=0x1a1f4f0) at /home/chinmay/ROOT/root-6.08.04/core/base/src/TSystem.cxx:358 #34 0x00007fc98cbf2d88 in TApplication::Run (this=0x1a707a0, retrn=false) at /home/chinmay/ROOT/root-6.08.04/core/base/src/TApplication.cxx:1153 #35 0x00007fc98d0b69d9 in TRint::Run (this=0x1a707a0, retrn=false) at /home/chinmay/ROOT/root-6.08.04/core/rint/src/TRint.cxx:463 #36 0x000000000040139c in main (argc=1, argv=0x7ffde625b978) at /home/chinmay/ROOT/root-6.08.04/main/src/rmain.cxx:30 =========================================================== The lines below might hint at the cause of the crash. You may get help by asking at the ROOT forum http://root.cern.ch/forum. Only if you are really convinced it is a bug in ROOT then please submit a report at http://root.cern.ch/bugs. Please post the ENTIRE stack trace from above as an attachment in addition to anything else that might help us fixing this issue. =========================================================== #10 0x00007fc96a4b80fe in SelSensitivity::Terminate() () from /home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/selectors/SelSensitivity_C.so #11 0x00007fc9712ff74b in TProofPlayerRemote::Finalize (this=0x392db10, force=false, sync=true) at /home/chinmay/ROOT/root-6.08.04/proof/proofplayer/src/TProofPlayer.cxx:2915 #12 0x00007fc9712fd020 in TProofPlayerRemote::Process (this=0x392db10, dset=0x36047c0, selector_file=0x2b816e0 "/home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/selectors/SelSensitivity.C+", option=0x7ffde6258509 "", nentries=-1, first=0) at /home/chinmay/ROOT/root-6.08.04/proof/proofplayer/src/TProofPlayer.cxx:2486 #13 0x00007fc971624cfa in TProof::Process (this=0x25b24d0, dset=0x36047c0, selector=0x2b816e0 "/home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/selectors/SelSensitivity.C+", option=0x7fc975ca2882 "", nentries=-1, first=0) at /home/chinmay/ROOT/root-6.08.04/proof/proof/src/TProof.cxx:5331 #14 0x00007fc975c9607a in MSimManagerProof::Sensitivity_iteration (this=this entry=0x7fc98d4eb028, Par_Cuts=Par_Cuts entry=0x7ffde6258880) at /home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/src/SimManager/MSimManagerProof.cxx:1057 #15 0x00007fc975c9753d in MSimManagerProof::Sensitivity (this=0x7fc98d4eb028, dataset=0x7fc98d2fc000 "TestTrigger", cutsfile=, friend_datasets=0x0, trgconf=0x7fc98d2fc018 "4CCNN", sizecut=50) at /home/chinmay/MaceSIM-PROOF-DEV/MaceSIM/src/SimManager/MSimManagerProof.cxx:920 #16 0x00007fc98d2fd07a in ?? () #17 0x0000000000000016 in ?? () #18 0x00007fc98bb22bcc in free () from /lib64/libc.so.6 #19 0x00007fc986894356 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /home/chinmay/ROOT/root-6.08.04/PROOF_INSTALLATION/lib/libCling.so #20 0x00007fc986895bcd in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string, std::allocator > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /home/chinmay/ROOT/root-6.08.04/PROOF_INSTALLATION/lib/libCling.so #21 0x00007fc986895e48 in cling::Interpreter::process(std::__cxx11::basic_string, std::allocator > const&, cling::Value*, cling::Transaction**) () from /home/chinmay/ROOT/root-6.08.04/PROOF_INSTALLATION/lib/libCling.so #22 0x00007fc986918f37 in cling::MetaProcessor::process(char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /home/chinmay/ROOT/root-6.08.04/PROOF_INSTALLATION/lib/libCling.so #23 0x00007fc986731157 in HandleInterpreterException (metaProcessor=0x1c86fd0, input_line=0x4501bb0 "#line 1 "ROOT_prompt_5"nmgr.Sensitivity("TestTrigger","cutfile.txt")", compRes= 0x7ffde62591bc: cling::Interpreter::kSuccess, result=0x7ffde62591c0) at /home/chinmay/ROOT/root-6.08.04/core/meta/src/TCling.cxx:1882 =========================================================== Root > *** No workers left: cannot continue! Terminating ... *** Info in : +++ Message from local session : marking chinmay:1093 (0) as bad +++ Reason: received kPROOF_FATAL +++ Message from local session : marking chinmay:1093 (0) as bad +++ Reason: received kPROOF_FATAL +++ Most likely your code crashed +++ Please check the session logs for error messages either using +++ the 'Show logs' button or executing +++ +++ root [] TProof::Mgr("chinmay@chinmay:1093")->GetSessionLogs()->Display("*") 180305 11:27:40 3605 Proofx-E: Conn::LowWrite: sending header to server [chinmay:1093] (rc=-3) 180305 11:27:40 3605 Proofx-E: Conn::SendRecv: problems sending request to server [chinmay:1093]