TSelector works, except in PROOF-Lite

Hi,

I wrote a TSelector, and include a version of it (with all analysis gutted out but with enough to reproduce my error) below. When I run the selector on a single thread, it works just fine. However, if I try to run it in PROOFLite I get a segfault that I think is happening in the Process method at “fChain->GetTree()->GetEntry()”

Here is testSelector.C

#define testSelector_cxx
#include "testSelector.h"
#include <TH2.h>
#include <TStyle.h>
#include <iostream>

void testSelector::Begin(TTree * /*tree*/)
{
}

void testSelector::SlaveBegin(TTree * tree)
{

   TString option = GetOption();


}

Bool_t testSelector::Process(Long64_t entry)
{
  fChain->GetTree()->GetEntry(entry);

  Double_t x = theS800->crdc1.x;
  if (entry%100==0){
    cout << "THIS IS X " << x << endl;
  }

   return kTRUE;
}

void testSelector::SlaveTerminate()
{


}

void testSelector::Terminate()
{

}

And here is testSelector.h:

#ifndef testSelector_h
#define testSelector_h

#include <TROOT.h>
#include <TChain.h>
#include <TFile.h>
#include <TSelector.h>
#include <TObject.h>
#include <TNamed.h>
#include <TCutG.h>
#include <TCanvas.h>

// Header file for the classes stored in the TTree if any.
#include "./RBS800.h"
//#include "/scratch/blair/root/core/base/inc/TObject.h"
#include "./RBDetector.h"
//#include "/scratch/blair/root/core/base/inc/TNamed.h"
#include "./RBS800Crdc.h"
#include "./RBS800CrdcPads.h"
#include "./RBS800CrdcCalc.h"
#include "./RBS800IonChamber.h"
#include "./RBS800Scintillator.h"
#include "./RBS800Hodoscope.h"
#include "./RBS800FpTrack.h"
#include "./RBS800FpParticle.h"
#include "./RBS800Trigger.h"
#include "./RBS800TimeOfFlight.h"

#include <iostream>
using namespace std;
#include "./settings/S800Calibrations.C"
//This is where the loadS800RunSettings function lives

// Fixed size dimensions of array or collections stored in the TTree if any.
const Int_t kMaxfpTrack_fFpParticles = 1;

class testSelector : public TSelector {
public :
   TTree          *fChain=0;   //!pointer to the analyzed TTree or TChain

   // Declaration of leaf types
   RBS800          *theS800=0;

   TBranch         *b_theS800=0;

   testSelector(TTree * /*tree*/ =0) : fChain(0) { }
   virtual ~testSelector() { }
   virtual Int_t   Version() const { return 2; }
   virtual void    Begin(TTree *tree);
   virtual void    SlaveBegin(TTree *tree);
   virtual void    Init(TTree *tree);
   virtual Bool_t  Notify();
   virtual Bool_t  Process(Long64_t entry);
   virtual Int_t   GetEntry(Long64_t entry, Int_t getall = 0) { return fChain ? fChain->GetTr\
ee()->GetEntry(entry, getall) : 0; }
   virtual void    SetOption(const char *option) { fOption = option; }
   virtual void    SetObject(TObject *obj) { fObject = obj; }
   virtual void    SetInputList(TList *input) { fInput = input; }
   virtual TList  *GetOutputList() const { return fOutput; }
   virtual void    SlaveTerminate();
   virtual void    Terminate();

   TString outputFile;
   TString runString;

   ClassDef(testSelector,0);
};

#endif

#ifdef testSelector_cxx
void testSelector::Init(TTree *tree)
{

   theS800 = new RBS800();

   // Set branch addresses and branch pointers
   if (!tree) return;
   fChain = tree;
   fChain->SetMakeClass(0);
   fChain->SetBranchAddress("s800",&theS800,&b_theS800);
}

Bool_t testSelector::Notify()
{
   return kTRUE;
}

#endif // #ifdef testSelector_cxx

where RBS800*.h are the relevant files defining the RBS800 class.

This TSelector works just fine when I run it on a single thread, in other words:

In the interpreter:
root [0] TFile f(“name/of/file.root”)
root [1] TreeName->Process(“testSelector.C+”)

I am trying to run this in PROOFLite in as simple a way as possible with this script. If I modify the TSelector and remove any mention of retrieving data from a file, this script works fine:

testProof.C

#include <sstream>
#include <vector>
#include <ctime>
#include <algorithm>
#endif

void testProof()
{

   //Open ROOT files and assemble TChain
  TChain* ch = new TChain("E09084Cal");
  ch->Add("/mnt/analysis/e09084/rootfilesCal/run-0343Cal.root");
  TProof * p = TProof::Open("","workers=1");

  ch->SetProof(kTRUE);

  ch->Process("testSelector.C+");



  delete ch;



}

Yet when I try to run this with testSelector I get the following message:

[code] +++ Starting PROOF-Lite with 1 workers +++
Opening connections to workers: OK (1 workers)
Setting up worker servers: OK (1 workers)
PROOF set to parallel mode (1 worker)
15:00:22 32516 Wrk-0.0 | Info in TProofServLite::HandleCache: loading macro testSelector.C+ …

Info in TProofLite::SetQueryRunning: starting query: 1
Info in TProofQueryResult::SetRunning: nwrks: 1
Looking up for exact location of files: OK (1 files)
Looking up for exact location of files: OK (1 files)
Info in TPacketizerAdaptive::TPacketizerAdaptive: Setting max number of workers per node to 1
Validating files: OK (1 files)
Info in TPacketizerAdaptive::InitStats: fraction of remote files 1.000000
0.0: caught exception triggered by signal ‘1’ while processing dset:‘TDSet:E09084Cal’, file:’/mnt/analysis/e09084/rootfilesCal/run-0343Cal.root’ - check logs for possible stacktrace - last event: 0
Info in TProofLite::MarkBad:
+++ Message from master at walleye.nscl.msu.edu : marking walleye:-1 (0.0) as bad
+++ Reason: undefined message in TProof::CollectInputFrom(…)

+++ Message from master at walleye.nscl.msu.edu : marking walleye:-1 (0.0) as bad
+++ Reason: undefined message in TProof::CollectInputFrom(…)

+++ Most likely your code crashed
+++ Please check the session logs for error messages either using
+++ the ‘Show logs’ button or executing
+++
+++ root [] TProof::Mgr(“walleye.nscl.msu.edu”)->GetSessionLogs()->Display("*")

entries: 1000 (1000)
Error in TPacketizerAdaptive::SplitPerHost: Error removing a missing file
Info in TPacketizerAdaptive::InitStats: fraction of remote files 1.000000
Lite-0: all output objects have been merged
[/code]

Here are the relevant logs:

[code]// --------- Start of element log -----------------

// Ordinal: 0 (role: master)

// Path: /user/manfredi/.proof/projects-e09084-ribbit/session-walleye-1436560628-41484/session-walleye-1436560628-41484.log
// # of retrieved lines: 1

// ------------------------------------------------

// --------- End of element log -------------------

Retrieving logs: 1 ok, 0 not ok (100 % processed)

// --------- Start of element log -----------------

// Ordinal: 0.0 (role: worker)

// Path: /user/manfredi/.proof/projects-e09084-ribbit/session-walleye-1436560628-41484/worker-0.0-walleye-1436560628-41496.log
// # of retrieved lines: 80

// ------------------------------------------------

16:37:08 41496 Wrk-0.0 | Info in TProofServLite::Setup: fWorkDir: /user/manfredi/.proof
16:37:08 41496 Wrk-0.0 | Info in TProofServLite::HandleCache: loading macro testSelector.C+ …
16:37:09 41496 Wrk-0.0 | Info in TProofServLite::HandleProcess: selector obj for ‘testSelector’ found
16:37:09 41496 Wrk-0.0 | Info in TProofServLite::HandleProcess: calling fPlayer->Process() with selector object: testSelector
16:37:09 41496 Wrk-0.0 | Info in TProofPlayerSlave::AssertSelector: Processing via TSelector object
16:37:09 41496 Wrk-0.0 | Info in TEventIter::TEventIter: fPackets list ‘ProcessedPackets_0.0’ created
16:37:09 41496 Wrk-0.0 | Info in TProofPlayerSlave::Process: save partial results? 0 per-packet? 0
16:37:09 41496 Wrk-0.0 | Info in TEventIterTree::GetTrees: the tree cache is in learning phase
Array size is 32
Run 343 Settings
Run settings loaded for run 343
Now I have Run 343
16:37:09 41496 Wrk-0.0 | *** Break ***: segmentation violation

There was a crash.
This is the entire stack trace of all threads:

#0 0x00007f16a325d06e in __libc_waitpid (pid=, stat_loc=0x7ffda69632ac, options=) at …/sysdeps/unix/sysv/linux/waitpid.c:32
#1 0x00007f16a31f1989 in do_system (line=) at …/sysdeps/posix/system.c:149
#2 0x00007f16a4f8c4ec in TUnixSystem::StackTrace() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#3 0x00007f16a4f8eb53 in TUnixSystem::DispatchSignals(ESignals) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#4
#5 0x00007f16a4ebde95 in ROOT::delete_TClonesArray(void*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#6 0x00007f16a4f58d40 in TClass::Destructor(void*, bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#7 0x00007f16a1c732f5 in TBufferFile::ReadFastArray(void**, TClass const*, int, bool, TMemberStreamer*, TClass const*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#8 0x00007f16a1c3f6df in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, int, int, int, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#9 0x00007f16a1b466fb in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#10 0x00007f16a1c71665 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#11 0x00007f16a0962333 in TBranchElement::ReadLeavesMember(TBuffer&) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#12 0x00007f16a0917437 in TBranch::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#13 0x00007f16a09699c5 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#14 0x00007f16a0969980 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#15 0x00007f16a0969980 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#16 0x00007f16a09236f3 in TTree::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#17 0x00007f16a01b3dba in testSelector::Process(long long) () from /user/manfredi/.proof/projects-e09084-ribbit/session-walleye-1436560628-41484/worker-0.0/./testSelector_C.so
#18 0x00007f169a266179 in TProofPlayer::Process(TDSet*, char const*, char const*, long long, long long) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProofPlayer.so
#19 0x00007f16a0533318 in TProofServ::HandleProcess(TMessage*, TString*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#20 0x00007f16a052c32e in TProofServ::HandleSocketInput(TMessage*, bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#21 0x00007f16a051f337 in TProofServ::HandleSocketInput() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#22 0x00007f16a053f9e1 in TProofServLiteInputHandler::Notify() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#23 0x00007f16a4f8dcbd in TUnixSystem::CheckDescriptors() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#24 0x00007f16a4f8f288 in TUnixSystem::DispatchOneEvent(bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#25 0x00007f16a4fe3c76 in TSystem::InnerLoop() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#26 0x00007f16a4fe5724 in TSystem::Run() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#27 0x00007f16a4ff3d0f in TApplication::Run(bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#28 0x0000000000401ce1 in main ()

The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.

#5 0x00007f16a4ebde95 in ROOT::delete_TClonesArray(void*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#6 0x00007f16a4f58d40 in TClass::Destructor(void*, bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#7 0x00007f16a1c732f5 in TBufferFile::ReadFastArray(void**, TClass const*, int, bool, TMemberStreamer*, TClass const*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#8 0x00007f16a1c3f6df in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, int, int, int, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#9 0x00007f16a1b466fb in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#10 0x00007f16a1c71665 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#11 0x00007f16a0962333 in TBranchElement::ReadLeavesMember(TBuffer&) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#12 0x00007f16a0917437 in TBranch::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#13 0x00007f16a09699c5 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#14 0x00007f16a0969980 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#15 0x00007f16a0969980 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#16 0x00007f16a09236f3 in TTree::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#17 0x00007f16a01b3dba in testSelector::Process(long long) () from /user/manfredi/.proof/projects-e09084-ribbit/session-walleye-1436560628-41484/worker-0.0/./testSelector_C.so
#18 0x00007f169a266179 in TProofPlayer::Process(TDSet*, char const*, char const*, long long, long long) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProofPlayer.so
#19 0x00007f16a0533318 in TProofServ::HandleProcess(TMessage*, TString*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#20 0x00007f16a052c32e in TProofServ::HandleSocketInput(TMessage*, bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#21 0x00007f16a051f337 in TProofServ::HandleSocketInput() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#22 0x00007f16a053f9e1 in TProofServLiteInputHandler::Notify() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#23 0x00007f16a4f8dcbd in TUnixSystem::CheckDescriptors() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#24 0x00007f16a4f8f288 in TUnixSystem::DispatchOneEvent(bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#25 0x00007f16a4fe3c76 in TSystem::InnerLoop() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#26 0x00007f16a4fe5724 in TSystem::Run() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#27 0x00007f16a4ff3d0f in TApplication::Run(bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#28 0x0000000000401ce1 in main ()

16:37:10 41496 Wrk-0.0 | Error in TProofServLite::HandleException: caugth exception triggered by signal ‘1’ while processing dset:‘TDSet:E09084Cal’, file:’/mnt/analysis/e09084/rootfilesCal/run-0343Cal.root’ - check logs for possible stacktrace - last event: 0

// --------- End of element log -------------------[/code]

I’d appreciate any help or insight you can provide. I would say the problem is with the RBS800 class but that’s contradicted by the fact that the testSelector works in a single thread. Perhaps I am not loading a library where and when I need to but I have tried explicitly doing that as well. These errors persist with different ROOT files and with different numbers of workers.

Thanks,
Juan

By the way, I am using 5.34.09 on Linux 3.2.0-4-amd64, Debian 3.2.68-1 x86_64 GNU/Linux

Hello,

If the ‘RBS800*.h are the relevant files defining the RBS800 class’ the problem is likely with these not being loaded in the PROOF-Lite workers.

How did you do that?
The way this is can be done in PROOF(-Lite) is described in
root.cern.ch/drupal/content/pre … l-software
If you have many classes or files defining our objects the easiest is to include them in a PAR file.
For examples, you can check:

tutorials/proof/event.par
etc/proof/proofbench/ProofBenchDataSel.par

G Ganis

Thank you for your response.

I meant that I had tried explicitly loading the selector library on each thread with something like:

TString selectorLibrary = "/path/to/testSelector_C.so";
TString libCommand = "gSystem->Load(\"" +selectorLibrary+ "\")";
p->Exec(libCommand);

I tried to use a par file as you suggested. I followed root.cern.ch/drupal/content/cre … -root-file, and ran

root[0] TFile *f = TFile::Open("/name/of/file.root")
root[1] f->MakeProject("/proof/sandbox/packages/testPack", "*", "par")

This is the output:

MakeProject has generated 0 classes in /projects/e09084/ribbit/PROOF/packages/testPack2
Files Makefile, Makefile.arch, PROOF-INF/BUILD.sh and PROOF-INF/SETUP.C have been generated under '/projects/e09084/ribbit/PROOF/packages/testPack2'
testPack2/
testPack2/PROOF-INF/
testPack2/PROOF-INF/BUILD.sh
testPack2/PROOF-INF/SETUP.C
testPack2/testPack2ProjectSource.cxx
testPack2/testPack2ProjectHeaders.h
testPack2/testPack2ProjectInstances.h
testPack2/testPack2LinkDef.h
testPack2/Makefile
testPack2/Makefile.arch
Info in <TFile::MakeProject>: PAR file /projects/e09084/ribbit/PROOF/packages/testPack2.par generated

Although the PAR file was generated, the line “MakeProject has generated 0 classes” gives me some concern. When I examine this file, it obviously has objects in it and the classes are indeed defined with ClassDef in the source code. Still, I suppose its possible there could be some problem with my tree? If the problem is with my root file, I am confused as to why I can successfully use the TSelector in a single thread. If you’d like I can try to make a very small ROOT file and post it here. Do you know what the size limit is for attachments?

Regardless, I continued on. I put the following lines into my testProof.C macro

p->UploadPackage("testPack");
p->EnablePackage("testPack");

When running this, I first got an error complaining that a sym link could not be created in proof/sandbox/packages because testPack.par already existed. So I moved testPack.par to the working directory and tried again, but still see a segfault. But, I increased the number of workers to 10, and now I see a similar segfault for the first worker but the rest seem to work. Here are the logs. I included a simple output statement (“THIS IS X”) in my selector to test to show that I am properly getting the entry (I expect to see a lot nan’s, so I don’t think that’s an issue).


// --------- Start of element log -----------------

// Ordinal: 0 (role: master)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/session-walleye-1436734861-6789.log 
// # of retrieved lines: 1 


// ------------------------------------------------

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.0 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.0-walleye-1436734861-6804.log 
// # of retrieved lines: 92 


// ------------------------------------------------

17:01:01  6804 Wrk-0.0 | Info in <TProofServLite::Setup>: fWorkDir: /projects/e09084/ribbit/PROOF
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800 is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBDetector is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800Crdc is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800CrdcPads is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800CrdcCalc is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800IonChamber is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800Scintillator is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800Hodoscope is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800FpTrack is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800Trigger is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800TimeOfFlight is available
17:01:05  6804 Wrk-0.0 | Warning in <TClass::TClass>: no dictionary for class RBS800FpParticle is available
17:01:05  6804 Wrk-0.0 | Info in <TProofServLite::HandleProcess>: calling fPlayer->Process() with selector name: testSelector.C+
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.C' and `./testSelector.C' are the same file
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.h' and `./testSelector.h' are the same file
17:01:06  6804 Wrk-0.0 | Info in <TProofPlayerSlave::AssertSelector>: Processing via filename
17:01:06  6804 Wrk-0.0 | Info in <TEventIter::TEventIter>: fPackets list 'ProcessedPackets_0.0' created
17:01:06  6804 Wrk-0.0 | Info in <TProofPlayerSlave::Process>: save partial results? 0  per-packet? 0
17:01:06  6804 Wrk-0.0 | Info in <TEventIterTree::GetTrees>: the tree cache is in learning phase
Array size is 32
Run 343 Settings 
Run settings loaded for run 343
Now I have Run 343
17:01:07  6804 Wrk-0.0 | *** Break ***: segmentation violation
===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f95a9ed706e in __libc_waitpid (pid=<optimized out>, stat_loc=0x7ffea7e9f8ec, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:32
#1  0x00007f95a9e6b989 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:149
#2  0x00007f95abc064ec in TUnixSystem::StackTrace() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#3  0x00007f95abc08b53 in TUnixSystem::DispatchSignals(ESignals) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#4  <signal handler called>
#5  0x00007f95abb37e95 in ROOT::delete_TClonesArray(void*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#6  0x00007f95abbd2d40 in TClass::Destructor(void*, bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#7  0x00007f95a88ed2f5 in TBufferFile::ReadFastArray(void**, TClass const*, int, bool, TMemberStreamer*, TClass const*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#8  0x00007f95a88b96df in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, int, int, int, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#9  0x00007f95a87c06fb in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#10 0x00007f95a88eb665 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#11 0x00007f95a75dc333 in TBranchElement::ReadLeavesMember(TBuffer&) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#12 0x00007f95a7591437 in TBranch::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#13 0x00007f95a75e39c5 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#14 0x00007f95a75e3980 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#15 0x00007f95a75e3980 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#16 0x00007f95a759d6f3 in TTree::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#17 0x00007f95a4a566ac in testSelector::Process(long long) () from /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.0/./testSelector_C.so
#18 0x00007f95a4d0d179 in TProofPlayer::Process(TDSet*, char const*, char const*, long long, long long) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProofPlayer.so
#19 0x00007f95a71ad9ac in TProofServ::HandleProcess(TMessage*, TString*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#20 0x00007f95a71a632e in TProofServ::HandleSocketInput(TMessage*, bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#21 0x00007f95a7199337 in TProofServ::HandleSocketInput() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#22 0x00007f95a71b99e1 in TProofServLiteInputHandler::Notify() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#23 0x00007f95abc07cbd in TUnixSystem::CheckDescriptors() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#24 0x00007f95abc09288 in TUnixSystem::DispatchOneEvent(bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#25 0x00007f95abc5dc76 in TSystem::InnerLoop() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#26 0x00007f95abc5f724 in TSystem::Run() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#27 0x00007f95abc6dd0f in TApplication::Run(bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#28 0x0000000000401ce1 in main ()
===========================================================
The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007f95abb37e95 in ROOT::delete_TClonesArray(void*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#6  0x00007f95abbd2d40 in TClass::Destructor(void*, bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#7  0x00007f95a88ed2f5 in TBufferFile::ReadFastArray(void**, TClass const*, int, bool, TMemberStreamer*, TClass const*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#8  0x00007f95a88b96df in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, int, int, int, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#9  0x00007f95a87c06fb in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#10 0x00007f95a88eb665 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libRIO.so
#11 0x00007f95a75dc333 in TBranchElement::ReadLeavesMember(TBuffer&) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#12 0x00007f95a7591437 in TBranch::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#13 0x00007f95a75e39c5 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#14 0x00007f95a75e3980 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#15 0x00007f95a75e3980 in TBranchElement::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#16 0x00007f95a759d6f3 in TTree::GetEntry(long long, int) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libTree.so
#17 0x00007f95a4a566ac in testSelector::Process(long long) () from /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.0/./testSelector_C.so
#18 0x00007f95a4d0d179 in TProofPlayer::Process(TDSet*, char const*, char const*, long long, long long) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProofPlayer.so
#19 0x00007f95a71ad9ac in TProofServ::HandleProcess(TMessage*, TString*) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#20 0x00007f95a71a632e in TProofServ::HandleSocketInput(TMessage*, bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#21 0x00007f95a7199337 in TProofServ::HandleSocketInput() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#22 0x00007f95a71b99e1 in TProofServLiteInputHandler::Notify() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libProof.so
#23 0x00007f95abc07cbd in TUnixSystem::CheckDescriptors() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#24 0x00007f95abc09288 in TUnixSystem::DispatchOneEvent(bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#25 0x00007f95abc5dc76 in TSystem::InnerLoop() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#26 0x00007f95abc5f724 in TSystem::Run() () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#27 0x00007f95abc6dd0f in TApplication::Run(bool) () from /mnt/misc/sw/x86_64/Debian/7/root/gnu/5.34.09/lib/libCore.so.5
#28 0x0000000000401ce1 in main ()
===========================================================
17:01:08  6804 Wrk-0.0 | Error in <TProofServLite::HandleException>: caugth exception triggered by signal '1' while processing dset:'TDSet:E09084Cal', file:'/mnt/analysis/e09084/rootfilesCal/run-0343Cal.root' - check logs for possible stacktrace - last event: 92240

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.1 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.1-walleye-1436734861-6806.log 
// # of retrieved lines: 116 
(displaying lines: 17 -> 116)

// ------------------------------------------------

THIS IS X -nan
THIS IS X 162.029
THIS IS X -nan
THIS IS X 257.673
THIS IS X 161.739
THIS IS X -nan
THIS IS X -nan
THIS IS X 167.288
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 502.484
THIS IS X 541.123
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 441.037
THIS IS X 251.404
THIS IS X 434.185
THIS IS X 416.447
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 265.323
THIS IS X -nan
THIS IS X -nan
THIS IS X 143.209
THIS IS X 171.296
THIS IS X 445.514
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 208.91
THIS IS X -nan
THIS IS X 160.516
THIS IS X -nan
THIS IS X 156.915
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 354.305
THIS IS X 412.249
THIS IS X 276.976
THIS IS X 468.011
THIS IS X 423.988
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 333.31
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 296.61
THIS IS X -nan
THIS IS X 372.114
THIS IS X -nan
THIS IS X -nan
THIS IS X 333.855
THIS IS X -nan
THIS IS X 238.899
THIS IS X -nan
THIS IS X 233.408
THIS IS X -nan
THIS IS X -nan
THIS IS X 144.997
THIS IS X -nan
THIS IS X 197.33
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 523.748
THIS IS X 452.67
THIS IS X 463.539
THIS IS X -nan
THIS IS X -nan
THIS IS X 478.795
THIS IS X 139.376
THIS IS X -nan
THIS IS X 492.558
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.2 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.2-walleye-1436734861-6808.log 
// # of retrieved lines: 103 
(displaying lines: 4 -> 103)

// ------------------------------------------------

/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.h' and `./testSelector.h' are the same file
17:01:06  6808 Wrk-0.2 | Info in <TProofPlayerSlave::AssertSelector>: Processing via filename
17:01:06  6808 Wrk-0.2 | Info in <TEventIter::TEventIter>: fPackets list 'ProcessedPackets_0.2' created
17:01:06  6808 Wrk-0.2 | Info in <TProofPlayerSlave::Process>: save partial results? 0  per-packet? 0
17:01:06  6808 Wrk-0.2 | Info in <TEventIterTree::GetTrees>: the tree cache is in learning phase
Array size is 32
Run 343 Settings 
Run settings loaded for run 343
Now I have Run 343
THIS IS X 210.728
17:01:06  6808 Wrk-0.2 | Info in <TProofServLite::RestartComputeTime>: compute time restarted after 0.313840 secs (100 entries)
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 292.194
THIS IS X -nan
THIS IS X -nan
THIS IS X 242.342
THIS IS X 370.826
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 143.89
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 414.015
THIS IS X -nan
THIS IS X -nan
THIS IS X 313.066
THIS IS X 474.773
THIS IS X 335.905
THIS IS X 492.172
THIS IS X 227.316
THIS IS X 364.284
THIS IS X 169.92
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 225.18
THIS IS X -nan
THIS IS X 198.022
THIS IS X -nan
THIS IS X -nan
THIS IS X 192.78
THIS IS X -nan
THIS IS X 235.945
THIS IS X 185.482
THIS IS X 294.597
THIS IS X -nan
THIS IS X -nan
THIS IS X 323.702
THIS IS X -nan
THIS IS X 425.61
THIS IS X -nan
THIS IS X 327.068
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 313.303
THIS IS X -nan
THIS IS X 161.704
THIS IS X 433.27
THIS IS X 159.292
THIS IS X -nan
THIS IS X -nan
THIS IS X 282.691
THIS IS X 409.304
THIS IS X 356.824
THIS IS X -nan
THIS IS X 249.747
THIS IS X -nan
THIS IS X 143.128
THIS IS X 494.609
THIS IS X -nan
THIS IS X -nan
THIS IS X 490.641
THIS IS X 252.435
THIS IS X 190.654
THIS IS X -nan
THIS IS X 423.856
THIS IS X -nan
THIS IS X 285.639
THIS IS X -nan
THIS IS X -nan
THIS IS X 202.977
THIS IS X 392.662
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 414.661

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.3 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.3-walleye-1436734861-6810.log 
// # of retrieved lines: 87 


// ------------------------------------------------

17:01:01  6810 Wrk-0.3 | Info in <TProofServLite::Setup>: fWorkDir: /projects/e09084/ribbit/PROOF
17:01:05  6810 Wrk-0.3 | Info in <TProofServLite::HandleProcess>: calling fPlayer->Process() with selector name: testSelector.C+
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.C' and `./testSelector.C' are the same file
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.h' and `./testSelector.h' are the same file
17:01:06  6810 Wrk-0.3 | Info in <TProofPlayerSlave::AssertSelector>: Processing via filename
17:01:06  6810 Wrk-0.3 | Info in <TEventIter::TEventIter>: fPackets list 'ProcessedPackets_0.3' created
17:01:06  6810 Wrk-0.3 | Info in <TProofPlayerSlave::Process>: save partial results? 0  per-packet? 0
17:01:06  6810 Wrk-0.3 | Info in <TEventIterTree::GetTrees>: the tree cache is in learning phase
Array size is 32
Run 343 Settings 
Run settings loaded for run 343
Now I have Run 343
17:01:06  6810 Wrk-0.3 | Info in <TProofServLite::RestartComputeTime>: compute time restarted after 0.370440 secs (100 entries)
THIS IS X 339.61
THIS IS X -nan
THIS IS X 283.166
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 421.488
THIS IS X 407.086
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 194.418
THIS IS X -nan
THIS IS X 311.744
THIS IS X 385.781
THIS IS X -nan
THIS IS X 310.886
THIS IS X -nan
THIS IS X 457.741
THIS IS X 388.805
THIS IS X 167.12
THIS IS X 228.706
THIS IS X -nan
THIS IS X -nan
THIS IS X 156.438
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 328.749
THIS IS X -nan
THIS IS X 391.425
THIS IS X -nan
THIS IS X 328.826
THIS IS X -nan
THIS IS X 373.326
THIS IS X 209.382
THIS IS X -nan
THIS IS X -nan
THIS IS X 196.44
THIS IS X -nan
THIS IS X -nan
THIS IS X 194.226
THIS IS X -nan
THIS IS X 450.841
THIS IS X 404.812
THIS IS X -nan
THIS IS X -nan
THIS IS X 286.679
THIS IS X -nan
THIS IS X 485.477
THIS IS X -nan
THIS IS X 270.485
THIS IS X -nan
THIS IS X 410.486
THIS IS X 341.362
THIS IS X -nan
THIS IS X -nan
THIS IS X 324.319
THIS IS X 398.339
THIS IS X 214.716
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 195.572
THIS IS X 228.497
THIS IS X 346.204
THIS IS X 467.688
THIS IS X 143.476
THIS IS X 275.804
THIS IS X -nan
THIS IS X 198.07

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.4 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.4-walleye-1436734861-6812.log 
// # of retrieved lines: 60 


// ------------------------------------------------

17:01:01  6812 Wrk-0.4 | Info in <TProofServLite::Setup>: fWorkDir: /projects/e09084/ribbit/PROOF
17:01:05  6812 Wrk-0.4 | Info in <TProofServLite::HandleProcess>: calling fPlayer->Process() with selector name: testSelector.C+
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.C' and `./testSelector.C' are the same file
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.h' and `./testSelector.h' are the same file
17:01:06  6812 Wrk-0.4 | Info in <TProofPlayerSlave::AssertSelector>: Processing via filename
17:01:06  6812 Wrk-0.4 | Info in <TEventIter::TEventIter>: fPackets list 'ProcessedPackets_0.4' created
17:01:06  6812 Wrk-0.4 | Info in <TProofPlayerSlave::Process>: save partial results? 0  per-packet? 0
17:01:07  6812 Wrk-0.4 | Info in <TEventIterTree::GetTrees>: the tree cache is in learning phase
Array size is 32
Run 343 Settings 
Run settings loaded for run 343
Now I have Run 343
17:01:07  6812 Wrk-0.4 | Info in <TProofServLite::RestartComputeTime>: compute time restarted after 0.524356 secs (100 entries)
THIS IS X 312.134
THIS IS X -nan
THIS IS X 462.958
THIS IS X -nan
THIS IS X 368.848
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 335.755
THIS IS X 521.995
THIS IS X 290.385
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 389.3
THIS IS X 302.186
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 220.76
THIS IS X 282.557
THIS IS X 335.409
THIS IS X -nan
THIS IS X -nan
THIS IS X 274.375
THIS IS X 288.106
THIS IS X 140.79
THIS IS X 143.722
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 348.521
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 445.529
THIS IS X -nan

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.5 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.5-walleye-1436734861-6814.log 
// # of retrieved lines: 64 


// ------------------------------------------------

17:01:01  6814 Wrk-0.5 | Info in <TProofServLite::Setup>: fWorkDir: /projects/e09084/ribbit/PROOF
17:01:05  6814 Wrk-0.5 | Info in <TProofServLite::HandleProcess>: calling fPlayer->Process() with selector name: testSelector.C+
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.C' and `./testSelector.C' are the same file
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.h' and `./testSelector.h' are the same file
17:01:06  6814 Wrk-0.5 | Info in <TProofPlayerSlave::AssertSelector>: Processing via filename
17:01:06  6814 Wrk-0.5 | Info in <TEventIter::TEventIter>: fPackets list 'ProcessedPackets_0.5' created
17:01:06  6814 Wrk-0.5 | Info in <TProofPlayerSlave::Process>: save partial results? 0  per-packet? 0
17:01:06  6814 Wrk-0.5 | Info in <TEventIterTree::GetTrees>: the tree cache is in learning phase
Array size is 32
Run 343 Settings 
Run settings loaded for run 343
Now I have Run 343
THIS IS X -nan
17:01:06  6814 Wrk-0.5 | Info in <TProofServLite::RestartComputeTime>: compute time restarted after 0.255401 secs (100 entries)
THIS IS X -nan
THIS IS X 379.516
THIS IS X 395.66
THIS IS X 344.205
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 226.03
THIS IS X -nan
THIS IS X -nan
THIS IS X 374.87
THIS IS X 329.021
THIS IS X 396.56
THIS IS X -nan
THIS IS X 242.428
THIS IS X 243.453
THIS IS X 182.26
THIS IS X 228.535
THIS IS X -nan
THIS IS X 206.586
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 218.589
THIS IS X 148.918
THIS IS X -nan
THIS IS X -nan
THIS IS X 393.184
THIS IS X -nan
THIS IS X 411.889
THIS IS X -nan
THIS IS X 505.208
THIS IS X 234.785
THIS IS X -nan
THIS IS X -nan
THIS IS X 299.327
THIS IS X 352.869
THIS IS X -nan
THIS IS X 280.268
THIS IS X 174.503
THIS IS X 174.604
THIS IS X 143.434
THIS IS X -nan
THIS IS X -nan
THIS IS X 388.47
THIS IS X 273.01

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.6 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.6-walleye-1436734861-6816.log 
// # of retrieved lines: 115 
(displaying lines: 16 -> 115)

// ------------------------------------------------

THIS IS X -nan
THIS IS X 267.587
THIS IS X 163.525
THIS IS X 317.344
THIS IS X 472.829
THIS IS X -nan
THIS IS X -nan
THIS IS X 258.877
THIS IS X 230.691
THIS IS X 433.205
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 302.25
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 148.595
THIS IS X -nan
THIS IS X 405.574
THIS IS X -nan
THIS IS X 193.828
THIS IS X -nan
THIS IS X 448.379
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 217.04
THIS IS X -nan
THIS IS X 234.59
THIS IS X 220.681
THIS IS X 397.824
THIS IS X 481.191
THIS IS X -nan
THIS IS X 421.683
THIS IS X -nan
THIS IS X 412.392
THIS IS X 283.941
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 314.17
THIS IS X 385.884
THIS IS X -nan
THIS IS X -nan
THIS IS X 558.168
THIS IS X 292.882
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 159.05
THIS IS X 257.711
THIS IS X 544.509
THIS IS X 246.518
THIS IS X -nan
THIS IS X 340.809
THIS IS X -nan
THIS IS X 159.49
THIS IS X -nan
THIS IS X -nan
THIS IS X 329.199
THIS IS X 325.677
THIS IS X 146.384
THIS IS X 384.814
THIS IS X -nan
THIS IS X 156.54
THIS IS X 314.693
THIS IS X 330.759
THIS IS X -nan
THIS IS X 139.477
THIS IS X -nan
THIS IS X 269.025
THIS IS X 190.59
THIS IS X -nan
THIS IS X 223.664
THIS IS X -nan
THIS IS X 319.165
THIS IS X 146.971
THIS IS X 195.32
THIS IS X -nan
THIS IS X -nan
THIS IS X 419.046
THIS IS X -nan
THIS IS X -nan
THIS IS X 405.405
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 397.644

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.7 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.7-walleye-1436734861-6818.log 
// # of retrieved lines: 52 


// ------------------------------------------------

17:01:01  6818 Wrk-0.7 | Info in <TProofServLite::Setup>: fWorkDir: /projects/e09084/ribbit/PROOF
17:01:05  6818 Wrk-0.7 | Info in <TProofServLite::HandleProcess>: calling fPlayer->Process() with selector name: testSelector.C+
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.C' and `./testSelector.C' are the same file
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.h' and `./testSelector.h' are the same file
17:01:06  6818 Wrk-0.7 | Info in <TProofPlayerSlave::AssertSelector>: Processing via filename
17:01:06  6818 Wrk-0.7 | Info in <TEventIter::TEventIter>: fPackets list 'ProcessedPackets_0.7' created
17:01:06  6818 Wrk-0.7 | Info in <TProofPlayerSlave::Process>: save partial results? 0  per-packet? 0
17:01:06  6818 Wrk-0.7 | Info in <TEventIterTree::GetTrees>: the tree cache is in learning phase
Array size is 32
Run 343 Settings 
Run settings loaded for run 343
Now I have Run 343
THIS IS X 450.017
THIS IS X -nan
THIS IS X 514.584
THIS IS X -nan
THIS IS X -nan
THIS IS X 349.08
17:01:08  6818 Wrk-0.7 | Info in <TProofServLite::RestartComputeTime>: compute time restarted after 0.718738 secs (100 entries)
THIS IS X 435.29
THIS IS X 145.019
THIS IS X -nan
THIS IS X -nan
THIS IS X 281.67
THIS IS X -nan
THIS IS X 222.193
THIS IS X 284.137
THIS IS X 332.385
THIS IS X -nan
THIS IS X 368.47
THIS IS X -nan
THIS IS X 328.06
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 226.187
THIS IS X -nan
THIS IS X 161.988
THIS IS X -nan
THIS IS X 210.784
THIS IS X 486.343
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 285.584
THIS IS X 364.546
THIS IS X -nan

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.8 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.8-walleye-1436734861-6820.log 
// # of retrieved lines: 52 


// ------------------------------------------------

17:01:01  6820 Wrk-0.8 | Info in <TProofServLite::Setup>: fWorkDir: /projects/e09084/ribbit/PROOF
17:01:05  6820 Wrk-0.8 | Info in <TProofServLite::HandleProcess>: calling fPlayer->Process() with selector name: testSelector.C+
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.C' and `./testSelector.C' are the same file
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.h' and `./testSelector.h' are the same file
17:01:06  6820 Wrk-0.8 | Info in <TProofPlayerSlave::AssertSelector>: Processing via filename
17:01:06  6820 Wrk-0.8 | Info in <TEventIter::TEventIter>: fPackets list 'ProcessedPackets_0.8' created
17:01:06  6820 Wrk-0.8 | Info in <TProofPlayerSlave::Process>: save partial results? 0  per-packet? 0
17:01:06  6820 Wrk-0.8 | Info in <TEventIterTree::GetTrees>: the tree cache is in learning phase
Array size is 32
Run 343 Settings 
Run settings loaded for run 343
Now I have Run 343
THIS IS X 254.513
17:01:07  6820 Wrk-0.8 | Info in <TProofServLite::RestartComputeTime>: compute time restarted after 0.716530 secs (100 entries)
THIS IS X 150.053
THIS IS X 475.454
THIS IS X -nan
THIS IS X 330.994
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 218.341
THIS IS X 311.736
THIS IS X 220.182
THIS IS X -nan
THIS IS X 371.664
THIS IS X 320.291
THIS IS X -nan
THIS IS X 303.262
THIS IS X 303.231
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 338.636
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 215.667
THIS IS X 288.125
THIS IS X 421.539
THIS IS X 321.155
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan

// --------- End of element log -------------------


Retrieving logs: 1 ok, 0 not ok (100 % processed) 


// --------- Start of element log -----------------

// Ordinal: 0.9 (role: worker)

// Path: /projects/e09084/ribbit/PROOF/projects-e09084-ribbit/session-walleye-1436734861-6789/worker-0.9-walleye-1436734861-6822.log 
// # of retrieved lines: 60 


// ------------------------------------------------

17:01:01  6822 Wrk-0.9 | Info in <TProofServLite::Setup>: fWorkDir: /projects/e09084/ribbit/PROOF
17:01:05  6822 Wrk-0.9 | Info in <TProofServLite::HandleProcess>: calling fPlayer->Process() with selector name: testSelector.C+
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.C' and `./testSelector.C' are the same file
/bin/cp: `/projects/e09084/ribbit/PROOF/cache/testSelector.h' and `./testSelector.h' are the same file
17:01:06  6822 Wrk-0.9 | Info in <TProofPlayerSlave::AssertSelector>: Processing via filename
17:01:06  6822 Wrk-0.9 | Info in <TEventIter::TEventIter>: fPackets list 'ProcessedPackets_0.9' created
17:01:06  6822 Wrk-0.9 | Info in <TProofPlayerSlave::Process>: save partial results? 0  per-packet? 0
17:01:06  6822 Wrk-0.9 | Info in <TEventIterTree::GetTrees>: the tree cache is in learning phase
Array size is 32
Run 343 Settings 
Run settings loaded for run 343
Now I have Run 343
THIS IS X 140.1
17:01:06  6822 Wrk-0.9 | Info in <TProofServLite::RestartComputeTime>: compute time restarted after 0.245800 secs (100 entries)
THIS IS X -nan
THIS IS X 236.187
THIS IS X -nan
THIS IS X 201.548
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 364.84
THIS IS X -nan
THIS IS X 506.792
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 163.162
THIS IS X -nan
THIS IS X -nan
THIS IS X 444.574
THIS IS X -nan
THIS IS X 177.856
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 298.419
THIS IS X -nan
THIS IS X -nan
THIS IS X 435.718
THIS IS X 179.154
THIS IS X 411.666
THIS IS X -nan
THIS IS X 244.931
THIS IS X 266.658
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 435.062
THIS IS X -nan
THIS IS X -nan
THIS IS X -nan
THIS IS X 363.819

// --------- End of element log -------------------

I suspect that if I were to do this with more files (events) then more of the workers would seg fault as well, although I can’t imagine where I would have a memory leak. I will try this later tonight or tomorrow in the morning.

Thank for your help,
Juan