Using data from d-cache

ROOT
1

I am attempting to use PROOF on a small test cluster of SL3 machines, using ROOT version 5.11.06 compiled from source using the following configure command:

Additionally, I commented out line 1575 of TFile.cxx before compiling:

With this setup, I can add a .root file containing a TTree object from d-cache to a TChain, and process it using a TSelector in a local session:

root [0] TChain *chain = new TChain("h1")
root [1] chain->Add("dcache:/pnfs/cms/WAX/resilient/jforbes/data_04p_47010_47139.root")
(Int_t)1
root [2] chain->Process("Analyzer.C")
(Long64_t)0

I can run over the same data file using PROOF, if I read the data from a local disk. I cannot, however, use PROOF to process the chain if I read the file from d-cache:

root [0] TChain *chain = new TChain("h1")
root [1] chain->Add("dcache:/pnfs/cms/WAX/resilient/jforbes/data_04p_47010_47139.root")
(Int_t)1
root [2] gROOT->Proof("lpcdt003")
...
root [3] chain->SetProof()
Error in <TDSet>: Error getting a tree header
Error in <TDChain>: can't set PROOF
root [4] chain->Process("Analyzer.C")
(Long64_t)0

Note: PROOF starts up normally except for a number of complaints that it can’t locate libProofx. This is because I disabled xrootd when I configured ROOT, but PROOF seems to work just fine despite these messages, when running on data from a local disk.

Although the chain is, in fact, processed, it only runs on the local machine, as opposed to the PROOF cluster.

So: is it necessary for TChainProof::MakeChainProof to get a tree header (some of the comments in older versions of the source code seemed to indicate not)? Or is there a different/better way to go about using PROOF than with chain->SetProof() ?

BTW, I am using LibDCAP version 1-2-35.

Thanks.

2

The problem here was that the worker nodes could not open the d-cache files, and was solved by using a wraparound script that would set the $LD_LIBRARY_PATH on each worker to include the location of the dcap library.

Create a file proofserv.sh

#!/bin/sh


DCAPLIB=/usr/local/d-cache/dcap/lib

if [ -z "$LD_LIBRARY_PATH ]; then
  export LD_LIBRARY_PATH=$DCAPLIB
else
  export LD_LIBRARY_PATH=$DCAPLIB:$LD_LIBRARY_PATH
fi

exec $ROOTSYS/bin/proofserv.exe "$@"

then make this file executable

$ chmod 755 $ROOTSYS/bin/proofserv.sh

rename the real proofserv

$ mv $ROOTSYS/bin/proofserv $ROOTSYS/bin/proofserv.exe

and link the script in place

$ ln -s $ROOTSYS/bin/proofserv.sh $ROOTSYS/bin/proofserv

Thanks to Philippe Canal, Gerri Ganis, and Maarten Ballintijn for their assisstance, and for this solution.