I have a user code that stores TNtuple’s as output.
Each worker creates his own ntuple in a ROOT file, the final merged ntuple file is correctly created on the master node.
Here is what happens on the client :
If I work with a ‘normal’ PROOF session on dedicated interactive machines, there is no problem, the transfer from the master to the client works fine and the ntuple is OK.
However, when I am in ‘remote’ PROOF mode from home, the output does not reach the client (see the client log at end of post).
If I use only histos, Tlist’s and only memory-resident object as my output, everything is fine in local and remote modes.
==> The problem happens when using TNtuple’s stored in dedicated intermediate files on worker and master, and only in remote session.
I am using ALICE code for proof handling.
The problem may be on ALICE side, but I’d like to know first whether this is related to my config.
Is there some special setting should be set for that kind of matter ?
Thanks a lot !
Cheers,
Renaud
------ end of client log that doesn’t say much ---------
[TProof:] Total 10000 events2 w|====================| 100.00 % [902.1 evts/s]
Mst-0: merging output objects … done
Output file: root://ccapl0008.in2p3.fr//scratch/proo … t?remote=1
Mst-0: grand total: sent 13 objects, size: 18378157 bytes
->AliAnalysisSelector::Terminate()
->AliAnalysisManager::UnpackOutput()
Source list contains 13 containers
->AliAnalysisManager::ImportWrappers()
Importing data for container mylist
-> file MyESDOutput.root
ImportData: Unwrapping data mylist for container mylist
(const int)84
(const int)37
*** Break *** segmentation violation
Root > Function ProofTaskESDL() busy flag cleared
Most likely you have a firewall problem between the master and your home machine. The master tries to open the output file back on the home machine and for that it needs to be able to open a connection to that machine.
Indeed my proof nodes can only be accessed by clients inside the LAN of ccin2p3.
This is why we have interactive machines for users, such as lxplus at cern, and users outide ccin2p3 usually work with remote sessions using ssh tunnel.
As I said, usually everything works fine with remote sessions, but it appeared that in the end of a session, retrieving the output files.root from the master can’t be done correctly.
Although with memory-resident objects, it works really fine.
I create two ssh tunnel, one for xproofd and on for xrootd services ports that run on the master (1093 and 1094 in my case).
I tryied the xrootd tunnel successfully : I can manually write and read files to/from the master from my home machine.
However for some reason it seems that PROOF doesn’t do it.
SSH tunnels are directional: if you open the tunnel from the client to the machine, it this does not automatically work in the other direction.
PROOF does not do a ‘xrdcp’ but a TFile::Open to your final location: this is done by the Master trying to open a file on your client machine.
You have to make sure that the final destination can be accessed by the master. The best choice is a an easily accessible machine, like lxplus or similar, that your client machine can access easily. Sort of (temporary) storage element. A client in a home network is not good choice, you will always have problems of this kind.
However, you can try with the reverse tunnel (option -R instead -L):
and specifying the outfile as 'root://localhost:3000//…" . In this way all connection done by the master on its local port 3000 will be redirected via the tunnel to the port 1094 of your local client machine (where a xrootd instance is supposed to be listening).